MDS and stretched clusters
Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other? We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode.
Hi, I'm not aware of any service settings that would allow that. You'll have to monitor each MDS state and restart any non-local active MDSs to reverse roles. Regards, Frédéric. ----- Le 29 Oct 24, à 14:06, Sake Ceph ceph@paulusma.eu a écrit :
Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other?
We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I hope someone of the development team can share some light on this. Will search the tracker if some else made a request about this.
Op 29-10-2024 16:02 CET schreef Frédéric Nass <frederic.nass@univ-lorraine.fr>:
Hi,
I'm not aware of any service settings that would allow that.
You'll have to monitor each MDS state and restart any non-local active MDSs to reverse roles.
Regards, Frédéric.
----- Le 29 Oct 24, à 14:06, Sake Ceph ceph@paulusma.eu a écrit :
Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other?
We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
No, unfortunately this needs to be done at a higher level and is not included in Ceph right now. Rook may be able to do this, but I don't think cephadm does. Adam, is there some way to finagle this with pod placement rules (ie, tagging nodes as mds and mds-standby, and then assigning special mds config info to corresponding pods)? -Greg On Tue, Oct 29, 2024 at 12:46 PM Sake Ceph <ceph@paulusma.eu> wrote:
I hope someone of the development team can share some light on this. Will search the tracker if some else made a request about this.
Op 29-10-2024 16:02 CET schreef Frédéric Nass < frederic.nass@univ-lorraine.fr>:
Hi,
I'm not aware of any service settings that would allow that.
You'll have to monitor each MDS state and restart any non-local active MDSs to reverse roles.
Regards, Frédéric.
----- Le 29 Oct 24, à 14:06, Sake Ceph ceph@paulusma.eu a écrit :
Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other?
We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, with Rook this is possible by adding zone anti-affinity for the MDS pods. Travis On Tue, Oct 29, 2024 at 3:35 PM Gregory Farnum <gfarnum@redhat.com> wrote:
No, unfortunately this needs to be done at a higher level and is not included in Ceph right now. Rook may be able to do this, but I don't think cephadm does. Adam, is there some way to finagle this with pod placement rules (ie, tagging nodes as mds and mds-standby, and then assigning special mds config info to corresponding pods)? -Greg
On Tue, Oct 29, 2024 at 12:46 PM Sake Ceph <ceph@paulusma.eu> wrote:
I hope someone of the development team can share some light on this. Will search the tracker if some else made a request about this.
Op 29-10-2024 16:02 CET schreef Frédéric Nass < frederic.nass@univ-lorraine.fr>:
Hi,
I'm not aware of any service settings that would allow that.
You'll have to monitor each MDS state and restart any non-local active MDSs to reverse roles.
Regards, Frédéric.
----- Le 29 Oct 24, à 14:06, Sake Ceph ceph@paulusma.eu a écrit :
Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other?
We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
But you don't get to choose which one is active and which one is standby, as these are states that permute over time, not configurations, or do you? I mean there's no way to tell Rook 'i want this one to be active preferably' and have Rook operator monitor MDSs and restart the non-local one if ever active so that the local preferred one becomes active. Or is Rook doing this any better than cephadm? Frédéric. ________________________________ De : Travis Nielsen <tnielsen@redhat.com> Envoyé : mardi 29 octobre 2024 23:56 À : Gregory Farnum Cc: Sake Ceph; Adam King; ceph-users Objet : [ceph-users] Re: MDS and stretched clusters Yes, with Rook this is possible by adding zone anti-affinity for the MDS pods. Travis On Tue, Oct 29, 2024 at 3:35 PM Gregory Farnum <gfarnum@redhat.com> wrote:
No, unfortunately this needs to be done at a higher level and is not included in Ceph right now. Rook may be able to do this, but I don't think cephadm does. Adam, is there some way to finagle this with pod placement rules (ie, tagging nodes as mds and mds-standby, and then assigning special mds config info to corresponding pods)? -Greg
On Tue, Oct 29, 2024 at 12:46 PM Sake Ceph <ceph@paulusma.eu> wrote:
I hope someone of the development team can share some light on this. Will search the tracker if some else made a request about this.
Op 29-10-2024 16:02 CET schreef Frédéric Nass < frederic.nass@univ-lorraine.fr>:
Hi,
I'm not aware of any service settings that would allow that.
You'll have to monitor each MDS state and restart any non-local active MDSs to reverse roles.
Regards, Frédéric.
----- Le 29 Oct 24, à 14:06, Sake Ceph ceph@paulusma.eu a écrit :
Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other?
We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just noticed this thread. A couple questions. Is what we want to have MDS daemons in say zone A and zone B, but the ones in zone A are prioritized to be active and ones in zone B remain as standby unless absolutely necessary (all the ones in zone A are down) or is it that we want to have some subset of a pool of hosts in zone A and zone B have mds daemons? If it's the former, cephadm doesn't do it. The followup question in that case would be if there is some way to tell the mds daemons to prioritize certain ones to be active over others? If there is, I didn't know about it, but I assume we'd need that functionality to get that case to work. On Tue, Oct 29, 2024 at 5:34 PM Gregory Farnum <gfarnum@redhat.com> wrote:
No, unfortunately this needs to be done at a higher level and is not included in Ceph right now. Rook may be able to do this, but I don't think cephadm does. Adam, is there some way to finagle this with pod placement rules (ie, tagging nodes as mds and mds-standby, and then assigning special mds config info to corresponding pods)? -Greg
On Tue, Oct 29, 2024 at 12:46 PM Sake Ceph <ceph@paulusma.eu> wrote:
I hope someone of the development team can share some light on this. Will search the tracker if some else made a request about this.
Op 29-10-2024 16:02 CET schreef Frédéric Nass < frederic.nass@univ-lorraine.fr>:
Hi,
I'm not aware of any service settings that would allow that.
You'll have to monitor each MDS state and restart any non-local active MDSs to reverse roles.
Regards, Frédéric.
----- Le 29 Oct 24, à 14:06, Sake Ceph ceph@paulusma.eu a écrit :
Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other?
We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We're looking for the multiple mds daemons to be active in zone A and standby(-replay) in zone B. This scenario would also benefit people who have more powerfull hardware in zone A than zone B. Kind regards, Sake
Op 31-10-2024 15:50 CET schreef Adam King <adking@redhat.com>:
Just noticed this thread. A couple questions. Is what we want to have MDS daemons in say zone A and zone B, but the ones in zone A are prioritized to be active and ones in zone B remain as standby unless absolutely necessary (all the ones in zone A are down) or is it that we want to have some subset of a pool of hosts in zone A and zone B have mds daemons? If it's the former, cephadm doesn't do it. The followup question in that case would be if there is some way to tell the mds daemons to prioritize certain ones to be active over others? If there is, I didn't know about it, but I assume we'd need that functionality to get that case to work.
On Tue, Oct 29, 2024 at 5:34 PM Gregory Farnum <gfarnum@redhat.com> wrote:
No, unfortunately this needs to be done at a higher level and is not included in Ceph right now. Rook may be able to do this, but I don't think cephadm does. Adam, is there some way to finagle this with pod placement rules (ie, tagging nodes as mds and mds-standby, and then assigning special mds config info to corresponding pods)? -Greg
On Tue, Oct 29, 2024 at 12:46 PM Sake Ceph <ceph@paulusma.eu> wrote:
I hope someone of the development team can share some light on this. Will search the tracker if some else made a request about this.
Op 29-10-2024 16:02 CET schreef Frédéric Nass < frederic.nass@univ-lorraine.fr>:
Hi,
I'm not aware of any service settings that would allow that.
You'll have to monitor each MDS state and restart any non-local active MDSs to reverse roles.
Regards, Frédéric.
----- Le 29 Oct 24, à 14:06, Sake Ceph ceph@paulusma.eu a écrit :
Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other?
We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, Oct 31, 2024 at 12:54 PM Sake Ceph <ceph@paulusma.eu> wrote:
We're looking for the multiple mds daemons to be active in zone A and standby(-replay) in zone B. This scenario would also benefit people who have more powerfull hardware in zone A than zone B.
Kind regards, Sake
Op 31-10-2024 15:50 CET schreef Adam King <adking@redhat.com>:
Just noticed this thread. A couple questions. Is what we want to have MDS daemons in say zone A and zone B, but the ones in zone A are prioritized to be active and ones in zone B remain as standby unless absolutely necessary (all the ones in zone A are down) or is it that we want to have some subset of a pool of hosts in zone A and zone B have mds daemons? If it's the former, cephadm doesn't do it. The followup question in that case would be if there is some way to tell the mds daemons to prioritize certain ones to be active over others? If there is, I didn't know about it, but I assume we'd need that functionality to get that case to work.
You used to be able to do something like this in Ceph with the standby_for_[name|rank] params, but it looks like that got removed a while ago. Patrick, is there a new way to make this happen with the mds_join_fs paradigm? -Greg
participants (5)
-
Adam King
-
Frédéric Nass
-
Gregory Farnum
-
Sake Ceph
-
Travis Nielsen