Stately MDS Transitions
All; We just started really fiddling with CephFS on our production cluster (Nautilus - 14.2.5 / 14.2.6), and I have a question... Is there a command / set of commands that transitions a standby-replay MDS server to the active role, while swapping the active MDS to standby-replay, or even just standby? I'm looking for a way to seamlessly, and without down time, prepare the active MDS to go offline (reboot) as part of planned /periodic maintenance. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
ceph mds fail c? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland t: +48 (0)124466845 f: +48 (0)124466843 e: marc@f1-outsourcing.eu -----Original Message----- From: DHilsbos@performair.com [mailto:DHilsbos@performair.com] Sent: vrijdag 28 februari 2020 16:17 To: ceph-users@ceph.io Subject: [ceph-users] Stately MDS Transitions All; We just started really fiddling with CephFS on our production cluster (Nautilus - 14.2.5 / 14.2.6), and I have a question... Is there a command / set of commands that transitions a standby-replay MDS server to the active role, while swapping the active MDS to standby-replay, or even just standby? I'm looking for a way to seamlessly, and without down time, prepare the active MDS to go offline (reboot) as part of planned /periodic maintenance. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Marc; If I understand that command correctly, it's tells MDS 'c' to disappear, the same as rebooting would, right? Let me just clarify something then... When I run ceph fs dump I get the following: 110248: [v2:10.2.80.10:6800/1470324937,v1:10.2.80.10:6801/1470324937] 'S700041' mds.0.29 up:active seq 997315 120758: [v2:10.2.80.11:6800/2691008522,v1:10.2.80.11:6801/2691008522] 'S700042' mds.0.0 up:standby-replay seq 2 Doesn't that indicate that the standby-replay server is out of date, and would need to pause the FS to replay? Is there something wrong in my setup, or is the above expected? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Marc Roos [mailto:M.Roos@f1-outsourcing.eu] Sent: Friday, February 28, 2020 8:19 AM To: ceph-users; Dominic Hilsbos Subject: RE: [ceph-users] Stately MDS Transitions ceph mds fail c? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland t: +48 (0)124466845 f: +48 (0)124466843 e: marc@f1-outsourcing.eu -----Original Message----- From: DHilsbos@performair.com [mailto:DHilsbos@performair.com] Sent: vrijdag 28 februari 2020 16:17 To: ceph-users@ceph.io Subject: [ceph-users] Stately MDS Transitions All; We just started really fiddling with CephFS on our production cluster (Nautilus - 14.2.5 / 14.2.6), and I have a question... Is there a command / set of commands that transitions a standby-replay MDS server to the active role, while swapping the active MDS to standby-replay, or even just standby? I'm looking for a way to seamlessly, and without down time, prepare the active MDS to go offline (reboot) as part of planned /periodic maintenance. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Feb 28, 2020 at 9:35 AM <DHilsbos@performair.com> wrote:
Marc;
If I understand that command correctly, it's tells MDS 'c' to disappear, the same as rebooting would, right?
Let me just clarify something then...
When I run ceph fs dump I get the following: 110248: [v2:10.2.80.10:6800/1470324937,v1:10.2.80.10:6801/1470324937] 'S700041' mds.0.29 up:active seq 997315 120758: [v2:10.2.80.11:6800/2691008522,v1:10.2.80.11:6801/2691008522] 'S700042' mds.0.0 up:standby-replay seq 2
Doesn't that indicate that the standby-replay server is out of date, and would need to pause the FS to replay?
Is there something wrong in my setup, or is the above expected?
That seq value is not about how current it is on the FS state. But you are right that even a standby-replay MDS will need to go through *some* recovery. This is usually fast, as standby-replay nodes should have just about all the relevant metadata already cached, but they still need to reconnect clients. We aren't really set up for affirmatively migrating active state between MDSes in the way you describe — if you have a multi-MDS config you can "drain" one of the running daemons, but I think that's always from the highest rank downwards, so you'd still need mds.0 to stay alive. This is a good thought for dealing with extreme-HA requirements, but I suspect you'll be okay with the standby-replay's failover ability. -Greg
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Marc Roos [mailto:M.Roos@f1-outsourcing.eu] Sent: Friday, February 28, 2020 8:19 AM To: ceph-users; Dominic Hilsbos Subject: RE: [ceph-users] Stately MDS Transitions
ceph mds fail c?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland
t: +48 (0)124466845 f: +48 (0)124466843 e: marc@f1-outsourcing.eu
-----Original Message----- From: DHilsbos@performair.com [mailto:DHilsbos@performair.com] Sent: vrijdag 28 februari 2020 16:17 To: ceph-users@ceph.io Subject: [ceph-users] Stately MDS Transitions
All;
We just started really fiddling with CephFS on our production cluster (Nautilus - 14.2.5 / 14.2.6), and I have a question...
Is there a command / set of commands that transitions a standby-replay MDS server to the active role, while swapping the active MDS to standby-replay, or even just standby?
I'm looking for a way to seamlessly, and without down time, prepare the active MDS to go offline (reboot) as part of planned /periodic maintenance.
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
DHilsbos＠performair.com
-
Gregory Farnum
-
Marc Roos