CEPHFS - MDS gracefull handover of rank 0
Hello fellow CEPH-users, currently we are updating our CEPH(14.2.16) and making changes to some config settings. TLDR: is there a way to make a graceful MDS active node shutdown without loosing the caps, open files and client connections? Something like handover active state, promote standby to active, ...? Sadly we run into some difficulties when restarting MDS Nodes. While we had two active nodes and one standby we initially though that this would have a nice handover when restarting the active rank ... sadly we saw how the node was going through the states: replay-reconnect-rejoin-active as nicely visualized here https://docs.ceph.com/en/latest/cephfs/mds-states/ This left some nodes going into timeouts until the standby node has gone into the active state again, most probably since the cephfs hast already some 600k folders and 3M files and from the client side it took more than 30s. So before the next MDS the FS config where changed to one active and one standby-replay node, the idea was that since the MDS replay nodes follows the active one the handover would be smoother. The active state was reached faster, but we still noticed some hiccups on the clients while the new active MDS was waiting for clients to reconnect(state up:reconnect) after the failover. The next idea was to do a manual node promotion, graceful shutdown or something similar - where the open caps and sessions would be handed over ... but I did not find any hint in the docs regarding this functionality. But, this should somehow be possible (imho), since when adding a second active mds node (max_mds 2) and then removing it again (max_mds 1) the rank 1 node goes to stopping-state and hands over all clients/caps to rank 0 without interruptions for the clients. Therefore my question: how can one gracefully shutdown an active rank 0 mds node or promote an standby node to the active state without loosing open files/caps or client sessions? Thanks in advance, M
Hi, In our experience failovers are largely transparent if the mds has: mds session blacklist on timeout = false mds session blacklist on evict = false And clients have client reconnect stale = true Cheers, Dan On Wed, Jan 27, 2021 at 9:09 AM Martin Hronek <martin.hronek@rise-world.com> wrote:
Hello fellow CEPH-users, currently we are updating our CEPH(14.2.16) and making changes to some config settings.
TLDR: is there a way to make a graceful MDS active node shutdown without loosing the caps, open files and client connections? Something like handover active state, promote standby to active, ...?
Sadly we run into some difficulties when restarting MDS Nodes. While we had two active nodes and one standby we initially though that this would have a nice handover when restarting the active rank ... sadly we saw how the node was going through the states: replay-reconnect-rejoin-active as nicely visualized here https://docs.ceph.com/en/latest/cephfs/mds-states/
This left some nodes going into timeouts until the standby node has gone into the active state again, most probably since the cephfs hast already some 600k folders and 3M files and from the client side it took more than 30s.
So before the next MDS the FS config where changed to one active and one standby-replay node, the idea was that since the MDS replay nodes follows the active one the handover would be smoother. The active state was reached faster, but we still noticed some hiccups on the clients while the new active MDS was waiting for clients to reconnect(state up:reconnect) after the failover.
The next idea was to do a manual node promotion, graceful shutdown or something similar - where the open caps and sessions would be handed over ... but I did not find any hint in the docs regarding this functionality. But, this should somehow be possible (imho), since when adding a second active mds node (max_mds 2) and then removing it again (max_mds 1) the rank 1 node goes to stopping-state and hands over all clients/caps to rank 0 without interruptions for the clients.
Therefore my question: how can one gracefully shutdown an active rank 0 mds node or promote an standby node to the active state without loosing open files/caps or client sessions?
Thanks in advance, M _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Martin, also before restart - issue cache drop command to active mds k Sent from my iPhone
On 27 Jan 2021, at 11:58, Dan van der Ster <dan@vanderster.com> wrote:
In our experience failovers are largely transparent if the mds has:
mds session blacklist on timeout = false mds session blacklist on evict = false
And clients have
client reconnect stale = true
On 1/27/21 3:51 PM, Konstantin Shalygin wrote:
Martin, also before restart - issue cache drop command to active mds
Don't do this if you have a large cache. It will make your MDS unresponsive and replaced by a standby if available. There is a PR to fix this: https://github.com/ceph/ceph/pull/36823 I killed a 170 GB MDS this way. YMMV. Gr. Stefan
Interesting, thanks. Do you know tracker ticket for this? Thanks, k Sent from my iPhone
On 28 Jan 2021, at 00:53, Stefan Kooman <stefan@bit.nl> wrote:
Don't do this if you have a large cache. It will make your MDS unresponsive and replaced by a standby if available. There is a PR to fix this: https://github.com/ceph/ceph/pull/36823
I killed a 170 GB MDS this way. YMMV.
On 1/27/21 9:08 AM, Martin Hronek wrote:
So before the next MDS the FS config where changed to one active and one standby-replay node, the idea was that since the MDS replay nodes follows the active one the handover would be smoother. The active state was reached faster, but we still noticed some hiccups on the clients while the new active MDS was waiting for clients to reconnect(state up:reconnect) after the failover.
The next idea was to do a manual node promotion, graceful shutdown or something similar - where the open caps and sessions would be handed over ... but I did not find any hint in the docs regarding this functionality. But, this should somehow be possible (imho), since when adding a second active mds node (max_mds 2) and then removing it again (max_mds 1) the rank 1 node goes to stopping-state and hands over all clients/caps to rank 0 without interruptions for the clients.
Therefore my question: how can one gracefully shutdown an active rank 0 mds node or promote an standby node to the active state without loosing open files/caps or client sessions?
The way to upgrade a cluster, and the current limitations of it, are described here [1]. Most relevant part for you in there: Currently the MDS cluster does not have built-in versioning or file system flags to support seamless upgrades of the MDSs without potentially causing assertions or other faults due to incompatible messages or other functional differences. For this reason, it’s necessary during any cluster upgrade to reduce the number of active MDS for a file system to one first so that two active MDS do not communicate with different versions. Further, it’s also necessary to take standbys offline as any new CompatSet flags will propagate via the MDSMap to all MDS and cause older MDS to suicide. So best practices are that you have only _1_ active, upgrade the software of the last running MDS and then restart the MDS. It would be *really* nice if this could be fixed in a newer version of Ceph. Proably not trivial, but AFAIK the only part of Ceph that gives noticable impact during maintenance (like upgrades). If having this fixed is important for you, make sure you leave a note about this in the upcoming Ceph user survey. Gr. Stefan [1]: https://docs.ceph.com/en/latest/cephfs/upgrading/
participants (4)
-
Dan van der Ster
-
Konstantin Shalygin
-
Martin Hronek
-
Stefan Kooman