27 Jan
2021
27 Jan
'21
12:08 a.m.
Hello fellow CEPH-users,
currently we are updating our CEPH(14.2.16) and making changes to some
config settings.
TLDR: is there a way to make a graceful MDS active node shutdown without
loosing the caps, open files and client connections? Something like
handover active state, promote standby to active, ...?
Sadly we run into some difficulties when restarting MDS Nodes. While we
had two active nodes and one standby we initially though that this would
have a nice handover when restarting the active rank ... sadly we saw
how the node was going through the states:
replay-reconnect-rejoin-active as nicely visualized here
https://docs.ceph.com/en/latest/cephfs/mds-states/
This left some nodes going into timeouts until the standby node has gone
into the active state again, most probably since the cephfs hast already
some 600k folders and 3M files and from the client side it took more
than 30s.
So before the next MDS the FS config where changed to one active and one
standby-replay node, the idea was that since the MDS replay nodes
follows the active one the handover would be smoother. The active state
was reached faster, but we still noticed some hiccups on the clients
while the new active MDS was waiting for clients to reconnect(state
up:reconnect) after the failover.
The next idea was to do a manual node promotion, graceful shutdown or
something similar - where the open caps and sessions would be handed
over ... but I did not find any hint in the docs regarding this
functionality.
But, this should somehow be possible (imho), since when adding a second
active mds node (max_mds 2) and then removing it again (max_mds 1) the
rank 1 node goes to stopping-state and hands over all clients/caps to
rank 0 without interruptions for the clients.
Therefore my question: how can one gracefully shutdown an active rank 0
mds node or promote an standby node to the active state without loosing
open files/caps or client sessions?
Thanks in advance,
M
27 Jan
27 Jan
12:56 a.m.
Hi,
In our experience failovers are largely transparent if the mds has:
mds session blacklist on timeout = false
mds session blacklist on evict = false
And clients have
client reconnect stale = true
Cheers, Dan
On Wed, Jan 27, 2021 at 9:09 AM Martin Hronek
<martin.hronek(a)rise-world.com> wrote:
>
> Hello fellow CEPH-users,
> currently we are updating our CEPH(14.2.16) and making changes to some
> config settings.
>
> TLDR: is there a way to make a graceful MDS active node shutdown without
> loosing the caps, open files and client connections? Something like
> handover active state, promote standby to active, ...?
>
>
> Sadly we run into some difficulties when restarting MDS Nodes. While we
> had two active nodes and one standby we initially though that this would
> have a nice handover when restarting the active rank ... sadly we saw
> how the node was going through the states:
> replay-reconnect-rejoin-active as nicely visualized here
> https://docs.ceph.com/en/latest/cephfs/mds-states/
>
> This left some nodes going into timeouts until the standby node has gone
> into the active state again, most probably since the cephfs hast already
> some 600k folders and 3M files and from the client side it took more
> than 30s.
>
> So before the next MDS the FS config where changed to one active and one
> standby-replay node, the idea was that since the MDS replay nodes
> follows the active one the handover would be smoother. The active state
> was reached faster, but we still noticed some hiccups on the clients
> while the new active MDS was waiting for clients to reconnect(state
> up:reconnect) after the failover.
>
> The next idea was to do a manual node promotion, graceful shutdown or
> something similar - where the open caps and sessions would be handed
> over ... but I did not find any hint in the docs regarding this
> functionality.
> But, this should somehow be possible (imho), since when adding a second
> active mds node (max_mds 2) and then removing it again (max_mds 1) the
> rank 1 node goes to stopping-state and hands over all clients/caps to
> rank 0 without interruptions for the clients.
>
> Therefore my question: how can one gracefully shutdown an active rank 0
> mds node or promote an standby node to the active state without loosing
> open files/caps or client sessions?
>
> Thanks in advance,
> M
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:51 a.m.
Martin, also before restart - issue cache drop command to active mds
k
Sent from my iPhone
> On 27 Jan 2021, at 11:58, Dan van der Ster <dan(a)vanderster.com> wrote:
>
> In our experience failovers are largely transparent if the mds has:
>
> mds session blacklist on timeout = false
> mds session blacklist on evict = false
>
> And clients have
>
> client reconnect stale = true
1:47 p.m.
On 1/27/21 3:51 PM, Konstantin Shalygin wrote:
Martin, also before restart - issue cache drop command
to active mds
Don't do this if you have a large cache. It will make your MDS
unresponsive and replaced by a standby if available. There is a PR to
fix this: https://github.com/ceph/ceph/pull/36823
I killed a 170 GB MDS this way. YMMV.
Gr. Stefan
8:10 p.m.
Interesting, thanks.
Do you know tracker ticket for this?
Thanks,
k
Sent from my iPhone
On 28 Jan 2021, at 00:53, Stefan Kooman
<stefan(a)bit.nl> wrote:
Don't do this if you have a large cache. It will make your MDS unresponsive and
replaced by a standby if available. There is a PR to fix this:
https://github.com/ceph/ceph/pull/36823
I killed a 170 GB MDS this way. YMMV.
12 Feb
12 Feb
2:24 a.m.
On 1/27/21 9:08 AM, Martin Hronek wrote:
So before the next MDS the FS config where changed to one active and one
standby-replay node, the idea was that since the MDS replay nodes
follows the active one the handover would be smoother. The active state
was reached faster, but we still noticed some hiccups on the clients
while the new active MDS was waiting for clients to reconnect(state
up:reconnect) after the failover.
The next idea was to do a manual node promotion, graceful shutdown or
something similar - where the open caps and sessions would be handed
over ... but I did not find any hint in the docs regarding this
functionality.
But, this should somehow be possible (imho), since when adding a second
active mds node (max_mds 2) and then removing it again (max_mds 1) the
rank 1 node goes to stopping-state and hands over all clients/caps to
rank 0 without interruptions for the clients.
Therefore my question: how can one gracefully shutdown an active rank 0
mds node or promote an standby node to the active state without loosing
open files/caps or client sessions?
The way to upgrade a cluster, and the current limitations of it, are
described here [1]. Most relevant part for you in there:
Currently the MDS cluster does not have built-in versioning or file
system flags to support seamless upgrades of the MDSs without
potentially causing assertions or other faults due to incompatible
messages or other functional differences. For this reason, it’s
necessary during any cluster upgrade to reduce the number of active MDS
for a file system to one first so that two active MDS do not communicate
with different versions. Further, it’s also necessary to take standbys
offline as any new CompatSet flags will propagate via the MDSMap to all
MDS and cause older MDS to suicide.
So best practices are that you have only _1_ active, upgrade the
software of the last running MDS and then restart the MDS.
It would be *really* nice if this could be fixed in a newer version of
Ceph. Proably not trivial, but AFAIK the only part of Ceph that gives
noticable impact during maintenance (like upgrades). If having this
fixed is important for you, make sure you leave a note about this in the
upcoming Ceph user survey.
Gr. Stefan
[1]: https://docs.ceph.com/en/latest/cephfs/upgrading/
1169
days inactive
1185
days old
6 comments
4 participants
participants (4)
-
Dan van der Ster -
Konstantin Shalygin -
Martin Hronek -
Stefan Kooman