Ceph MDS stuck in reconnect -> rejoin -> failover loop
Hi Ceph pacific 16.2.15 I have 5 MDS hosts, 4 active (4 FS), and 1 standby. One MDS was restarted today (as part of OS Patching), resulting in a failover. This is usually not an issue but today,it got stuck in a reconnect -> rejoin -> failover loop for the specific FS. A ceph fs status shows that during the time the FS is in state "rejoin" the INOS rise to +50M (usually it is around 10-12M ) The memory on the MDS host is eaten, (MDS cache size is 36GB, but it rises to +140 GB.) Finaly it fails over, and the cycle starts over. We are currently restarting all clients, in an effort to rule out buggy clients. Any help on this issue will be very much appreciated. Thank you
ISSUE FIXED After rebooting all clients, and setting - ceph config set mon mds_beacon_grace 120 The MDS finally went active ________________________________ From: Kasper Rasmussen <kasper_steengaard@hotmail.com> Sent: Monday, March 31, 2025 09:46 To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Ceph MDS stuck in reconnect -> rejoin -> failover loop Hi Ceph pacific 16.2.15 I have 5 MDS hosts, 4 active (4 FS), and 1 standby. One MDS was restarted today (as part of OS Patching), resulting in a failover. This is usually not an issue but today,it got stuck in a reconnect -> rejoin -> failover loop for the specific FS. A ceph fs status shows that during the time the FS is in state "rejoin" the INOS rise to +50M (usually it is around 10-12M ) The memory on the MDS host is eaten, (MDS cache size is 36GB, but it rises to +140 GB.) Finaly it fails over, and the cycle starts over. We are currently restarting all clients, in an effort to rule out buggy clients. Any help on this issue will be very much appreciated. Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Kasper, Glad to hear you got it fixed. If you want to dig deeper into the root cause, check the mds logs for any backtraces. Here's an issue that comes to mind as possibly causing this for you: https://tracker.ceph.com/issues/64717 https://tracker.ceph.com/issues/64348 https://tracker.ceph.com/issues/68374 Cheers, Dan On Mon, Mar 31, 2025 at 1:37 AM Kasper Rasmussen <kasper_steengaard@hotmail.com> wrote:
ISSUE FIXED
After rebooting all clients, and setting - ceph config set mon mds_beacon_grace 120 The MDS finally went active
________________________________ From: Kasper Rasmussen <kasper_steengaard@hotmail.com> Sent: Monday, March 31, 2025 09:46 To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Ceph MDS stuck in reconnect -> rejoin -> failover loop
Hi
Ceph pacific 16.2.15
I have 5 MDS hosts, 4 active (4 FS), and 1 standby.
One MDS was restarted today (as part of OS Patching), resulting in a failover. This is usually not an issue but today,it got stuck in a reconnect -> rejoin -> failover loop for the specific FS.
A ceph fs status shows that during the time the FS is in state "rejoin" the INOS rise to +50M (usually it is around 10-12M )
The memory on the MDS host is eaten, (MDS cache size is 36GB, but it rises to +140 GB.)
Finaly it fails over, and the cycle starts over.
We are currently restarting all clients, in an effort to rule out buggy clients.
Any help on this issue will be very much appreciated. Thank you
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
participants (2)
-
Dan van der Ster
-
Kasper Rasmussen