Hi Erich, reduce max_mds to 1! Deploy a new MDS with cephadm on another node where no MDS is running yet. And yes, the guys on Slack are right. But first try doing the reducing and deploying. And watch the logs! Best, Malte On 9/30/26 15:20, Erich Weiler wrote:
Hello Malte,
Here's 'ceph fs dump':
[root@pr-md-01 ~]# ceph fs dump e745793 btime 2026-09-29T19:50:54:737610+0000 enable_multiple, ever_enabled_multiple: 1,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: -1
Filesystem 'slugfs' (3) fs_name slugfs epoch 745791 flags 10 joinable allow_multimds_snaps created 2024-03-01T19:25:37.017712+0000 modified 2026-09-29T19:38:09.856131+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 3900 max_file_size 5000000000000 max_xattr_size 65536 required_client_features {} last_failure 0 last_failure_osd_epoch 76803 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} max_mds 3 in 0,1,2 up {0=19345540} failed damaged stopped data_pools [9,8] metadata_pool 7 inline_data disabled balancer bal_rank_mask -1 standby_count_wanted 0 qdb_cluster leader: 0 members: [mds.slugfs.pr-md-01.umxlxt{0:19345540} state up:resolve seq 5 join_fscid=3 addr [v2:10.50.1.74:6800/4070831430,v1:10.50.1.74:6801/4070831430] compat {c=[1],r=[1],i=[1fff]}]
Standby daemons:
[mds.slugfs.pr-backup-01.mlsebh{-1:19330192} state up:standby seq 1 join_fscid=3 addr [v2:10.50.1.77:6800/3223140040,v1:10.50.1.77:6801/3223140040] compat {c=[1],r=[1],i=[1fff]}] [mds.slugfs.pr-md-02.liywgj{-1:19330276} state up:standby seq 1 join_fscid=3 addr [v2:10.50.1.75:6800/216688727,v1:10.50.1.75:6801/216688727] compat {c=[1],r=[1],i=[1fff]}] [mds.slugfs.pr-md-03.pvppub{-1:19330286} state up:standby seq 1 join_fscid=3 addr [v2:10.50.1.76:6800/1948080487,v1:10.50.1.76:6801/1948080487] compat {c=[1],r=[1],i=[1fff]}] dumped fsmap epoch 745793
Logs show:
Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745763 from mon.0 Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Monitors have assigned me to become a standby Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745764 from mon.0 Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 handle_mds_map I am now mds.0.745764 Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 handle_mds_map state change up:standby --> up:replay Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 replay_start Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 waiting for osdmap 76803 (which blocklists prior instance) Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.cache creating system inode with ino:0x100 Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.cache creating system inode with ino:0x1 Sep 29 11:22:42 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745765 from mon.0 Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 Finished replaying journal Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 making mds journal writeable Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745766 from mon.0 Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 handle_mds_map I am now mds.0.745764 Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 handle_mds_map state change up:replay --> up:resolve Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 resolve_start Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 reopen_log Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 recovery set is Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 recovery set is Sep 29 11:22:50 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745767 from mon.0 Sep 29 11:23:30 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:24:48 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:25:35 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:31:10 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:32:00 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745768 from mon.0 Sep 29 11:32:07 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745769 from mon.0 Sep 29 11:32:15 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:33:42 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:35:06 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745770 from mon.0 Sep 29 11:36:38 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745774 from mon.0 Sep 29 11:36:46 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745775 from mon.0 Sep 29 11:37:13 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:38:49 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745776 from mon.0 Sep 29 11:39:34 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745777 from mon.0 Sep 29 11:39:42 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745778 from mon.0 Sep 29 11:39:47 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745779 from mon.0 Sep 29 11:39:52 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745780 from mon.0 Sep 29 11:40:11 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:48:12 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 12:13:10 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745784 from mon.0 Sep 29 12:15:16 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745785 from mon.0 Sep 29 12:15:22 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745786 from mon.0 Sep 29 12:15:35 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745787 from mon.0 Sep 29 12:16:01 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745788 from mon.0 Sep 29 12:16:08 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745789 from mon.0 Sep 29 12:37:46 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745790 from mon.0 Sep 29 12:38:10 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt Updating MDS map to version 745791 from mon.0 Sep 29 13:09:25 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 13:09:49 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr- md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 30 00:00:03 pr-md-01.prism ceph-mds[2844807]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Sep 30 00:00:03 pr-md-01.prism ceph-58bde08a- d7ed-11ee-9098-506b4b4da440-mds-slugfs-pr-md-01-umxlxt[2844803]: 2026-09-30T07:00:03.179+0000 7f1e0495d640 -1 received signal: Hangup from Kernel ( Could be generated by p> Sep 30 00:00:03 pr-md-01.prism ceph-58bde08a- d7ed-11ee-9098-506b4b4da440-mds-slugfs-pr-md-01-umxlxt[2844803]: 2026-09-30T07:00:03.190+0000 7f1e0495d640 -1 received signal: Hangup from Kernel ( Could be generated by p> Sep 30 00:00:03 pr-md-01.prism ceph-mds[2844807]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
I think I messed things up by failing ranks 1 and 2, so I think I need to somehow bring those back into existence, then possibly figure out what to do from there. Some folks on the slack channel were suggesting I save the journals, then reset the journals, then reset the FS, then it should work. That will likely work, but if there is a way to bring ranks 1 and 2 back with a FS reset that would be nice just in the sense that I wouldn't have to reconfigure the FS settings again (I assume the reset removes all custom settings I have in place).
Thanks for the help!!
cheers, erich
On 9/30/26 12:12 AM, Malte Stroem wrote:
Hi Erich,
show us:
ceph fs dump
There are many ways to fix this.
You can even try to deploy a new MDS on another node and look what happens.
And regarding the state. Do you see changes while watching the process? And what does the log of slugfs.pr-md-01.umxlxt say?
Best, Malte
On 9/29/26 23:19, Erich Weiler wrote:
Hey All, I’m hoping someone can help with an issue I’m having… I have Ceph 19.2.3 on Rocky 9.3, and I have a CephFS filesystem. It’s been working for years OK, through several upgrades. Just today it went offline, with all MDS daemons (3 of them) stuck in “replay” forever. I tried restarting them, etc, but nothing helped. I tried manually failing ranks 2 and 3 to make it a simple one MDS daemon, then set "ceph fs set slugfs max_mds 1" and it seems to be stuck in “resolve” state after restarting that one daemon:
[root@pr-md-01 ~]# ceph fs status slugfs - 30 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve slugfs.pr-md-01.umxlxt 679k 557k 66.4k 0 1 failed 2 failed POOL TYPE USED AVAIL cephfs_metadata metadata 797G 3026G cephfs_md_and_data data 0 36.2T cephfs_data data 1449T 72.5T MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@pr-md-01 ~]# cephadm enter --name mds.slugfs.pr-md-01.umxlxt -- ceph daemon /var/run/ceph/ceph-mds.slugfs.pr-md-01.umxlxt.asok status Inferring fsid 58bde08a-d7ed-11ee-9098-506b4b4da440 { "cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", "whoami": 0, "id": 19345540, "want_state": "up:resolve", "state": "up:resolve", "fs_name": "slugfs", "resolve_status": { "resolve_gather": "", "resolve_ack_gather": "" }, "rank_uptime": 1538.6363653870001, "mdsmap_epoch": 745780, "osdmap_epoch": 76803, "osdmap_epoch_barrier": 0, "uptime": 1538.823382218 }
Ranks 1 and 2 are manually failed for now just to try to get things going again.
I ran these commands to remove the “failed” ranks (which I probably should not have done, in retrospect):
ceph mds fail slugfs:1 --yes-i-really-mean-it ceph mds fail slugfs:2 --yes-i-really-mean-it ceph mds rmfailed slugfs:1 --yes-i-really-mean-it ceph mds rmfailed slugfs:2 --yes-i-really-mean-it
Any ideas how I can get the MDS back? I am up against a wall… I did dump the journals. I think I just need to get ranks 1 and 2 back online in addition to rank 0, then get the MDS's to move from "resolving" to "up"...
Thanks for any advice!!
cheers, erich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io