Hello Malte, Here's 'ceph fs dump': [root@pr-md-01 ~]# ceph fs dump e745793 btime 2026-09-29T19:50:54:737610+0000 enable_multiple, ever_enabled_multiple: 1,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: -1 Filesystem 'slugfs' (3) fs_name slugfs epoch 745791 flags 10 joinable allow_multimds_snaps created 2024-03-01T19:25:37.017712+0000 modified 2026-09-29T19:38:09.856131+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 3900 max_file_size 5000000000000 max_xattr_size 65536 required_client_features {} last_failure 0 last_failure_osd_epoch 76803 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} max_mds 3 in 0,1,2 up {0=19345540} failed damaged stopped data_pools [9,8] metadata_pool 7 inline_data disabled balancer bal_rank_mask -1 standby_count_wanted 0 qdb_cluster leader: 0 members: [mds.slugfs.pr-md-01.umxlxt{0:19345540} state up:resolve seq 5 join_fscid=3 addr [v2:10.50.1.74:6800/4070831430,v1:10.50.1.74:6801/4070831430] compat {c=[1],r=[1],i=[1fff]}] Standby daemons: [mds.slugfs.pr-backup-01.mlsebh{-1:19330192} state up:standby seq 1 join_fscid=3 addr [v2:10.50.1.77:6800/3223140040,v1:10.50.1.77:6801/3223140040] compat {c=[1],r=[1],i=[1fff]}] [mds.slugfs.pr-md-02.liywgj{-1:19330276} state up:standby seq 1 join_fscid=3 addr [v2:10.50.1.75:6800/216688727,v1:10.50.1.75:6801/216688727] compat {c=[1],r=[1],i=[1fff]}] [mds.slugfs.pr-md-03.pvppub{-1:19330286} state up:standby seq 1 join_fscid=3 addr [v2:10.50.1.76:6800/1948080487,v1:10.50.1.76:6801/1948080487] compat {c=[1],r=[1],i=[1fff]}] dumped fsmap epoch 745793 Logs show: Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745763 from mon.0 Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Monitors have assigned me to become a standby Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745764 from mon.0 Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 handle_mds_map I am now mds.0.745764 Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 handle_mds_map state change up:standby --> up:replay Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 replay_start Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 waiting for osdmap 76803 (which blocklists prior instance) Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.cache creating system inode with ino:0x100 Sep 29 11:22:34 pr-md-01.prism ceph-mds[2844807]: mds.0.cache creating system inode with ino:0x1 Sep 29 11:22:42 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745765 from mon.0 Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 Finished replaying journal Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 making mds journal writeable Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745766 from mon.0 Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 handle_mds_map I am now mds.0.745764 Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 handle_mds_map state change up:replay --> up:resolve Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 resolve_start Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 reopen_log Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 recovery set is Sep 29 11:22:46 pr-md-01.prism ceph-mds[2844807]: mds.0.745764 recovery set is Sep 29 11:22:50 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745767 from mon.0 Sep 29 11:23:30 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:24:48 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:25:35 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:31:10 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:32:00 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745768 from mon.0 Sep 29 11:32:07 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745769 from mon.0 Sep 29 11:32:15 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:33:42 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:35:06 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745770 from mon.0 Sep 29 11:36:38 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745774 from mon.0 Sep 29 11:36:46 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745775 from mon.0 Sep 29 11:37:13 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:38:49 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745776 from mon.0 Sep 29 11:39:34 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745777 from mon.0 Sep 29 11:39:42 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745778 from mon.0 Sep 29 11:39:47 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745779 from mon.0 Sep 29 11:39:52 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745780 from mon.0 Sep 29 11:40:11 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 11:48:12 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 12:13:10 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745784 from mon.0 Sep 29 12:15:16 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745785 from mon.0 Sep 29 12:15:22 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745786 from mon.0 Sep 29 12:15:35 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745787 from mon.0 Sep 29 12:16:01 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745788 from mon.0 Sep 29 12:16:08 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745789 from mon.0 Sep 29 12:37:46 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745790 from mon.0 Sep 29 12:38:10 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt Updating MDS map to version 745791 from mon.0 Sep 29 13:09:25 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 29 13:09:49 pr-md-01.prism ceph-mds[2844807]: mds.slugfs.pr-md-01.umxlxt asok_command: status {prefix=status} (starting...) Sep 30 00:00:03 pr-md-01.prism ceph-mds[2844807]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Sep 30 00:00:03 pr-md-01.prism ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mds-slugfs-pr-md-01-umxlxt[2844803]: 2026-09-30T07:00:03.179+0000 7f1e0495d640 -1 received signal: Hangup from Kernel ( Could be generated by p> Sep 30 00:00:03 pr-md-01.prism ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mds-slugfs-pr-md-01-umxlxt[2844803]: 2026-09-30T07:00:03.190+0000 7f1e0495d640 -1 received signal: Hangup from Kernel ( Could be generated by p> Sep 30 00:00:03 pr-md-01.prism ceph-mds[2844807]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 I think I messed things up by failing ranks 1 and 2, so I think I need to somehow bring those back into existence, then possibly figure out what to do from there. Some folks on the slack channel were suggesting I save the journals, then reset the journals, then reset the FS, then it should work. That will likely work, but if there is a way to bring ranks 1 and 2 back with a FS reset that would be nice just in the sense that I wouldn't have to reconfigure the FS settings again (I assume the reset removes all custom settings I have in place). Thanks for the help!! cheers, erich On 9/30/26 12:12 AM, Malte Stroem wrote:
Hi Erich,
show us:
ceph fs dump
There are many ways to fix this.
You can even try to deploy a new MDS on another node and look what happens.
And regarding the state. Do you see changes while watching the process? And what does the log of slugfs.pr-md-01.umxlxt say?
Best, Malte
On 9/29/26 23:19, Erich Weiler wrote:
Hey All, I’m hoping someone can help with an issue I’m having… I have Ceph 19.2.3 on Rocky 9.3, and I have a CephFS filesystem. It’s been working for years OK, through several upgrades. Just today it went offline, with all MDS daemons (3 of them) stuck in “replay” forever. I tried restarting them, etc, but nothing helped. I tried manually failing ranks 2 and 3 to make it a simple one MDS daemon, then set "ceph fs set slugfs max_mds 1" and it seems to be stuck in “resolve” state after restarting that one daemon:
[root@pr-md-01 ~]# ceph fs status slugfs - 30 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve slugfs.pr-md-01.umxlxt 679k 557k 66.4k 0 1 failed 2 failed POOL TYPE USED AVAIL cephfs_metadata metadata 797G 3026G cephfs_md_and_data data 0 36.2T cephfs_data data 1449T 72.5T MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@pr-md-01 ~]# cephadm enter --name mds.slugfs.pr-md-01.umxlxt -- ceph daemon /var/run/ceph/ceph-mds.slugfs.pr-md-01.umxlxt.asok status Inferring fsid 58bde08a-d7ed-11ee-9098-506b4b4da440 { "cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", "whoami": 0, "id": 19345540, "want_state": "up:resolve", "state": "up:resolve", "fs_name": "slugfs", "resolve_status": { "resolve_gather": "", "resolve_ack_gather": "" }, "rank_uptime": 1538.6363653870001, "mdsmap_epoch": 745780, "osdmap_epoch": 76803, "osdmap_epoch_barrier": 0, "uptime": 1538.823382218 }
Ranks 1 and 2 are manually failed for now just to try to get things going again.
I ran these commands to remove the “failed” ranks (which I probably should not have done, in retrospect):
ceph mds fail slugfs:1 --yes-i-really-mean-it ceph mds fail slugfs:2 --yes-i-really-mean-it ceph mds rmfailed slugfs:1 --yes-i-really-mean-it ceph mds rmfailed slugfs:2 --yes-i-really-mean-it
Any ideas how I can get the MDS back? I am up against a wall… I did dump the journals. I think I just need to get ranks 1 and 2 back online in addition to rank 0, then get the MDS's to move from "resolving" to "up"...
Thanks for any advice!!
cheers, erich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io