Hey All, I’m hoping someone can help with an issue I’m having… I have Ceph 19.2.3 on Rocky 9.3, and I have a CephFS filesystem. It’s been working for years OK, through several upgrades. Just today it went offline, with all MDS daemons (3 of them) stuck in “replay” forever. I tried restarting them, etc, but nothing helped. I tried manually failing ranks 2 and 3 to make it a simple one MDS daemon, then set "ceph fs set slugfs max_mds 1" and it seems to be stuck in “resolve” state after restarting that one daemon: [root@pr-md-01 ~]# ceph fs status slugfs - 30 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve slugfs.pr-md-01.umxlxt 679k 557k 66.4k 0 1 failed 2 failed POOL TYPE USED AVAIL cephfs_metadata metadata 797G 3026G cephfs_md_and_data data 0 36.2T cephfs_data data 1449T 72.5T MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@pr-md-01 ~]# cephadm enter --name mds.slugfs.pr-md-01.umxlxt -- ceph daemon /var/run/ceph/ceph-mds.slugfs.pr-md-01.umxlxt.asok status Inferring fsid 58bde08a-d7ed-11ee-9098-506b4b4da440 { "cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", "whoami": 0, "id": 19345540, "want_state": "up:resolve", "state": "up:resolve", "fs_name": "slugfs", "resolve_status": { "resolve_gather": "", "resolve_ack_gather": "" }, "rank_uptime": 1538.6363653870001, "mdsmap_epoch": 745780, "osdmap_epoch": 76803, "osdmap_epoch_barrier": 0, "uptime": 1538.823382218 } Ranks 1 and 2 are manually failed for now just to try to get things going again. I ran these commands to remove the “failed” ranks (which I probably should not have done, in retrospect): ceph mds fail slugfs:1 --yes-i-really-mean-it ceph mds fail slugfs:2 --yes-i-really-mean-it ceph mds rmfailed slugfs:1 --yes-i-really-mean-it ceph mds rmfailed slugfs:2 --yes-i-really-mean-it Any ideas how I can get the MDS back? I am up against a wall… I did dump the journals. I think I just need to get ranks 1 and 2 back online in addition to rank 0, then get the MDS's to move from "resolving" to "up"... Thanks for any advice!! cheers, erich