Hi all, we have the following problem with our CephFS Setup (Ceph version 19.2.2). Today our two active MDS nodes failed and then the nodes that were in „stand-by replay“ took over and failed as well. The CephFS system is equipped as follows: - 3 monitor nodes - 4 MDS nodes (2 active/ 2 stand-by) - 2 active - 2 stand-by replay - CephFS Pool - max_mds = 2 - 1x Meta data pool - 2x data pools (hdd_pool, ssd_pool) << ----------------- Ceph fs status output START----------------- >> cephfs - 0 clients RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 failed << ----------------- Ceph fs status output END----------------- >> To restore the service, we used the documentation https://docs.ceph.com/en/quincy/cephfs/disaster-recovery-experts/?highlight=... we carried out the steps up to and including "MDS table wipes". We did not carry out the MDS MAP RESET step as we were not sure that we would then lose all the data from RANK 1. We also carried out the steps under "Avoiding recovery roadblocks" https://docs.ceph.com/en/quincy/cephfs/troubleshooting/#avoiding-recovery-ro.... Parameters set of the MDS nodes: mds advanced mds_abort_on_newly_corrupt_dentry false mds advanced mds_bal_interval 0 mds basic mds_cache_memory_limit 274877906944 mds advanced mds_cache_trim_threshold 524288 mds advanced mds_go_bad_corrupt_dentry false mds advanced mds_heartbeat_grace 3600. 000000 mds advanced mds_min_caps_working_set 60000 mds advanced mds_oft_prefetch_dirfrags false * After trying the recovery steps (truncating the journal) the MDS daemons are in a crash -> restart loop behavior. << ----------------- Example log file mds-1: START ----------------- >> -14> 2025-05-21T17:49:27.003+0200 7f36cfc7e640 1 mds.0.42052 active_start -13> 2025-05-21T17:49:27.003+0200 7f36d2c84640 10 monclient: get_auth_request con 0x559a76e8a400 auth_method 0 -12> 2025-05-21T17:49:27.003+0200 7f36d2483640 10 monclient: get_auth_request con 0x559a7e041400 auth_method 0 -11> 2025-05-21T17:49:27.003+0200 7f36d3485640 10 monclient: get_auth_request con 0x559a76e8b000 auth_method 0 -10> 2025-05-21T17:49:27.003+0200 7f36d3485640 10 monclient: get_auth_request con 0x559ab89cb800 auth_method 0 -9> 2025-05-21T17:49:27.003+0200 7f36d2c84640 10 monclient: get_auth_request con 0x559a7bcc6400 auth_method 0 -8> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 1 mds.0.42052 cluster recovered. -7> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 4 mds.0.42052 set_osd_epoch_barrier: epoch=492573 -6> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 5 quiesce.mds.0 <quiesce_cluster_update> epoch:42055 me:7764062 leader:7764062 members:7764062 -5> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 5 quiesce.mgr.0 <update_membership> starting the db mgr thread at epoch: 42055 -4> 2025-05-21T17:49:27.015+0200 7f36c5c6a640 5 quiesce.mgr.0 <quiesce_db_thread_main> Entering the main thread -3> 2025-05-21T17:49:27.015+0200 7f36c5c6a640 5 quiesce.mgr.0 <membership_upkeep> a reset of the db has been requested -2> 2025-05-21T17:49:27.015+0200 7f36c9471640 -1 mds.0.cache.den(0x1 techfak) newly corrupt dentry to be committed: [dentry #0x1/techfak [c,head] auth (dversion lock) pv=0 v=52947746 ino=0x1000a58d072 state=1073741824 | inodepin=1 0x559a755b2c80] -1> 2025-05-21T17:49:27.015+0200 7f36c9471640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/mds/MDCache.cc: In function 'void MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)' thread 7f36c9471640 time 2025-05-21T17:49:27.020101+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/mds/MDCache.cc: 1687: FAILED ceph_assert(follows >= realm->get_newest_seq()) ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f36d5709cf9] 2: /usr/lib64/ceph/libceph-common.so.2(+0x182eb8) [0x7f36d5709eb8] 3: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xac3) [0x559a51ca4583] 4: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xbd) [0x559a51ca50cd] 5: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0xe71) [0x559a51cab8f1] 6: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x473) [0x559a51d55a33] 7: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x390) [0x559a51d2e750] 8: (MDSContext::complete(int)+0x5c) [0x559a51e4617c] 9: (MDSIOContextBase::complete(int)+0x34c) [0x559a51e4884c] 10: /usr/bin/ceph-mds(+0x4f5970) [0x559a51eed970] 11: /usr/bin/ceph-mds(+0x160f0d) [0x559a51b58f0d] 12: (Finisher::finisher_thread_entry()+0x17d) [0x7f36d57c885d] 13: /lib64/libc.so.6(+0x8a0ca) [0x7f36d50a30ca] 14: /lib64/libc.so.6(+0x10f150) [0x7f36d5128150] 0> 2025-05-21T17:49:27.019+0200 7f36c9471640 -1 *** Caught signal (Aborted) ** in thread 7f36c9471640 thread_name: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f36d5057bf0] 2: /lib64/libc.so.6(+0x8be0c) [0x7f36d50a4e0c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f36d5709d53] 6: /usr/lib64/ceph/libceph-common.so.2(+0x182eb8) [0x7f36d5709eb8] 7: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xac3) [0x559a51ca4583] 8: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xbd) [0x559a51ca50cd] 9: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0xe71) [0x559a51cab8f1] 10: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x473) [0x559a51d55a33] 11: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x390) [0x559a51d2e750] 12: (MDSContext::complete(int)+0x5c) [0x559a51e4617c] 13: (MDSIOContextBase::complete(int)+0x34c) [0x559a51e4884c] 14: /usr/bin/ceph-mds(+0x4f5970) [0x559a51eed970] 15: /usr/bin/ceph-mds(+0x160f0d) [0x559a51b58f0d] 16: (Finisher::finisher_thread_entry()+0x17d) [0x7f36d57c885d] 17: /lib64/libc.so.6(+0x8a0ca) [0x7f36d50a30ca] 18: /lib64/libc.so.6(+0x10f150) [0x7f36d5128150] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. << ----------------- Example log file mds-1: END ----------------- >> << ----------------- Ceph fs fs dump output START----------------- >> e41371 btime 2025-05-21T16:19:27:085643+0200 enable_multiple, ever_enabled_multiple: 1,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 3 Filesystem 'cephfs' (3) fs_name cephfs epoch 41370 flags 73 allow_snaps allow_multimds_snaps allow_standby_replay refuse_client_session created 2024-03-31T23:36:25.302389+0200 modified 2025-05-21T16:19:09.237977+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 max_xattr_size 65536 required_client_features {} last_failure 0 last_failure_osd_epoch 492429 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} max_mds 2 in 0,1 up {1=7761302} failed 0 damaged stopped data_pools [5,3] metadata_pool 4 inline_data disabled balancer bal_rank_mask -1 standby_count_wanted 2 qdb_cluster leader: 7761302 members: 7761302 [mds.mds-1{1:7761302} state up:active seq 5 addr [v2:[2001:638:504:2011:9:3:1:1]:6800/2463826788,v1:[2001:638:504:2011:9:3:1:1]:6801/2463826788] compat {c=[1],r=[1],i=[1fff]}] Standby daemons: [mds.mds-2{-1:7771506} state up:standby seq 1 addr [v2:[2001:638:504:2011:6:3:2:2]:6800/2809657192,v1:[2001:638:504:2011:6:3:2:2]:6801/2809657192] compat {c=[1],r=[1],i=[1fff]}] dumped fsmap epoch 41371 << ----------------- Ceph fs fs dump output END———————— >> But to be honest, out of all those things we tried, I don't know what to provide exactly. We can provide much more but ... We really need the service back online, so help will be very much appreciated. Regards, Michael