Hi, we ran into a bigger problem today with our ceph cluster (Quincy, Alma8.9). We have 4 filesystems and a total of 6 MDs, the largest fs having two ranks assigned (i.e. one standby). Since we often have the problem of MDs lagging behind, we restart the MDs occasionally. Helps ususally, the standby taking over. Today however, the restart didn't work and the rank 1 MDs started to crash for unclear reasons. Rank 0 seemed ok. We decided at some point to go back to one rank by settings max_mds to 1. Due to the permanent crashing, the rank1 didn't stop however, and at some point we set it to failed and the fs not joinable. At this point it looked like this: fs_cluster - 716 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephmd6a Reqs: 0 /s 13.1M 13.1M 1419k 79.2k 1 failed POOL TYPE USED AVAIL fs_cluster_meta metadata 1791G 54.2T fs_cluster_data data 421T 54.2T with rank1 still being listed. The next attempt was to remove that failed ceph mds rmfailed fs_cluster:1 --yes-i-really-mean-it which, after a short while brought down 3 out of five MONs. They keep crashing shortly after restart with stack traces like this: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff8813adcf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff8809bf09b] 5: /lib64/libstdc++.so.6(+0x9654c) [0x7ff8809c554c] 6: /lib64/libstdc++.so.6(+0x965a7) [0x7ff8809c55a7] 7: /lib64/libstdc++.so.6(+0x96808) [0x7ff8809c5808] 8: /lib64/libstdc++.so.6(+0x92045) [0x7ff8809c1045] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xa9e) [0x55f05d9a5e8e] 10: (MDSMonitor::tick()+0x18a) [0x55f05d9b18da] 11: (MDSMonitor::on_active()+0x2c) [0x55f05d99a17c] 12: (Context::complete(int)+0xd) [0x55f05d76c56d] 13: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0x9d) [0x55f05 d799d7d] 14: (Paxos::finish_round()+0x74) [0x55f05d8c5c24] 15: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x41b) [0x55f05d8c7e5b] 16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x123e) [0x55f05d76a2ae] 17: (Monitor::_ms_dispatch(Message*)+0x406) [0x55f05d76a976] 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x55f05d79b3ed] 19: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7ff88367fed8] 20: (DispatchQueue::entry()+0x50f) [0x7ff88367d31f] 21: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff883747381] 22: /lib64/libpthread.so.0(+0x81ca) [0x7ff8813a31ca] 23: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. The MDSMonitor::maybe_resize_cluster somehow suggests a connection to the above MDs operation. Does anyone have an idea how to get this cluster back together again ? Like manually fixing the MD ranks ? 'fs get' can still be called in short moments where enough MONs are reachable: fs_name fs_cluster epoch 3065486 flags 13 allow_snaps allow_multimds_snaps created 2022-08-26T15:55:07.186477+0200 modified 2024-05-28T12:21:59.294364+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 4398046511104 required_client_features {} last_failure 0 last_failure_osd_epoch 1777109 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0,1 up {0=911794623} failed damaged stopped 2,3 data_pools [32] metadata_pool 33 inline_data disabled balancer standby_count_wanted 1 [mds.cephmd6a{0:911794623} state up:active seq 22777 addr [v2:10.13.5.6:6800/189084355,v1:10.13.5.6:6801/189084355] compat {c=[1],r=[1],i=[7ff]}] Regards, Noe