Ceph Version: 18.2.7 I've just migrated to cephadm, and upgrade from pacific to reef 18.2.7 last week. All successful except some minor issues with BlueFS Spillover Today the MDS of a specific fs refuse to start, and the ceph orch ps shows the daemons with status "error". I have three other cephfs that still works(though I haven't tested if they can fail over.) I've restartet the MDSs - No luck (the selected MDS just start/crash in a loop until it gives up) I've deployed 2 new MDSs - No luck same issue In all scenarios I see in ceph fs status, that a MDS is chosen. FS status goes to "replay" or "replay(laggy)" On the host with the MDS I see the MDS container just crashes after way less than 5 mins.. And status reported by ceph orch ps is error. (btw - mds_beacon_grace has been set to 360) I've managed to get a good 500 lines of log out with info like this: << ----------------- LOG EXAMPLE START ----------------- >> -7> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient: _check_auth_tickets -6> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2025-05-19T16:04:32.845551+0000) -5> 2025-05-19T16:05:02.860+0000 7f673e3c1640 10 monclient: get_auth_request con 0x5616e9616c00 auth_method 0 -4> 2025-05-19T16:05:02.916+0000 7f673dbc0640 10 monclient: get_auth_request con 0x5616e7422800 auth_method 0 -3> 2025-05-19T16:05:02.968+0000 7f673d3bf640 10 monclient: get_auth_request con 0x5616f5eac800 auth_method 0 -2> 2025-05-19T16:05:02.972+0000 7f6736bb2640 2 mds.0.cache Memory usage: total 574800, rss 343772, heap 207124, baseline 182548, 0 / 7535 inodes have caps, 0 caps, 0 caps per inode -1> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f67333ab640 time 2025-05-19T16:05:03.680495+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7f67406e6d2c] 2: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb] 3: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe] 4: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745] 5: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c] 6: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd] 7: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e] 8: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1] 9: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a] 10: clone() 0> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 *** Caught signal (Aborted) ** in thread 7f67333ab640 thread_name:mds-log-replay ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f674004bbf0] 2: /lib64/libc.so.6(+0x8bf5c) [0x7f6740098f5c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x178) [0x7f67406e6d86] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb] 7: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe] 8: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745] 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c] 10: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd] 11: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e] 12: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1] 13: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a] 14: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. << ----------------- LOG EXAMPLE END ----------------- >> But to be honest, out of all those lines, I don't know what to provide (all +500 might be a bit to much) I really need this FS back online, so help will be very much appreciated