Hi, a 17.2.7 cluster with two filesystems has suddenly non-working MDSs: # ceph -s cluster: id: f54eea86-265a-11eb-a5d0-457857ba5742 health: HEALTH_ERR 22 failed cephadm daemon(s) 2 filesystems are degraded 1 mds daemon damaged insufficient standby MDS daemons available services: mon: 5 daemons, quorum ceph00,ceph03,ceph04,ceph01,ceph02 (age 4h) mgr: ceph03.odfupq(active, since 4h), standbys: ppc721.vsincn, ceph00.lvbddp, ceph02.zhyxjg, ceph06.eifppc mds: 4/5 daemons up osd: 145 osds: 145 up (since 20h), 145 in (since 2d) rgw: 12 daemons active (4 hosts, 1 zones) data: volumes: 0/2 healthy, 2 recovering; 1 damaged pools: 15 pools, 4897 pgs objects: 195.64M objects, 195 TiB usage: 617 TiB used, 527 TiB / 1.1 PiB avail pgs: 4892 active+clean 5 active+clean+scrubbing+deep io: client: 2.5 MiB/s rd, 20 MiB/s wr, 665 op/s rd, 938 op/s wr # ceph fs status ABC - 4 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 resolve ABC.ceph04.lzlkdu 0 3 1 0 2 resolve ABC.ppc721.rzfmyi 0 3 1 0 3 resolve ABC.ceph04.jiepaw 249 252 13 0 POOL TYPE USED AVAIL cephfs.ABC.meta metadata 33.0G 104T cephfs.ABC.data data 390T 104T DEF - 154 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 rejoin(laggy) DEF.ceph06.etthum 30.9k 30.8k 5084 0 POOL TYPE USED AVAIL cephfs.DEF.meta metadata 190G 104T cephfs.DEF.data data 118T 104T MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) The first filesystem will not get an MDS in rank 0, we already tried to set max_msd to 1 but to no avail. The second filesystem's MDS shows "replay" for a while and then it crashes in the rejoin phase with: -92> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 handle_mds_map i am now mds.0.501522 -91> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 handle_mds_map state change up:reconnect --> up:rejoin -90> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 rejoin_start -89> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 rejoin_joint_start -88> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfece err -22/0 -87> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -86> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -85> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -84> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -83> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -82> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -81> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ebd err -22/-22 -80> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ecd err -22/-22 -79> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9ea err -22/-22 -78> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -77> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9c3 err -22/-22 -76> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc978 err -22/-22 -75> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -74> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -73> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -72> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -71> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7e err -22/-22 -70> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -69> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -68> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -67> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -66> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ebd err -22/-22 -65> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ecd err -22/-22 -64> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9ea err -22/-22 -63> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc978 err -22/-22 -62> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x3000069373a err -22/-22 -61> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012dc5d8 err -22/-22 -60> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a32e8 err -22/-22 -59> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000696952 err -22/-22 -58> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -57> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -56> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -55> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -54> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7e err -22/-22 -53> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -52> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012c3a0e err -22/-22 -51> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -50> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -49> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7f err -22/-22 -48> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -47> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -46> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -45> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -44> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac2 err -22/-22 -43> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -42> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -41> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -40> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -39> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012c3a0e err -22/-22 -38> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -37> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x40000d63bff err -22/-22 -36> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -35> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -34> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -33> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac2 err -22/-22 -32> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -31> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac4 err -22/-22 -30> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -29> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -28> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -27> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -26> 2024-05-06T16:07:15.530+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -25> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -24> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -23> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -22> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -21> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a5fc4 err -22/-22 -20> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a634d err -22/-22 -19> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a63bf err -22/-22 -18> 2024-05-06T16:07:15.542+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -17> 2024-05-06T16:07:15.542+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -16> 2024-05-06T16:07:15.546+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -15> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -14> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -13> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a5fc4 err -22/-22 -12> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a634d err -22/-22 -11> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a63bf err -22/-22 -10> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bfd9c err -22/-22 -9> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bfb78 err -22/-22 -8> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -7> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -6> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -5> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -4> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -3> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -2> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x40000d5a226 err -22/-22 -1> 2024-05-06T16:07:15.634+0000 7f1921e91700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDCache.cc: In function 'void MDCache::rejoin_send_rejoins()' thread 7f1921e91700 time 2024-05-06T16:07:15.635683+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDCache.cc: 4086: FAILED ceph_assert(auth >= 0) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1930ad94a3] 2: /usr/lib64/ceph/libceph-common.so.2(+0x269669) [0x7f1930ad9669] 3: (MDCache::rejoin_send_rejoins()+0x216b) [0x5614ac8747eb] 4: (MDCache::process_imported_caps()+0x1993) [0x5614ac872353] 5: (Context::complete(int)+0xd) [0x5614ac6e182d] 6: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 7: (void finish_contexts<std::vector<MDSContext*, std::allocator<MDSContext*> > >(ceph::common::CephContext*, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int)+0x8d) [0x5614ac6e6f5d] 8: (OpenFileTable::_open_ino_finish(inodeno_t, int)+0x156) [0x5614aca765a6] 9: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 10: (void finish_contexts<std::vector<MDSContext*, std::allocator<MDSContext*> > >(ceph::common::CephContext*, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int)+0x8d) [0x5614ac6e6f5d] 11: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0x138) [0x5614ac867168] 12: (MDCache::_open_ino_backtrace_fetched(inodeno_t, ceph::buffer::v15_2_0::list&, int)+0x290) [0x5614ac87ff90] 13: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 14: (MDSIOContextBase::complete(int)+0x534) [0x5614aca426e4] 15: (Finisher::finisher_thread_entry()+0x18d) [0x7f1930b7884d] 16: /lib64/libpthread.so.0(+0x81ca) [0x7f192fac81ca] 17: clone() How do we solve this issue? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin