May 05 22:42:43 ceph06 bash[707251]: debug -1> 2024-05-05T20:42:43.006+0000 7f6892752700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDCache.cc: In function 'void MDCache::rejoin_send_rejoins()' thread 7f6892752700 time 2024-05-05T20:42:43.008448+0000 May 05 22:42:43 ceph06 bash[707251]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDCache.cc: 4086: FAILED ceph_assert(auth >= 0) May 05 22:42:43 ceph06 bash[707251]: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) May 05 22:42:43 ceph06 bash[707251]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f689fb974a3] May 05 22:42:43 ceph06 bash[707251]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x269669) [0x7f689fb97669] May 05 22:42:43 ceph06 bash[707251]: 3: (MDCache::rejoin_send_rejoins()+0x216b) [0x5605d03da7eb] May 05 22:42:43 ceph06 bash[707251]: 4: (MDCache::process_imported_caps()+0x1993) [0x5605d03d8353] May 05 22:42:43 ceph06 bash[707251]: 5: (MDCache::rejoin_open_ino_finish(inodeno_t, int)+0x217) [0x5605d03e5837] May 05 22:42:43 ceph06 bash[707251]: 6: (MDSContext::complete(int)+0x5f) [0x5605d05a7f4f] May 05 22:42:43 ceph06 bash[707251]: 7: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x5605d024cf5d] May 05 22:42:43 ceph06 bash[707251]: 8: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0x138) [0x5605d03cd168] May 05 22:42:43 ceph06 bash[707251]: 9: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbb) [0x5605d03cd4bb] May 05 22:42:43 ceph06 bash[707251]: 10: (MDSContext::complete(int)+0x5f) [0x5605d05a7f4f] May 05 22:42:43 ceph06 bash[707251]: 11: (MDSRank::_advance_queues()+0xaa) [0x5605d025b34a] May 05 22:42:43 ceph06 bash[707251]: 12: (MDSRank::ProgressThread::entry()+0xb8) [0x5605d025b918] May 05 22:42:43 ceph06 bash[707251]: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f689eb861ca] May 05 22:42:43 ceph06 bash[707251]: 14: clone() May 05 22:42:43 ceph06 bash[707251]: debug 0> 2024-05-05T20:42:43.010+0000 7f6892752700 -1 *** Caught signal (Aborted) ** May 05 22:42:43 ceph06 bash[707251]: in thread 7f6892752700 thread_name:mds_rank_progr May 05 22:42:43 ceph06 bash[707251]: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) May 05 22:42:43 ceph06 bash[707251]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f689eb90cf0] May 05 22:42:43 ceph06 bash[707251]: 2: gsignal() May 05 22:42:43 ceph06 bash[707251]: 3: abort() May 05 22:42:43 ceph06 bash[707251]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f689fb974fd] May 05 22:42:43 ceph06 bash[707251]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x269669) [0x7f689fb97669] May 05 22:42:43 ceph06 bash[707251]: 6: (MDCache::rejoin_send_rejoins()+0x216b) [0x5605d03da7eb] May 05 22:42:43 ceph06 bash[707251]: 7: (MDCache::process_imported_caps()+0x1993) [0x5605d03d8353] May 05 22:42:43 ceph06 bash[707251]: 8: (MDCache::rejoin_open_ino_finish(inodeno_t, int)+0x217) [0x5605d03e5837] May 05 22:42:43 ceph06 bash[707251]: 9: (MDSContext::complete(int)+0x5f) [0x5605d05a7f4f] May 05 22:42:43 ceph06 bash[707251]: 10: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x5605d024cf5d] May 05 22:42:43 ceph06 bash[707251]: 11: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0x138) [0x5605d03cd168] May 05 22:42:43 ceph06 bash[707251]: 12: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbb) [0x5605d03cd4bb] May 05 22:42:43 ceph06 bash[707251]: 13: (MDSContext::complete(int)+0x5f) [0x5605d05a7f4f] May 05 22:42:43 ceph06 bash[707251]: 14: (MDSRank::_advance_queues()+0xaa) [0x5605d025b34a] May 05 22:42:43 ceph06 bash[707251]: 15: (MDSRank::ProgressThread::entry()+0xb8) [0x5605d025b918] May 05 22:42:43 ceph06 bash[707251]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f689eb861ca] May 05 22:42:43 ceph06 bash[707251]: 17: clone() May 05 22:42:43 ceph06 bash[707251]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. May 05 22:42:43 ceph06 bash[707251]: --- logging levels --- May 05 22:42:43 ceph06 bash[707251]: 0/ 5 none May 05 22:42:43 ceph06 bash[707251]: 0/ 1 lockdep May 05 22:42:43 ceph06 bash[707251]: 0/ 1 context May 05 22:42:43 ceph06 bash[707251]: 1/ 1 crush May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mds May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mds_balancer May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mds_locker May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mds_log May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mds_log_expire May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mds_migrator May 05 22:42:43 ceph06 bash[707251]: 0/ 1 buffer May 05 22:42:43 ceph06 bash[707251]: 0/ 1 timer May 05 22:42:43 ceph06 bash[707251]: 0/ 1 filer May 05 22:42:43 ceph06 bash[707251]: 0/ 1 striper May 05 22:42:43 ceph06 bash[707251]: 0/ 1 objecter May 05 22:42:43 ceph06 bash[707251]: 0/ 5 rados May 05 22:42:43 ceph06 bash[707251]: 0/ 5 rbd May 05 22:42:43 ceph06 bash[707251]: 0/ 5 rbd_mirror May 05 22:42:43 ceph06 bash[707251]: 0/ 5 rbd_replay May 05 22:42:43 ceph06 bash[707251]: 0/ 5 rbd_pwl May 05 22:42:43 ceph06 bash[707251]: 0/ 5 journaler May 05 22:42:43 ceph06 bash[707251]: 0/ 5 objectcacher May 05 22:42:43 ceph06 bash[707251]: 0/ 5 immutable_obj_cache May 05 22:42:43 ceph06 bash[707251]: 0/ 5 client May 05 22:42:43 ceph06 bash[707251]: 1/ 5 osd May 05 22:42:43 ceph06 bash[707251]: 0/ 5 optracker May 05 22:42:43 ceph06 bash[707251]: 0/ 5 objclass May 05 22:42:43 ceph06 bash[707251]: 1/ 3 filestore May 05 22:42:43 ceph06 bash[707251]: 1/ 3 journal May 05 22:42:43 ceph06 bash[707251]: 0/ 0 ms May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mon May 05 22:42:43 ceph06 bash[707251]: 0/10 monc May 05 22:42:43 ceph06 bash[707251]: 1/ 5 paxos May 05 22:42:43 ceph06 bash[707251]: 0/ 5 tp May 05 22:42:43 ceph06 bash[707251]: 1/ 5 auth May 05 22:42:43 ceph06 bash[707251]: 1/ 5 crypto May 05 22:42:43 ceph06 bash[707251]: 1/ 1 finisher May 05 22:42:43 ceph06 bash[707251]: 1/ 1 reserver May 05 22:42:43 ceph06 bash[707251]: 1/ 5 heartbeatmap May 05 22:42:43 ceph06 bash[707251]: 1/ 5 perfcounter May 05 22:42:43 ceph06 bash[707251]: 1/ 2 rgw May 05 22:42:43 ceph06 bash[707251]: 1/ 5 rgw_sync May 05 22:42:43 ceph06 bash[707251]: 1/ 5 rgw_datacache May 05 22:42:43 ceph06 bash[707251]: 1/10 civetweb May 05 22:42:43 ceph06 bash[707251]: 1/ 5 javaclient May 05 22:42:43 ceph06 bash[707251]: 1/ 5 asok May 05 22:42:43 ceph06 bash[707251]: 1/ 1 throttle May 05 22:42:43 ceph06 bash[707251]: 0/ 0 refs May 05 22:42:43 ceph06 bash[707251]: 1/ 5 compressor May 05 22:42:43 ceph06 bash[707251]: 1/ 5 bluestore May 05 22:42:43 ceph06 bash[707251]: 1/ 5 bluefs May 05 22:42:43 ceph06 bash[707251]: 1/ 3 bdev May 05 22:42:43 ceph06 bash[707251]: 1/ 5 kstore May 05 22:42:43 ceph06 bash[707251]: 4/ 5 rocksdb May 05 22:42:43 ceph06 bash[707251]: 4/ 5 leveldb May 05 22:42:43 ceph06 bash[707251]: 4/ 5 memdb May 05 22:42:43 ceph06 bash[707251]: 1/ 5 fuse May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mgr May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mgrc May 05 22:42:43 ceph06 bash[707251]: 1/ 5 dpdk May 05 22:42:43 ceph06 bash[707251]: 1/ 5 eventtrace May 05 22:42:43 ceph06 bash[707251]: 1/ 5 prioritycache May 05 22:42:43 ceph06 bash[707251]: 0/ 5 test May 05 22:42:43 ceph06 bash[707251]: 0/ 5 cephfs_mirror May 05 22:42:43 ceph06 bash[707251]: 0/ 5 cephsqlite May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_onode May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_odata May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_omap May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_tm May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_cleaner May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_lba May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_cache May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_journal May 05 22:42:43 ceph06 bash[707251]: 0/ 5 seastore_device May 05 22:42:43 ceph06 bash[707251]: 0/ 5 alienstore May 05 22:42:43 ceph06 bash[707251]: 1/ 5 mclock May 05 22:42:43 ceph06 bash[707251]: 1/ 5 ceph_exporter May 05 22:42:43 ceph06 bash[707251]: -2/-2 (syslog threshold) May 05 22:42:43 ceph06 bash[707251]: 99/99 (stderr threshold) May 05 22:42:43 ceph06 bash[707251]: --- pthread ID / name mapping for recent threads --- May 05 22:42:43 ceph06 bash[707251]: 7f688f74c700 / May 05 22:42:43 ceph06 bash[707251]: 7f689074e700 / May 05 22:42:43 ceph06 bash[707251]: 7f6890f4f700 / MR_Finisher May 05 22:42:43 ceph06 bash[707251]: 7f6891f51700 / PQ_Finisher May 05 22:42:43 ceph06 bash[707251]: 7f6892752700 / mds_rank_progr May 05 22:42:43 ceph06 bash[707251]: 7f6892f53700 / ms_dispatch May 05 22:42:43 ceph06 bash[707251]: 7f6894f57700 / ceph-mds May 05 22:42:43 ceph06 bash[707251]: 7f6895758700 / safe_timer May 05 22:42:43 ceph06 bash[707251]: 7f6895f59700 / safe_timer May 05 22:42:43 ceph06 bash[707251]: 7f6896f5b700 / ms_dispatch May 05 22:42:43 ceph06 bash[707251]: 7f6897f5d700 / io_context_pool May 05 22:42:43 ceph06 bash[707251]: 7f6898f5f700 / admin_socket May 05 22:42:43 ceph06 bash[707251]: 7f6899760700 / msgr-worker-2 May 05 22:42:43 ceph06 bash[707251]: 7f6899f61700 / msgr-worker-1 May 05 22:42:43 ceph06 bash[707251]: 7f689a762700 / msgr-worker-0 May 05 22:42:43 ceph06 bash[707251]: 7f68a0cd4ac0 / ceph-mds May 05 22:42:43 ceph06 bash[707251]: max_recent 10000 May 05 22:42:43 ceph06 bash[707251]: max_new 10000 May 05 22:42:43 ceph06 bash[707251]: log_file /var/lib/ceph/crash/2024-05-05T20:42:43.014159Z_b6ad6bc1-0faa-4a78-8cb0-c004f051c7d6/log May 05 22:42:43 ceph06 bash[707251]: --- end dump of recent events ---