--- begin dump of recent events --- -250> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command assert hook 0x555d1f85cb30 -249> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command abort hook 0x555d1f85cb30 -248> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command leak_some_memory hook 0x555d1f85cb30 -247> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perfcounters_dump hook 0x555d1f85cb30 -246> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command 1 hook 0x555d1f85cb30 -245> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf dump hook 0x555d1f85cb30 -244> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perfcounters_schema hook 0x555d1f85cb30 -243> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf histogram dump hook 0x555d1f85cb30 -242> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command 2 hook 0x555d1f85cb30 -241> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf schema hook 0x555d1f85cb30 -240> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command counter dump hook 0x555d1f85cb30 -239> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command counter schema hook 0x555d1f85cb30 -238> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf histogram schema hook 0x555d1f85cb30 -237> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf reset hook 0x555d1f85cb30 -236> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config show hook 0x555d1f85cb30 -235> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config help hook 0x555d1f85cb30 -234> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config set hook 0x555d1f85cb30 -233> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config unset hook 0x555d1f85cb30 -232> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config get hook 0x555d1f85cb30 -231> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config diff hook 0x555d1f85cb30 -230> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config diff get hook 0x555d1f85cb30 -229> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command injectargs hook 0x555d1f85cb30 -228> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command log flush hook 0x555d1f85cb30 -227> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command log dump hook 0x555d1f85cb30 -226> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command log reopen hook 0x555d1f85cb30 -225> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command dump_mempools hook 0x555d20602068 -224> 2023-12-22T21:26:57.860+0100 7f5158282a00 5 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_db_and_around::NCB::read_only=0, to_repair=1 -223> 2023-12-22T21:26:57.864+0100 7f5158282a00 1 bdev(0x555d1f8c2000 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block -222> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2000 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open size 8001561821184 (0x74702400000, 7.3 TiB) block_size 4096 (4 KiB) rotational device, discard not supported -221> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 -220> 2023-12-22T21:26:57.868+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore bluefs device info hook 0x555d1f85cb00 -219> 2023-12-22T21:26:57.868+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs stats hook 0x555d1f85cb00 -218> 2023-12-22T21:26:57.868+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs files list hook 0x555d1f85cb00 -217> 2023-12-22T21:26:57.868+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs debug_inject_read_zeros hook 0x555d1f85cb00 -216> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db -215> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) open size 68719476736 (0x1000000000, 64 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported -214> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db size 64 GiB -213> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block -212> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open size 8001561821184 (0x74702400000, 7.3 TiB) block_size 4096 (4 KiB) rotational device, discard not supported -211> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bluefs add_block_device bdev 2 path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block size 7.3 TiB -210> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0 -209> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824 -208> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8 -207> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4 -206> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216 -205> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel -204> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB -203> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8 -202> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64 -201> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824 -200> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6 -199> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression -198> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 bluefs mount -197> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 bluefs _init_alloc new, id 1, allocator name bluefs-db, allocator type hybrid, capacity 0x1000000000, block size 0x100000 -196> 2023-12-22T21:26:57.872+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump bluefs-db hook 0x555d1f970db0 -195> 2023-12-22T21:26:57.872+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score bluefs-db hook 0x555d1f970db0 -194> 2023-12-22T21:26:57.872+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation bluefs-db hook 0x555d1f970db0 -193> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 bluefs _init_alloc shared, id 2, capacity 0x74702400000, block size 0x10000 -192> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 bluefs mount shared_bdev_used = 0 -191> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _prepare_db_environment set db_paths to db,65283502899 db.slow,7601483730124 -190> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0 -189> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824 -188> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8 -187> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4 -186> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216 -185> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel -184> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB -183> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8 -182> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64 -181> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824 -180> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6 -179> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression -178> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0 -177> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824 -176> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8 -175> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4 -174> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216 -173> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel -172> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB -171> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8 -170> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64 -169> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824 -168> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6 -167> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression -166> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) -165> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) -164> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) -163> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) -162> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) -161> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) -160> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) -159> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) -158> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) -157> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) -156> 2023-12-22T21:26:58.036+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) -155> 2023-12-22T21:26:58.036+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) -154> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 -153> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta old nid_max 2302905 -152> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta old blobid_max 430253 -151> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta::NCB::freelist_type=null -150> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 -149> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta min_alloc_size 0x1000 -148> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 get compressor snappy = 0 -147> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 load compressor snappy -146> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 add compressor snappy 0x555d1fd40330 -145> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 get compressor snappy = 0x555d1fd40330 -144> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 load: compressor snappy loaded and registered -143> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 get compressor snappy = 0x555d1fd40330 -142> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_fm::NCB::freelist_type=null -141> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 freelist init -140> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 freelist _read_cfg -139> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump block hook 0x555d1fd418c0 -138> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score block hook 0x555d1fd418c0 -137> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation block hook 0x555d1fd418c0 -136> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump recovery hook 0x555d1fd41ce0 -135> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score recovery hook 0x555d1fd41ce0 -134> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation recovery hook 0x555d1fd41ce0 -133> 2023-12-22T21:26:58.168+0100 7f5158282a00 5 bluestore::NCB::__restore_allocator::file_size=0,sizeof(extent_t)=16 -132> 2023-12-22T21:26:58.168+0100 7f5158282a00 1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) -131> 2023-12-22T21:26:58.168+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator dump recovery -130> 2023-12-22T21:26:58.168+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator fragmentation recovery -129> 2023-12-22T21:26:58.168+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator score recovery -128> 2023-12-22T21:26:58.168+0100 7f5158282a00 0 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... -127> 2023-12-22T21:27:10.868+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 1000000 -126> 2023-12-22T21:27:23.400+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 2000000 -125> 2023-12-22T21:27:35.828+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 3000000 -124> 2023-12-22T21:27:48.348+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 4000000 -123> 2023-12-22T21:28:00.904+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 5000000 -122> 2023-12-22T21:28:13.621+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 6000000 -121> 2023-12-22T21:28:26.037+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 7000000 -120> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 1->store_statfs(0x0/0x0/0x0, data 0x1b9768318a4/0x1b977f9b000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -119> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 3->store_statfs(0x0/0x0/0x0, data 0x19400000/0x19400000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -118> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 4->store_statfs(0x0/0x0/0x0, data 0x95c6800/0x95c7000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -117> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 5->store_statfs(0x0/0x0/0x0, data 0x3c00000/0x3c00000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -116> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 6->store_statfs(0x0/0x0/0x0, data 0x6761298925/0x6782476000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -115> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 7->store_statfs(0x0/0x0/0x0, data 0x17b62e4/0x17b9000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -114> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 8->store_statfs(0x0/0x0/0x0, data 0x5a033600/0x5a036000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -113> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 9->store_statfs(0x0/0x0/0x0, data 0x2822d21000/0x2822d25000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -112> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool a->store_statfs(0x0/0x0/0x0, data 0x0/0x0, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -111> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool b->store_statfs(0x0/0x0/0x0, data 0x1f80da012/0x1f80e3000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -110> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool c->store_statfs(0x0/0x0/0x0, data 0x18d7400/0x18da000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -109> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool d->store_statfs(0x0/0x0/0x0, data 0x0/0x0, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -108> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool e->store_statfs(0x0/0x0/0x0, data 0x14a/0x3000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -107> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 10->store_statfs(0x0/0x0/0x0, data 0x0/0x0, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -106> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool ffffffffffffffff->store_statfs(0x0/0x0/0x0, data 0x57a83eb/0x6113000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1 -105> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered store_statfs(0x0/0x0/0x0, data 0x24b7b6f4bf4/0x24b9e9bf000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) -104> 2023-12-22T21:28:29.205+0100 7f5158282a00 1 HybridAllocator _spillover_range constructing fallback allocator -103> 2023-12-22T21:28:29.205+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump block.fallback hook 0x555d1fd6b0e0 -102> 2023-12-22T21:28:29.205+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score block.fallback hook 0x555d1fd6b0e0 -101> 2023-12-22T21:28:29.205+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation block.fallback hook 0x555d1fd6b0e0 -100> 2023-12-22T21:28:30.741+0100 7f5158282a00 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 92.570249 seconds, extent_count=38970352 -99> 2023-12-22T21:28:30.741+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x74702400000, block size 0x1000, free 0x4fb63a3f000, fragmentation 0.00727618 -98> 2023-12-22T21:28:30.817+0100 7f5158282a00 1 bluefs umount -97> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator dump bluefs-db -96> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator fragmentation bluefs-db -95> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator score bluefs-db -94> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluefs debug_inject_read_zeros -93> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluefs files list -92> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluefs stats -91> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore bluefs device info -90> 2023-12-22T21:28:30.817+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) close -89> 2023-12-22T21:28:31.293+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) close -88> 2023-12-22T21:28:31.449+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore bluefs device info hook 0x555d1f85ca90 -87> 2023-12-22T21:28:31.449+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs stats hook 0x555d1f85ca90 -86> 2023-12-22T21:28:31.449+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs files list hook 0x555d1f85ca90 -85> 2023-12-22T21:28:31.449+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs debug_inject_read_zeros hook 0x555d1f85ca90 -84> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db -83> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) open size 68719476736 (0x1000000000, 64 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported -82> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db size 64 GiB -81> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block -80> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open size 8001561821184 (0x74702400000, 7.3 TiB) block_size 4096 (4 KiB) rotational device, discard not supported -79> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bluefs add_block_device bdev 2 path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block size 7.3 TiB -78> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0 -77> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824 -76> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8 -75> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4 -74> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216 -73> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel -72> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB -71> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8 -70> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64 -69> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824 -68> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6 -67> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression -66> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 bluefs mount -65> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 bluefs _init_alloc new, id 1, allocator name bluefs-db, allocator type hybrid, capacity 0x1000000000, block size 0x100000 -64> 2023-12-22T21:28:31.457+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump bluefs-db hook 0x555d1f970fc0 -63> 2023-12-22T21:28:31.457+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score bluefs-db hook 0x555d1f970fc0 -62> 2023-12-22T21:28:31.457+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation bluefs-db hook 0x555d1f970fc0 -61> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 bluefs _init_alloc shared, id 2, capacity 0x74702400000, block size 0x10000 -60> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 bluefs mount shared_bdev_used = 0 -59> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _prepare_db_environment set db_paths to db,65283502899 db.slow,7601483730124 -58> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0 -57> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824 -56> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8 -55> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4 -54> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216 -53> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel -52> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB -51> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8 -50> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64 -49> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824 -48> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6 -47> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression -46> 2023-12-22T21:28:31.521+0100 7f5158282a00 5 bluestore::NCB::invalidate_allocation_file_on_bluefs::invalidate using bluefs->truncate(p_handle, 0) -45> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0 -44> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824 -43> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8 -42> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4 -41> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216 -40> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel -39> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB -38> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8 -37> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64 -36> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824 -35> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6 -34> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression -33> 2023-12-22T21:28:31.529+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) -32> 2023-12-22T21:28:31.529+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) -31> 2023-12-22T21:28:31.533+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) -30> 2023-12-22T21:28:32.957+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) -29> 2023-12-22T21:28:35.705+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) -28> 2023-12-22T21:28:35.717+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) -27> 2023-12-22T21:28:44.209+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) -26> 2023-12-22T21:28:52.773+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) -25> 2023-12-22T21:29:01.313+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) -24> 2023-12-22T21:29:01.485+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0 -23> 2023-12-22T21:29:01.485+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824 -22> 2023-12-22T21:29:01.485+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8 -21> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4 -20> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216 -19> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel -18> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB -17> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8 -16> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64 -15> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824 -14> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6 -13> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression -12> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) -11> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) -10> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) -9> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) -8> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) -7> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) -6> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) -5> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) -4> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) -3> 2023-12-22T21:29:01.561+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 12, name: m-0) -2> 2023-12-22T21:29:01.577+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 13, name: m-1) -1> 2023-12-22T21:29:01.593+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 14, name: m-2) 0> 2023-12-22T21:29:01.689+0100 7f5158282a00 -1 *** Caught signal (Segmentation fault) ** in thread 7f5158282a00 thread_name:ceph-kvstore-to ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0) [0x7f515956bfd0] 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb9) [0x555d1e4d2a19] 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x19f) [0x555d1e4d278f] 4: (BlueStore::_close_db()+0x14f) [0x555d1e39ee5f] 5: (BlueStore::_close_db_and_around()+0x13) [0x555d1e39fa03] 6: (BlueStore::umount()+0xdf) [0x555d1e3a548f] 7: (std::unique_ptr::~unique_ptr()+0x1e) [0x555d1e2f8d0e] 8: main() 9: /lib/x86_64-linux-gnu/libc.so.6(+0x271ca) [0x7f51595571ca] 10: __libc_start_main() 11: _start() NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/ 5 rgw_access 1/ 5 rgw_dbstore 1/ 5 rgw_flight 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 2/ 2 rocksdb 4/ 5 leveldb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_t 0/ 5 seastore_cleaner 0/ 5 seastore_epm 0/ 5 seastore_lba 0/ 5 seastore_fixedkv_tree 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 seastore_backref 0/ 5 alienstore 1/ 5 mclock 0/ 5 cyanstore 1/ 5 ceph_exporter 1/ 5 memstore -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7f5158282a00 / ceph-kvstore-to max_recent 500 max_new 1000 log_file /var/lib/ceph/crash/2023-12-22T20:29:01.692109Z_6d62fc0a-38a3-4725-9c37-5cbf70d0ee5f/log --- end dump of recent events ---