bdev_ioring -- true - Drives failing
Hello all Using 19.2.2, when enabling bdev_ioring and rebooting host then osd never comes online as below…. Why? Also Dasboard not matching with cli? Host output in dashboard… showing running. Tried failing manager too but it same.  Dashboard showing down  # ceph health detail HEALTH_WARN 4 osds down; 1 host (4 osds) down; Degraded data redundancy: 12998/38994 objects degraded (33.333%), 287 pgs degraded, 801 pgs undersized [WRN] OSD_DOWN: 4 osds down osd.4 (root=default,host= host07n) is down osd.5 (root=default,host= host07n) is down osd.6 (root=default,host= host07n) is down osd.7 (root=default,host= host07n) is down [WRN] OSD_HOST_DOWN: 1 host (4 osds) down host host07n (root=default) (4 osds) is down # ceph orch ps |grep -v running. ————> not showing anything down…. NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.4.service loaded activating auto-restart Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.5.service loaded activating auto-restart Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.6.service loaded activating auto-restart Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.7.service loaded activating auto-restart Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Came to running ….. # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.4.service loaded active running Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.5.service loaded active running Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.6.service loaded active running Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.7.service loaded active running Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Failed ….. # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.4.service loaded failed failed Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.5.service loaded failed failed Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.6.service loaded failed failed Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.7.service loaded failed failed Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Regards Dev
Could you share some of the OSD log, usually there some sort of assert or error message if it doesn’t want to come back up. Not too familiar with that config value but I’ve seen a bug in that version that causes osds to crash On Thu, Jul 24, 2025 at 10:52 AM Devender Singh <devender@netskrt.io> wrote:
Hello all
Using 19.2.2, when enabling bdev_ioring and rebooting host then osd never comes online as below…. Why?
Also Dasboard not matching with cli?
Host output in dashboard… showing running. Tried failing manager too but it same. 
Dashboard showing down 
# ceph health detail HEALTH_WARN 4 osds down; 1 host (4 osds) down; Degraded data redundancy: 12998/38994 objects degraded (33.333%), 287 pgs degraded, 801 pgs undersized [WRN] OSD_DOWN: 4 osds down osd.4 (root=default,host= host07n) is down osd.5 (root=default,host= host07n) is down osd.6 (root=default,host= host07n) is down osd.7 (root=default,host= host07n) is down [WRN] OSD_HOST_DOWN: 1 host (4 osds) down host host07n (root=default) (4 osds) is down
# ceph orch ps |grep -v running. ————> not showing anything down…. NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
# systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.4.service loaded activating auto-restart Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.5.service loaded activating auto-restart Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.6.service loaded activating auto-restart Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.7.service loaded activating auto-restart Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once
Came to running …..
# systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.4.service loaded active running Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.5.service loaded active running Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.6.service loaded active running Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.7.service loaded active running Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once
Failed …..
# systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.4.service loaded failed failed Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.5.service loaded failed failed Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.6.service loaded failed failed Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e@osd.7.service loaded failed failed Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once
Regards Dev _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your reply… Currently I disabled that option,, but below are the logs…. # zcat ceph-osd.*.log.1.gz |egrep -i "faile|error|down" 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.307+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:57:27.307+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.311+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:27.311+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.311+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:27.311+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.315+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:27.315+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.319+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:27.319+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.323+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:27.323+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.327+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:27.327+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.331+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:27.331+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.335+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:27.335+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.343+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:27.347+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:27.355+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:27.355+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:27.355+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:28.083+0000 7f2aeacb4680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:28.103+0000 7f2aeacb4680 0 bluestore(/var/lib/ceph/osd/ceph-4/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:28.487+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:28.487+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:29.559+0000 7f2aeacb4680 1 bluestore(/var/lib/ceph/osd/ceph-4/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.670384 seconds 2025-07-11T17:57:29.563+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:29.563+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:29.735+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:30.227+0000 7f2aeacb4680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:30.227+0000 7f2aeacb4680 0 bluestore(/var/lib/ceph/osd/ceph-4/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:30.379+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:30.379+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:30.695+0000 7f2aeacb4680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:57:30.755+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:30.755+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:31.639+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:44.583+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:57:44.587+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:44.587+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.591+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:44.591+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.591+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:44.591+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.595+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:44.595+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.599+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:44.599+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.607+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:44.607+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.611+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:44.611+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.615+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:44.615+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.623+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:44.623+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.627+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:44.635+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:44.647+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:57:44.647+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:44.647+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.187+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:45.187+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:45.267+0000 7f214047d680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:45.291+0000 7f214047d680 0 bluestore(/var/lib/ceph/osd/ceph-5/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:45.675+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:45.675+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:46.703+0000 7f214047d680 1 bluestore(/var/lib/ceph/osd/ceph-5/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.604180 seconds 2025-07-11T17:57:46.703+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:46.703+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:46.911+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:47.399+0000 7f214047d680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:47.399+0000 7f214047d680 0 bluestore(/var/lib/ceph/osd/ceph-5/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:47.579+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:47.579+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:47.871+0000 7f214047d680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:57:47.931+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:47.931+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:48.731+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:03.399+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.403+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:03.403+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:58:03.403+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:03.403+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.407+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:03.407+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.407+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:03.407+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.411+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:03.411+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.415+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:03.415+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.419+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:03.419+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.423+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:03.423+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.427+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:03.427+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.431+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:03.435+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:03.439+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:03.439+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:03.439+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:04.127+0000 7f082a781680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:04.147+0000 7f082a781680 0 bluestore(/var/lib/ceph/osd/ceph-6/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:04.459+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:04.459+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:05.243+0000 7f082a781680 1 bluestore(/var/lib/ceph/osd/ceph-6/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.553290 seconds 2025-07-11T17:58:05.243+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:05.243+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:05.295+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:05.783+0000 7f082a781680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:05.783+0000 7f082a781680 0 bluestore(/var/lib/ceph/osd/ceph-6/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:05.935+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:05.935+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:06.255+0000 7f082a781680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:58:06.315+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:06.315+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:07.251+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:26.279+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.283+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:58:26.283+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.283+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:26.283+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:26.287+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:26.287+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:26.291+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.291+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:26.291+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.295+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:26.295+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.295+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:26.299+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.299+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:26.299+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.303+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:26.307+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:26.311+0000 7f058d726680 1 freelist shutdown 2025-07-11T17:58:26.311+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:26.311+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:27.023+0000 7f058d726680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:27.039+0000 7f058d726680 0 bluestore(/var/lib/ceph/osd/ceph-7/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:27.347+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:27.347+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:28.127+0000 7f058d726680 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.565725 seconds 2025-07-11T17:58:28.127+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:28.131+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:28.179+0000 7f058d726680 1 freelist shutdown 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:28.655+0000 7f058d726680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:28.655+0000 7f058d726680 0 bluestore(/var/lib/ceph/osd/ceph-7/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:28.795+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:28.795+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:29.111+0000 7f058d726680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:58:29.171+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:29.171+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:30.015+0000 7f058d726680 1 freelist shutdown Regards Dev
On Jul 24, 2025, at 10:57 AM, Joshua Blanch <joshua.blanch@clyso.com> wrote:
r with that config value but I’ve seen a bug in that version that causes osds t
looks to be related to this issue. https://tracker.ceph.com/issues/66717 I just double checked squid 19.2.2 and it doesn't seem to have this fix listed in the pr. Although it seems to be present in 19.2.3. I'm not quite sure if there is a workaround for this in the meantime. On Thu, 24 Jul 2025 at 11:03, Devender Singh <devender@netskrt.io> wrote:
Thanks for your reply…
Currently I disabled that option,, but below are the logs….
# zcat ceph-osd.*.log.1.gz |egrep -i "faile|error|down" 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.307+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:57:27.307+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.311+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:27.311+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.311+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:27.311+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.315+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:27.315+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.319+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:27.319+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.323+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:27.323+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.327+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:27.327+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.331+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:27.331+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.335+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:27.335+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.343+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:27.347+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:27.355+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:27.355+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:27.355+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:28.083+0000 7f2aeacb4680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:28.103+0000 7f2aeacb4680 0 bluestore(/var/lib/ceph/osd/ceph-4/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:28.487+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:28.487+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:29.559+0000 7f2aeacb4680 1 bluestore(/var/lib/ceph/osd/ceph-4/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.670384 seconds 2025-07-11T17:57:29.563+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:29.563+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:29.735+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:30.227+0000 7f2aeacb4680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:30.227+0000 7f2aeacb4680 0 bluestore(/var/lib/ceph/osd/ceph-4/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:30.379+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:30.379+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:30.695+0000 7f2aeacb4680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:57:30.755+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:30.755+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:31.639+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:44.583+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:57:44.587+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:44.587+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.591+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:44.591+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.591+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:44.591+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.595+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:44.595+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.599+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:44.599+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.607+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:44.607+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.611+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:44.611+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.615+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:44.615+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.623+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:44.623+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.627+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:44.635+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:44.647+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:57:44.647+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:44.647+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.187+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:45.187+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:45.267+0000 7f214047d680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:45.291+0000 7f214047d680 0 bluestore(/var/lib/ceph/osd/ceph-5/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:45.675+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:45.675+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:46.703+0000 7f214047d680 1 bluestore(/var/lib/ceph/osd/ceph-5/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.604180 seconds 2025-07-11T17:57:46.703+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:46.703+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:46.911+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:47.399+0000 7f214047d680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:47.399+0000 7f214047d680 0 bluestore(/var/lib/ceph/osd/ceph-5/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:47.579+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:47.579+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:47.871+0000 7f214047d680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:57:47.931+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:47.931+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:48.731+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:03.399+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.403+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:03.403+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:58:03.403+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:03.403+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.407+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:03.407+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.407+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:03.407+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.411+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:03.411+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.415+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:03.415+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.419+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:03.419+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.423+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:03.423+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.427+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:03.427+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.431+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:03.435+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:03.439+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:03.439+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:03.439+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:04.127+0000 7f082a781680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:04.147+0000 7f082a781680 0 bluestore(/var/lib/ceph/osd/ceph-6/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:04.459+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:04.459+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:05.243+0000 7f082a781680 1 bluestore(/var/lib/ceph/osd/ceph-6/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.553290 seconds 2025-07-11T17:58:05.243+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:05.243+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:05.295+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:05.783+0000 7f082a781680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:05.783+0000 7f082a781680 0 bluestore(/var/lib/ceph/osd/ceph-6/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:05.935+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:05.935+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:06.255+0000 7f082a781680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:58:06.315+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:06.315+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:07.251+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:26.279+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.283+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:58:26.283+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.283+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:26.283+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:26.287+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:26.287+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:26.291+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.291+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:26.291+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.295+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:26.295+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.295+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:26.299+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.299+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:26.299+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.303+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:26.307+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:26.311+0000 7f058d726680 1 freelist shutdown 2025-07-11T17:58:26.311+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:26.311+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:27.023+0000 7f058d726680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:27.039+0000 7f058d726680 0 bluestore(/var/lib/ceph/osd/ceph-7/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:27.347+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:27.347+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:28.127+0000 7f058d726680 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.565725 seconds 2025-07-11T17:58:28.127+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:28.131+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:28.179+0000 7f058d726680 1 freelist shutdown 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:28.655+0000 7f058d726680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:28.655+0000 7f058d726680 0 bluestore(/var/lib/ceph/osd/ceph-7/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:28.795+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:28.795+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:29.111+0000 7f058d726680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:58:29.171+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:29.171+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:30.015+0000 7f058d726680 1 freelist shutdown
Regards Dev
On Jul 24, 2025, at 10:57 AM, Joshua Blanch <joshua.blanch@clyso.com> wrote:
r with that config value but I’ve seen a bug in that version that causes osds t
Hello Joshua I test after upgrading to 19.2.3, seems bdev_ioring -- true - not fixed in it.. It’s still failing Drives on rebooting the host they never come up. Regards Dev
On Jul 24, 2025, at 11:14 AM, Joshua Blanch <joshua.blanch@clyso.com> wrote:
looks to be related to this issue. https://tracker.ceph.com/issues/66717 <https://www.google.com/url?q=https://tracker.ceph.com/issues/66717&source=gmail-imap&ust=1753985674000000&usg=AOvVaw1dUKxlHPCWYJpwz0ZLzM7D>
I just double checked squid 19.2.2 and it doesn't seem to have this fix listed in the pr. Although it seems to be present in 19.2.3. I'm not quite sure if there is a workaround for this in the meantime.
On Thu, 24 Jul 2025 at 11:03, Devender Singh <devender@netskrt.io <mailto:devender@netskrt.io>> wrote:
Thanks for your reply…
Currently I disabled that option,, but below are the logs….
# zcat ceph-osd.*.log.1.gz |egrep -i "faile|error|down" 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:27.303+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:27.303+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.307+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:57:27.307+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.311+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:27.311+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.311+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:27.311+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.315+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:27.315+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.319+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:27.319+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.323+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:27.323+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.327+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:27.327+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.331+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:27.331+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.335+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:27.335+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.343+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:27.347+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:27.355+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:27.355+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:27.355+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:27.995+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:28.083+0000 7f2aeacb4680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:28.103+0000 7f2aeacb4680 0 bluestore(/var/lib/ceph/osd/ceph-4/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:28.487+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:28.487+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:28.783+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:29.559+0000 7f2aeacb4680 1 bluestore(/var/lib/ceph/osd/ceph-4/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.670384 seconds 2025-07-11T17:57:29.563+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:29.563+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:29.735+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:30.155+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:30.227+0000 7f2aeacb4680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:30.227+0000 7f2aeacb4680 0 bluestore(/var/lib/ceph/osd/ceph-4/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:30.379+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:30.379+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:30.687+0000 7f2aeacb4680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:30.695+0000 7f2aeacb4680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:57:30.755+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:30.755+0000 7f2aeacb4680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:31.639+0000 7f2aeacb4680 1 freelist shutdown 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:44.583+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:44.583+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:57:44.587+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:44.587+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.591+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:44.591+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.591+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:44.591+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.595+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:44.595+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.599+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:44.599+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.607+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:44.607+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.611+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:44.611+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.615+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:44.615+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.623+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:44.623+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:44.627+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:44.635+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:44.647+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:57:44.647+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:44.647+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.183+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:45.183+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.187+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:45.187+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:45.267+0000 7f214047d680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:45.291+0000 7f214047d680 0 bluestore(/var/lib/ceph/osd/ceph-5/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:45.675+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:45.675+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:45.991+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:45.991+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:46.703+0000 7f214047d680 1 bluestore(/var/lib/ceph/osd/ceph-5/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.604180 seconds 2025-07-11T17:57:46.703+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:46.703+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:46.911+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:47.327+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:47.327+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:47.399+0000 7f214047d680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:57:47.399+0000 7f214047d680 0 bluestore(/var/lib/ceph/osd/ceph-5/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:57:47.579+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:47.579+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.863+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:57:47.863+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:57:47.867+0000 7f214047d680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:57:47.867+0000 7f214047d680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:57:47.871+0000 7f214047d680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:57:47.931+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:57:47.931+0000 7f214047d680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:57:48.731+0000 7f214047d680 1 freelist shutdown 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:03.399+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:03.399+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.403+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:03.403+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:58:03.403+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:03.403+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.407+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:03.407+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.407+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:03.407+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.411+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:03.411+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.415+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:03.415+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.419+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:03.419+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.423+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:03.423+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.427+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:03.427+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:03.431+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:03.435+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:03.439+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:03.439+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:03.439+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:04.055+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:04.055+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:04.127+0000 7f082a781680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:04.147+0000 7f082a781680 0 bluestore(/var/lib/ceph/osd/ceph-6/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:04.459+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:04.459+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:04.583+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:04.583+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:05.243+0000 7f082a781680 1 bluestore(/var/lib/ceph/osd/ceph-6/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.553290 seconds 2025-07-11T17:58:05.243+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:05.243+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:05.295+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:05.711+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:05.711+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:05.783+0000 7f082a781680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:05.783+0000 7f082a781680 0 bluestore(/var/lib/ceph/osd/ceph-6/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:05.935+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:05.935+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:06.247+0000 7f082a781680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:06.247+0000 7f082a781680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:06.255+0000 7f082a781680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:58:06.315+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:06.315+0000 7f082a781680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:07.251+0000 7f082a781680 1 freelist shutdown 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:26.279+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:26.279+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.283+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2025-07-11T17:58:26.283+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.283+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:26.283+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:26.287+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:26.287+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.287+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:26.291+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.291+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:26.291+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.295+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:26.295+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.295+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:26.299+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.299+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:26.299+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.303+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:26.307+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:26.311+0000 7f058d726680 1 freelist shutdown 2025-07-11T17:58:26.311+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:26.311+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:26.943+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:26.943+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:27.023+0000 7f058d726680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:27.039+0000 7f058d726680 0 bluestore(/var/lib/ceph/osd/ceph-7/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:27.347+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:27.347+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:27.455+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:27.455+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:28.127+0000 7f058d726680 1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.565725 seconds 2025-07-11T17:58:28.127+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:28.131+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:28.179+0000 7f058d726680 1 freelist shutdown 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:28.579+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:28.583+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:28.583+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:28.655+0000 7f058d726680 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-07-11T17:58:28.655+0000 7f058d726680 0 bluestore(/var/lib/ceph/osd/ceph-7/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2025-07-11T17:58:28.795+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:28.795+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.error_if_exists: 0 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.allow_data_in_errors: 0 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2025-07-11T17:58:29.103+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.103+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2025-07-11T17:58:29.107+0000 7f058d726680 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2025-07-11T17:58:29.107+0000 7f058d726680 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2025-07-11T17:58:29.111+0000 7f058d726680 3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory 2025-07-11T17:58:29.171+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2025-07-11T17:58:29.171+0000 7f058d726680 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2025-07-11T17:58:30.015+0000 7f058d726680 1 freelist shutdown
Regards Dev
On Jul 24, 2025, at 10:57 AM, Joshua Blanch <joshua.blanch@clyso.com <mailto:joshua.blanch@clyso.com>> wrote:
r with that config value but I’ve seen a bug in that version that causes osds t
participants (2)
-
Devender Singh
-
Joshua Blanch