- ceph-users - lists.ceph.io

ceph df reports incorrect stats
by Frank Schilder 06 Dec '23

06 Dec '23

Dear fellow cephers, we got a problem with ceph df: ceph df reports incorrect USED. It would be great if someone could look at this, if a ceph operator doesn't discover this issue, they might run out of space without noticing. This has been reported before but didn't get much attention: https://www.spinics.net/lists/ceph-users/msg74602.html https://www.spinics.net/lists/ceph-users/msg74630.html The symptom: STORED=USED in output of ceph df. All reports I know of are for octopus clusters, but I suspect newer versions are affected as well. I don't have a reproducer yet (still lacking a test cluster). Here is a correct usage report: ==> logs/health_231203.log <== --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.29 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 20.95 3.7 PiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 42.78 3.3 PiB Here is an incorrect one: ==> logs/health_231204.log <== --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.06 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.82 3.6 PiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.09 3.3 PiB That the first report is correct and not the second is supported by the output of ceph osd df tree, showing a use of 4.6PB in alignment with the first output of ceph df. Note that the date of the ceph osd df tree output is identical to the date of the incorrect ceph df output, hence, ceph osd df tree is *not* affected by this issue: ==> ceph osd df tree 231204 <=== SIZE RAW USE DATA OMAP META AVAIL NAME 12 PiB 4.6 PiB 4.6 PiB 2.2 TiB 19 TiB 7.5 PiB datacenter ContainerSquare 0 B 0 B 0 B 0 B 0 B 0 B room CON-161-A 12 PiB 4.6 PiB 4.6 PiB 2.2 TiB 19 TiB 7.5 PiB room CON-161-A1 In our case, the problem showed up out of nowhere. Here the log snippet for the time window within which the flip happened (compare the lines for con-fs2-data?-pools): ==> logs/health_231203.log <== ceph status/df/pool stats/health detail at 16:30:03: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 14h), 1279 in (since 2w) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.1 PiB / 13 PiB avail pgs: 25035 active+clean 29 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 215 MiB/s rd, 140 MiB/s wr, 2.34k op/s rd, 1.89k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 831 GiB 6.0 TiB 11.84 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.08 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.44 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.60 TOTAL 13 PiB 8.1 PiB 4.9 PiB 5.0 PiB 38.04 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 13 GiB 16.57k 38 GiB 0.03 39 TiB sr-rbd-data-one 2 4096 121 TiB 32.06M 108 TiB 48.08 88 TiB sr-rbd-one-stretch 3 160 262 GiB 68.81k 573 GiB 0.48 39 TiB con-rbd-meta-hpc-one 7 50 12 KiB 45 372 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 24 GiB 6.10k 24 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 137 TiB 35.95M 193 TiB 46.57 166 TiB con-fs2-meta1 12 512 554 GiB 76.76M 2.2 TiB 7.26 6.9 TiB con-fs2-meta2 13 4096 0 B 574.23M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 21.09 3.6 PiB con-fs2-data-ec-ssd 17 256 700 GiB 7.27M 706 GiB 2.44 22 TiB ms-rbd-one 18 256 805 GiB 210.92k 1.4 TiB 1.18 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 42.96 3.3 PiB sr-rbd-data-one-perf 20 4096 6.8 TiB 1.81M 20 TiB 57.09 5.1 TiB device_health_metrics 21 1 1.4 GiB 1.11k 4.2 GiB 0 39 TiB ceph status/df/pool stats/health detail at 16:30:10: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 14h), 1279 in (since 2w) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.1 PiB / 13 PiB avail pgs: 25035 active+clean 29 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 241 MiB/s rd, 174 MiB/s wr, 2.68k op/s rd, 2.34k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 830 GiB 6.0 TiB 11.84 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.08 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.44 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.60 TOTAL 13 PiB 8.1 PiB 4.9 PiB 5.0 PiB 38.04 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 13 GiB 16.57k 13 GiB 0.01 39 TiB sr-rbd-data-one 2 4096 92 TiB 32.06M 92 TiB 44.11 88 TiB sr-rbd-one-stretch 3 160 222 GiB 68.81k 222 GiB 0.19 39 TiB con-rbd-meta-hpc-one 7 50 6.9 KiB 45 6.9 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 23 GiB 6.10k 23 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 135 TiB 35.95M 135 TiB 37.88 166 TiB con-fs2-meta1 12 512 367 GiB 76.76M 367 GiB 1.28 6.9 TiB con-fs2-meta2 13 4096 0 B 574.23M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.82 3.6 PiB con-fs2-data-ec-ssd 17 256 515 GiB 7.27M 515 GiB 1.79 22 TiB ms-rbd-one 18 256 579 GiB 210.92k 579 GiB 0.48 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.09 3.3 PiB sr-rbd-data-one-perf 20 4096 6.9 TiB 1.81M 6.9 TiB 31.29 5.1 TiB device_health_metrics 21 1 1.2 GiB 1.11k 1.2 GiB 0 39 TiB For us, the issue disappeared after taking some OSDs in a second root down. These OSDs were moved there for draining, we use a second crush root for this purpose. Here the log snippet with the time window within which the back-flip to correct reporting happened: ==> logs/health_231205.log <== ceph status/df/pool stats/health detail at 17:42:58: cluster: health: HEALTH_WARN 1 osds down 24 hosts (12 osds) down 1 root (12 osds) down services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1267 up (since 19m), 1268 in (since 0.401448s) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.0 PiB / 13 PiB avail pgs: 25034 active+clean 31 active+clean+scrubbing+deep io: client: 118 MiB/s rd, 789 MiB/s wr, 1.75k op/s rd, 2.14k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 731 GiB 5.9 TiB 11.65 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.36 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.59 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.19 TOTAL 13 PiB 8.0 PiB 4.9 PiB 5.0 PiB 38.32 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 14 GiB 16.94k 14 GiB 0.01 39 TiB sr-rbd-data-one 2 4096 93 TiB 32.32M 93 TiB 44.29 88 TiB sr-rbd-one-stretch 3 160 222 GiB 68.81k 222 GiB 0.19 39 TiB con-rbd-meta-hpc-one 7 50 6.9 KiB 45 6.9 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 23 GiB 6.10k 23 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 135 TiB 36.08M 135 TiB 38.00 165 TiB con-fs2-meta1 12 512 367 GiB 76.81M 367 GiB 1.28 6.9 TiB con-fs2-meta2 13 4096 0 B 572.65M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.83 3.6 PiB con-fs2-data-ec-ssd 17 256 515 GiB 7.27M 515 GiB 1.78 22 TiB ms-rbd-one 18 256 579 GiB 210.92k 579 GiB 0.48 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.16 3.3 PiB sr-rbd-data-one-perf 20 4096 6.9 TiB 1.81M 6.9 TiB 31.07 5.1 TiB device_health_metrics 21 1 1.2 GiB 1.11k 1.2 GiB 0 39 TiB ceph status/df/pool stats/health detail at 17:43:04: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1267 up (since 19m), 1267 in (since 6s) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.0 PiB / 13 PiB avail pgs: 25035 active+clean 30 active+clean+scrubbing+deep io: client: 151 MiB/s rd, 840 MiB/s wr, 2.13k op/s rd, 2.10k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 731 GiB 5.9 TiB 11.65 hdd 13 PiB 7.7 PiB 4.8 PiB 4.8 PiB 38.42 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.59 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.19 TOTAL 13 PiB 8.0 PiB 4.9 PiB 5.0 PiB 38.37 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 14 GiB 16.94k 42 GiB 0.04 39 TiB sr-rbd-data-one 2 4096 122 TiB 32.32M 109 TiB 48.26 88 TiB sr-rbd-one-stretch 3 160 262 GiB 68.81k 573 GiB 0.48 39 TiB con-rbd-meta-hpc-one 7 50 11 KiB 45 368 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 24 GiB 6.10k 24 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 138 TiB 36.08M 193 TiB 46.69 165 TiB con-fs2-meta1 12 512 555 GiB 76.81M 2.2 TiB 7.26 6.9 TiB con-fs2-meta2 13 4096 0 B 572.65M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 21.09 3.6 PiB con-fs2-data-ec-ssd 17 256 700 GiB 7.27M 706 GiB 2.43 22 TiB ms-rbd-one 18 256 805 GiB 210.92k 1.4 TiB 1.18 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 43.01 3.3 PiB sr-rbd-data-one-perf 20 4096 6.8 TiB 1.81M 20 TiB 56.75 5.1 TiB device_health_metrics 21 1 1.4 GiB 1.11k 4.2 GiB 0 39 TiB This observation leads me to suspect that having multiple crush roots might be a reason for this observation. Our crush tree looks like this (OSDs removed), it has 3 different roots (BB, DTU and default): ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -78 106.92188 root BB -99 0 host bb-04 -102 0 host bb-05 -105 0 host bb-06 -325 0 host bb-06-old -108 0 host bb-07 -331 0 host bb-07-old -3 8.91016 host bb-08 -9 8.91016 host bb-09 -18 8.91016 host bb-10 -21 8.91016 host bb-11 -28 8.91016 host bb-12 -34 8.91016 host bb-13 -72 8.91016 host bb-14 -75 8.91016 host bb-15 -111 8.91016 host bb-16 -114 8.91016 host bb-17 -117 0 host bb-18 -142 0 host bb-19 -145 0 host bb-20 -241 0 host bb-21 -246 0 host bb-22 -251 8.91016 host bb-23 -256 8.91016 host bb-24 -151 0 host bb-office -40 14614.77832 root DTU -42 0 region Lyngby -41 14614.77832 region Risoe -50 12843.79590 datacenter ContainerSquare -56 0 room CON-161-A -57 12843.79590 room CON-161-A1 -11 1092.49060 host ceph-08 -13 1074.27673 host ceph-09 -23 1075.67920 host ceph-10 -15 1067.16492 host ceph-11 -25 1080.21912 host ceph-12 -83 1061.17480 host ceph-13 -85 1047.70276 host ceph-14 -87 1079.02820 host ceph-15 -136 1012.55048 host ceph-16 -139 1073.61475 host ceph-17 -261 1125.57202 host ceph-23 -262 1054.32227 host ceph-24 -148 885.49133 datacenter MultiSite -65 86.16304 host ceph-04 -67 101.50623 host ceph-05 -69 104.85805 host ceph-06 -71 96.39923 host ceph-07 -81 97.54230 host ceph-18 -94 98.48271 host ceph-19 -4 97.20181 host ceph-20 -64 99.77657 host ceph-21 -66 103.56137 host ceph-22 -49 885.49133 datacenter ServerRoom -55 885.49133 room SR-113 -65 86.16304 host ceph-04 -67 101.50623 host ceph-05 -69 104.85805 host ceph-06 -71 96.39923 host ceph-07 -81 97.54230 host ceph-18 -94 98.48271 host ceph-19 -4 97.20181 host ceph-20 -64 99.77657 host ceph-21 -66 103.56137 host ceph-22 -1 0 root default Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 1

Re: After hardware failure tried to recover ceph and followed instructions for recovery using OSDS
by Eugen Block 06 Dec '23

06 Dec '23

And how about this? osd crush rm osd.14 Zitat von Manolis Daramas <mdaramas(a)motivian.com>: > Hi Eugen, > > I think that it cannot even be removed with ceph osd purge command, > see below: > > root@node01:~# ceph osd purge 14 --force --yes-i-really-mean-it > osd.14 does not exist > > root@node01:~# ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 2.05046 root default > -3 0.68349 host node01 > 0 hdd 0.14650 osd.0 up 1.00000 1.00000 > 4 hdd 0.04880 osd.4 up 1.00000 1.00000 > 8 hdd 0.04880 osd.8 up 1.00000 1.00000 > 10 hdd 0.04880 osd.10 up 1.00000 1.00000 > 14 hdd 0.39059 osd.14 DNE 0 > -5 0.68349 host node02 > 2 hdd 0.14650 osd.2 up 1.00000 1.00000 > 5 hdd 0.04880 osd.5 up 1.00000 1.00000 > 7 hdd 0.04880 osd.7 up 1.00000 1.00000 > 9 hdd 0.04880 osd.9 up 1.00000 1.00000 > 12 hdd 0.39059 osd.12 up 1.00000 1.00000 > -7 0.68349 host node03 > 1 hdd 0.14650 osd.1 up 1.00000 1.00000 > 3 hdd 0.04880 osd.3 up 1.00000 1.00000 > 6 hdd 0.04880 osd.6 up 1.00000 1.00000 > 11 hdd 0.04880 osd.11 up 1.00000 1.00000 > 13 hdd 0.39059 osd.13 up 1.00000 1.00000 > > The output of journalctl with --no-pager option is below: > > -- Logs begin at Thu 2023-11-23 06:16:17 EET, end at Tue 2023-12-05 > 16:01:49 EET. -- > Dec 05 15:23:16 node01 systemd[1]: Started Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: Dec 05 15:23:17 node01 bash[1969]: > debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: terminate called after throwing > an instance of 'std::invalid_argument' > Dec 05 15:23:17 node01 bash[1969]: what(): stoull > Dec 05 15:23:17 node01 bash[1969]: *** Caught signal (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: debug -290> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command assert hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -289> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command abort hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -288> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command leak_some_memory hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -287> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -286> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 1 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -285> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -284> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -283> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -282> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 2 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -281> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -280> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -279> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf reset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -278> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config show hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -277> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config help hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -276> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config set hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -275> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config unset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -274> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -273> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -272> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -271> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command injectargs hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -270> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log flush hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -269> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -268> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log reopen hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -267> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command dump_mempools hook 0x55be2168a068 > Dec 05 15:23:17 node01 bash[1969]: debug -266> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug -265> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug -264> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug -263> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > init /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -262> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > bind_and_listen /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -261> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 0 hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -260> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -259> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command git_version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -258> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command help hook 0x55be208de5b0 > Dec 05 15:23:17 node01 bash[1969]: debug -257> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command get_command_descriptions hook 0x55be208de5a0 > Dec 05 15:23:17 node01 bash[1969]: debug -256> > 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) > entry start > Dec 05 15:23:17 node01 bash[1969]: debug -255> > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug -254> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -253> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -252> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -251> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -250> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -249> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -248> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: > verify_sharding column families from rocksdb: [default] > Dec 05 15:23:17 node01 bash[1969]: debug -247> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug -246> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug -245> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug -244> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug -243> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug -242> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug -241> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug -240> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug -239> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug -238> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug -237> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -236> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -235> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -234> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -233> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug -232> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug -231> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug -230> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug -229> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -228> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -227> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -226> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -225> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -224> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug -223> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -222> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -221> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -220> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -219> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -218> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -217> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -216> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug -215> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug -214> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug -213> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -212> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -211> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -210> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -209> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -208> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -207> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -206> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug -205> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -204> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -203> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -202> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -201> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -200> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -199> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -198> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -197> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -196> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -195> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -194> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -193> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -192> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug -191> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug -190> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -189> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -188> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -187> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -186> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -185> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -184> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -183> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -182> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -181> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -180> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -179> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug -178> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -177> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug -176> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug -175> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -174> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug -173> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -172> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -171> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -170> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -169> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -168> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug -167> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -166> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug -165> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -164> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -163> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -162> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -161> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -160> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -159> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -158> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -157> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -156> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug -155> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -154> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -153> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -152> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -151> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -150> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -149> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -148> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -147> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug -146> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug -145> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug -144> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug -143> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug -142> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -141> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -140> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -139> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug -138> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug -137> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -136> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -135> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -134> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -133> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug -132> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -131> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -130> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug -129> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -128> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -127> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -126> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -125> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -124> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -123> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -122> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -121> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -120> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -119> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -118> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -117> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -116> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -115> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -114> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -113> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -112> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug -111> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug -110> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug -109> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug -108> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -107> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -106> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -105> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -104> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -103> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -102> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -101> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -100> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -99> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -98> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -97> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug -96> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug -95> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -94> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug -93> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug -92> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -91> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -90> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug -89> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug -88> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -87> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -86> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug -85> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug -84> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -83> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug -82> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -81> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -80> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug -79> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -78> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug -77> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -76> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -75> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -74> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -73> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -72> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -71> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -70> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -69> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -68> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug -67> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -66> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug -65> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -64> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -63> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -62> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug -61> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -60> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -59> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug -58> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug -57> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug -56> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -55> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug -54> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug -53> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug -52> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -51> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug -50> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -49> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug -48> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug -47> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug -46> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug -45> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -44> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -43> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -42> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -41> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -40> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -39> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -38> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -37> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -36> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -35> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -34> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -33> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -32> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -31> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -30> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -29> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -28> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -27> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -26> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -25> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -24> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -23> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -22> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -21> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -20> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -19> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -18> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -17> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -16> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -15> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -14> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -13> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -12> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -11> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -10> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -9> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -8> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -7> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -6> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -5> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01(a)-1(???).mds > e0 Unable to load 'last_metadata' > Dec 05 15:23:17 node01 bash[1969]: debug -4> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -3> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -2> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -1> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug 0> > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon > Dec 05 15:23:17 node01 bash[1969]: 0/10 monc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache > Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse > Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter > Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) > Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) > Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for > recent threads --- > Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket > Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 > Dec 05 15:23:17 node01 bash[1969]: max_new 10000 > Dec 05 15:23:17 node01 bash[1969]: log_file > Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: debug -290> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command assert hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -289> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command abort hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -288> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command leak_some_memory hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -287> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -286> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 1 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -285> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -284> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -283> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -282> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 2 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -281> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -280> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -279> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf reset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -278> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config show hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -277> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config help hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -276> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config set hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -275> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config unset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -274> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -273> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -272> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -271> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command injectargs hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -270> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log flush hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -269> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -268> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log reopen hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -267> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command dump_mempools hook 0x55be2168a068 > Dec 05 15:23:17 node01 bash[1969]: debug -266> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug -265> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug -264> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug -263> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > init /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -262> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > bind_and_listen /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -261> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 0 hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -260> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -259> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command git_version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -258> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command help hook 0x55be208de5b0 > Dec 05 15:23:17 node01 bash[1969]: debug -257> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command get_command_descriptions hook 0x55be208de5a0 > Dec 05 15:23:17 node01 bash[1969]: debug -256> > 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) > entry start > Dec 05 15:23:17 node01 bash[1969]: debug -255> > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug -254> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -253> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -252> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -251> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -250> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -249> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -248> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: > verify_sharding column families from rocksdb: [default] > Dec 05 15:23:17 node01 bash[1969]: debug -247> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug -246> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug -245> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug -244> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug -243> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug -242> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug -241> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug -240> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug -239> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug -238> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug -237> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -236> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -235> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -234> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -233> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug -232> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug -231> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug -230> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug -229> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -228> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -227> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -226> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -225> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -224> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug -223> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -222> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -221> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -220> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -219> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -218> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -217> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -216> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug -215> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug -214> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug -213> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -212> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -211> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -210> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -209> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -208> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -207> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -206> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug -205> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -204> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -203> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -202> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -201> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -200> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -199> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -198> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -197> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -196> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -195> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -194> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -193> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -192> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug -191> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug -190> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -189> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -188> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -187> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -186> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -185> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -184> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -183> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -182> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -181> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -180> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -179> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug -178> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -177> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug -176> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug -175> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -174> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug -173> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -172> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -171> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -170> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -169> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -168> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug -167> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -166> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug -165> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -164> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -163> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -162> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -161> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -160> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -159> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -158> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -157> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -156> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug -155> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -154> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -153> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -152> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -151> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -150> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -149> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -148> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -147> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug -146> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug -145> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug -144> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug -143> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug -142> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -141> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -140> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -139> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug -138> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug -137> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -136> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -135> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -134> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -133> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug -132> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -131> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -130> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug -129> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -128> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -127> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -126> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -125> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -124> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -123> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -122> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -121> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -120> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -119> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -118> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -117> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -116> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -115> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -114> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -113> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -112> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug -111> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug -110> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug -109> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug -108> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -107> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -106> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -105> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -104> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -103> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -102> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -101> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -100> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -99> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -98> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -97> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug -96> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug -95> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -94> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug -93> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug -92> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -91> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -90> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug -89> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug -88> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -87> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -86> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug -85> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug -84> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -83> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug -82> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -81> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -80> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug -79> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -78> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug -77> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -76> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -75> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -74> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -73> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -72> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -71> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -70> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -69> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -68> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug -67> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -66> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug -65> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -64> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -63> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -62> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug -61> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -60> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -59> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug -58> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug -57> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug -56> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -55> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug -54> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug -53> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug -52> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -51> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug -50> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -49> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug -48> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug -47> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug -46> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug -45> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -44> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -43> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -42> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -41> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -40> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -39> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -38> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -37> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -36> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -35> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -34> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -33> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -32> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -31> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -30> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -29> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -28> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -27> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -26> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -25> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -24> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -23> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -22> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -21> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -20> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -19> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -18> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -17> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -16> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -15> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -14> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -13> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -12> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -11> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -10> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -9> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -8> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -7> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -6> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -5> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01(a)-1(???).mds > e0 Unable to load 'last_metadata' > Dec 05 15:23:17 node01 bash[1969]: debug -4> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -3> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -2> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -1> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug 0> > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon > Dec 05 15:23:17 node01 bash[1969]: 0/10 monc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache > Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse > Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter > Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) > Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) > Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for > recent threads --- > Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket > Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 > Dec 05 15:23:17 node01 bash[1969]: max_new 10000 > Dec 05 15:23:17 node01 bash[1969]: log_file > /var/lib/ceph/crash/2023-12-05T13:23:17.739483Z_174a5e49-22df-4c2f-be14-4e88cc65b2f2/log > Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- > Dec 05 15:23:17 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: Main > process exited, code=exited, status=134/n/a > Dec 05 15:23:18 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: Failed > with result 'exit-code'. > Dec 05 15:23:29 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: > Scheduled restart job, restart counter is at 1. > Dec 05 15:23:29 node01 systemd[1]: Stopped Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:29 node01 systemd[1]: Started Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 0 load: jerasure load: lrc > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB SUMMARY > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB Session ID: > OQS561SAY9M8P46I122B > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: MANIFEST file: > MANIFEST-000231 size: 308 Bytes > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000232.log size: > 0 ; > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.env: 0x562241625a80 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.info_log: 0x562243b9a360 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_buffer_manager: 0x56224490e990 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000231 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x562243becfb0) > Dec 05 15:23:29 node01 bash[6375]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:29 node01 bash[6375]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:29 node01 bash[6375]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:29 node01 bash[6375]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:29 node01 bash[6375]: index_type: 0 > Dec 05 15:23:29 node01 bash[6375]: data_block_index_type: 0 > Dec 05 15:23:29 node01 bash[6375]: index_shortening: 1 > Dec 05 15:23:29 node01 bash[6375]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:29 node01 bash[6375]: hash_index_allow_collision: 1 > Dec 05 15:23:29 node01 bash[6375]: checksum: 1 > Dec 05 15:23:29 node01 bash[6375]: no_block_cache: 0 > Dec 05 15:23:29 node01 bash[6375]: block_cache: 0x562243c1f090 > Dec 05 15:23:29 node01 bash[6375]: block_cache_name: BinnedLRUCache > Dec 05 15:23:29 node01 bash[6375]: block_cache_options: > Dec 05 15:23:29 node01 bash[6375]: capacity : 536870912 > Dec 05 15:23:29 node01 bash[6375]: num_shard_bits : 4 > Dec 05 15:23:29 node01 bash[6375]: strict_capacity_limit : 0 > Dec 05 15:23:29 node01 bash[6375]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:29 node01 bash[6375]: block_cache_compressed: (nil) > Dec 05 15:23:29 node01 bash[6375]: persistent_cache: (nil) > Dec 05 15:23:29 node01 bash[6375]: block_size: 4096 > Dec 05 15:23:29 node01 bash[6375]: block_size_deviation: 10 > Dec 05 15:23:29 node01 bash[6375]: block_restart_interval: 16 > Dec 05 15:23:29 node01 bash[6375]: index_block_restart_interval: 1 > Dec 05 15:23:29 node01 bash[6375]: metadata_block_size: 4096 > Dec 05 15:23:29 node01 bash[6375]: partition_filters: 0 > Dec 05 15:23:29 node01 bash[6375]: use_delta_encoding: 1 > Dec 05 15:23:29 node01 bash[6375]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:29 node01 bash[6375]: whole_key_filtering: 1 > Dec 05 15:23:29 node01 bash[6375]: verify_compression: 0 > Dec 05 15:23:29 node01 bash[6375]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:29 node01 bash[6375]: format_version: 4 > Dec 05 15:23:29 node01 bash[6375]: enable_index_compression: 1 > Dec 05 15:23:29 node01 bash[6375]: block_align: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.paranoid_file_checks: Dec 05 15:23:29 node01 bash[6375]: > debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:29 > [Text gekürzt]

3 6

Space reclaim doesn't happening in nautilus RBD pool
by Szabo, Istvan (Agoda) 06 Dec '23

06 Dec '23

Hi, Is there any config on Ceph that block/not perform space reclaim? I test on one pool which has only one image 1.8 TiB in used. rbd $p du im/root warning: fast-diff map is not enabled for root. operation may be slow. NAME PROVISIONED USED root 2.2 TiB 1.8 TiB I already removed all snaphots and now pool has only one image alone. I run both fstrim over the filesystem (XFS) and try rbd sparsify im/root (don't know what it is exactly but it mentions to reclaim something) It still shows the pool used 6.9 TiB which totally not make sense right? It should be up to 3.6 (1.8 * 2) according to its replica? POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR im 19 32 3.5 TiB 918.34k 6.9 TiB 4.80 69 TiB N/A 10 TiB 918.34k 0 B 0 B I think now some of others pool have this issue too, we do clean up a lot but seems space not reclaimed. I estimate more than 50 TiB should be able to reclaim, actual usage of this cluster much less than current reported number. Thank you for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 4

Ceph 16.2.14: osd crash, bdev() _aio_thread got r=-1 ((1) Operation not permitted)
by Zakhar Kirpichenko 06 Dec '23

06 Dec '23

Hi, One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug. Usually an OSD, which crashed because of this bug, restarts within seconds and continues normal operation. This time it failed to restart and kept crashing: "assert_condition": "abort", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc", "assert_func": "void KernelDevice::_aio_thread()", "assert_line": 604, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f08520e2700 time 2023-12-03T04:00:36.689614+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg(\"Unexpected IO error. This may suggest HW issue. Please check your dmesg!\")\n", "assert_thread_name": "bstore_aio", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7f085e308cf0]", "gsignal()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x55f01d9494cb]", "(KernelDevice::_aio_thread()+0x1285) [0x55f01e4b5c15]", "(KernelDevice::AioCompletionThread::entry()+0x11) [0x55f01e4c0ee1]", "/lib64/libpthread.so.0(+0x81ca) [0x7f085e2fe1ca]", "clone()" ], There was nothing in dmesg though and the block device looked healthy. I took the OSD down, ran a long SMART test on its block drive, ran a read test on the drive and found no issues. I tried restarting the OSD again and found in its debug that it failed because of an "2023-12-03T04:00:36.686+0000 7f08520e2700 -1 bdev(0x55f02a28a400 /var/lib/ceph/osd/ceph-56/block) _aio_thread got r=-1 ((1) Operation not permitted)" error: https://pastebin.com/gDat6rfk I remember hitting this previously: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/GYL72G3F4P…, and this time a host reboot completely resolved the issue. It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions. Best regards, Zakhar

3 6

CephFS pool not releasing space after data deletion
by Kuhring, Mathias 05 Dec '23

05 Dec '23

Dear Ceph users, Our CephFS is not releasing/freeing up space after deleting hundreds of terabytes of data. By now, this drives us in a "nearfull" osd/pool situation and thus throttles IO. We are on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable). Recently, we moved a bunch of data to a new pool with better EC. This was done by adding a new EC pool to the FS. Then assigning the FS root to the new EC pool via the directory layout xattr (so all new data is written to the new pool). And finally copying old data to new folders. I swapped the data as follows to remain the old directory structures. I also made snapshots for validation purposes. So basically: cp -r mymount/mydata/ mymount/new/ # this creates copy on new pool mkdir mymount/mydata/.snap/tovalidate mkdir mymount/new/mydata/.snap/tovalidate mv mymount/mydata/ mymount/old/ mv mymount/new/mydata mymount/ I could see the increase of data in the new pool as expected (ceph df). I compared the snapshots with hashdeep to make sure the new data is alright. Then I went ahead deleting the old data, basically: rmdir mymount/old/mydata/.snap/* # this also included a bunch of other older snapshots rm -r mymount/old/mydata At first we had a bunch of PGs with snaptrim/snaptrim_wait. But they are done for quite some time now. And now, already two weeks later the size of the old pool still hasn't really decreased. I'm still waiting for around 500 TB to be released (and much more is planned). I honestly have no clue, where to go from here. From my point of view (i.e. the CephFS mount), the data is gone. I also never hard/soft-linked it anywhere. This doesn't seem to be a regular issue. At least I couldn't find anything related or resolved in the docs or user list, yet. If anybody has an idea how to resolve this, I would highly appreciate it. Best Wishes, Mathias

3 4

After hardware failure tried to recover ceph and followed instructions for recovery using OSDS
by Manolis Daramas 05 Dec '23

05 Dec '23

Hello everyone, We had a recent power failure on a server which hosts a 3-node ceph cluster (with Ubuntu 20.04 and Ceph version 17.2.7) and we think that we may have lost some of our data if not all of them. We have followed the instructions on https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#re… but with no luck. We have kept a backup of store.db folder on all 3 nodes prior the below steps. We have stopped ceph.target on all 3 nodes. We have run the first part of the script and we have altered it according to our configuration: ms=/root/mon-store mkdir $ms hosts="node01 node02 node03" # collect the cluster map from stopped OSDs for host in $hosts; do rsync -avz $ms/. root@$host:$ms.remote rm -rf $ms ssh root@$host <<EOF for osd in /var/lib/ceph/be4304e4-b0d5-11ec-8c6a-2965d4229f37/osd*; do ceph-objectstore-tool --data-path \$osd --no-mon-config --op update-mon-db --mon-store-path $ms.remote done EOF rsync -avz root@$host:$ms.remote/. $ms done and the results were: for node01 sd.0 : 0 osdmaps trimmed, 673 osdmaps added. osd.10 : 9225 osdmaps trimmed, 0 osdmaps added. Mount failed with '(5) Input/output error' osd.4 : 0 osdmaps trimmed, 0 osdmaps added. osd.8 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000022.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000021 store.db/OPTIONS-000018 store.db/OPTIONS-000024 sent 248 bytes received 286,474 bytes 191,148.00 bytes/sec total size is 7,869,025 speedup is 27.44 sending incremental file list created directory /root/mon-store.remote ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000022.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000021 store.db/OPTIONS-000018 store.db/OPTIONS-000024 sent 286,478 bytes received 285 bytes 191,175.33 bytes/sec total size is 7,869,025 speedup is 27.44 for node02 osd.12 : 0 osdmaps trimmed, 0 osdmaps added. osd.2 : 0 osdmaps trimmed, 0 osdmaps added. osd.5 : 0 osdmaps trimmed, 0 osdmaps added. osd.7 : 0 osdmaps trimmed, 0 osdmaps added. osd.9 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000052.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000051 store.db/OPTIONS-000048 store.db/OPTIONS-000054 sent 343 bytes received 291,082 bytes 194,283.33 bytes/sec total size is 7,875,746 speedup is 27.02 sending incremental file list created directory /root/mon-store.remote ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000052.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000051 store.db/OPTIONS-000048 store.db/OPTIONS-000054 sent 291,078 bytes received 380 bytes 582,916.00 bytes/sec total size is 7,875,746 speedup is 27.02 for node03 osd.1 : 0 osdmaps trimmed, 0 osdmaps added. osd.11 : 0 osdmaps trimmed, 0 osdmaps added. osd.13 : 0 osdmaps trimmed, 0 osdmaps added. osd.3 : 0 osdmaps trimmed, 0 osdmaps added. osd.6 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000056.sst store.db/000062.sst store.db/000068.sst store.db/000074.sst store.db/000080.sst store.db/000082.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000081 store.db/OPTIONS-000078 store.db/OPTIONS-000084 sent 438 bytes received 295,659 bytes 592,194.00 bytes/sec total size is 7,882,477 speedup is 26.62 Then we have run the (in order to rebuild the monstore DB and fix it): ceph-monstore-tool /root/mon-store rebuild -- --keyring /etc/ceph/ceph.client.admin.keyring --mon-ids node01 node02 node03 and the output is below: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: RocksDB version: 6.15.5 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compile date Oct 25 2023 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB SUMMARY 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB Session ID: OS2T69IQ02SU5OKHBI40 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: CURRENT file: CURRENT 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: IDENTITY file: IDENTITY 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: MANIFEST file: MANIFEST-000081 size: 1083 Bytes 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: SST files in /root/mon-store/store.db dir, Total Num: 13, files: 000008.sst 000014.sst 000020.sst 000026.sst 000032.sst 000038.sst 000044.sst 000050.sst 000056.sst 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Write Ahead Log file in /root/mon-store/store.db: 000082.log size: 244 ; 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.error_if_exists: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.create_if_missing: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.env: 0x56017c8d1c20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.fs: Posix File System 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.info_log: 0x56017d4c3860 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.statistics: (nil) 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_fsync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_log_dir: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_dir: /root/mon-store/store.db 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_buffer_manager: 0x56017d1f6a20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.unordered_write: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.row_cache: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_filter: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.two_write_queues: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.atomic_flush: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.best_efforts_recovery: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_data_in_errors: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_host_id: __hostname__ 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_open_files: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compression algorithms supported: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTD supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kXpressCompression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZlibCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /root/mon-store/store.db/MANIFEST-000081 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.merge_operator: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_filter: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.sst_partitioner_factory: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56017d234f80) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x56017d22f610 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 4 enable_index_compression: 1 block_align: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression: NoCompression 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.num_levels: 7 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_properties_collectors: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bloom_locality: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.force_consistency_checks: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.ttl: 2592000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_blob_files: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.min_blob_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_file_size: 268435456 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_compression_type: NoCompression 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/root/mon-store/store.db/MANIFEST-000081 succeeded,manifest_file_number is 81, next_file_number is 83, last_sequence is 21183, log_number is 77,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 77 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4082] Creating manifest 85 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784162798, "job": 1, "event": "recovery_started", "wal_files": [82]} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #82 mode 2 2023-11-17T12:26:24.160+0200 7f482b393600 3 rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784163944, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 86, "file_size": 1266, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 238, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 72, "raw_average_key_size": 24, "raw_value_size": 148, "raw_average_value_size": 49, "num_data_blocks": 1, "num_entries": 3, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216784, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id": "OS2T69IQ02SU5OKHBI40"}} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4082] Creating manifest 87 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784166273, "job": 1, "event": "recovery_finished"} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/column_family.cc:983] [default] Increasing compaction threads because we have 14 level-0 files 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000082.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56017d230700 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: DB pointer 0x56017df56000 adding auth for 'client.admin': auth(key=AQCsdUViHYjTGBAAf7/1KYZjb0h3x3EOywqbbQ==) with caps({mds=allow *,mgr=allow *,mon=allow *,osd=allow *}) 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1881] [default] [JOB 3] Compacting 14@0 files to L6, score 3.50 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1887] [default] Compaction start summary: Base version 3 Base level 0, inputs: [86(1266B) 80(1266B) 74(1267B) 68(1267B) 62(1266B) 56(1265B) 50(1265B) 44(1265B) 38(1265B) 32(1266B) 26(1265B) 20(1265B) 14(283KB) 8(7387KB)] 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784169200, "job": 3, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [86, 80, 74, 68, 62, 56, 50, 44, 38, 32, 26, 20, 14, 8], "score": 3.5, "input_data_size": 7870219} 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1516] [default] [JOB 3] Generated table #91: 1366 keys, 7566988 bytes 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784213586, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 91, "file_size": 7566988, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 7541895, "index_size": 20610, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3525, "raw_key_size": 29308, "raw_average_key_size": 21, "raw_value_size": 7503048, "raw_average_value_size": 5492, "num_data_blocks": 764, "num_entries": 1366, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216681, "oldest_key_time": 0, "file_creation_time": 1700216784, "db_id": "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id": "OS2T69IQ02SU5OKHBI40"}} 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1594] [default] [JOB 3] Compacted 14@0 files to L6 => 7566988 bytes 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log Time 2023/11/17-12:26:24.215298) [db/compaction/compaction_job.cc:812] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 177.1 rd, 170.3 wr, level 6, files in(14, 0) out(1) MB in(7.5, 0.0) out(7.2), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 19842, records dropped: 18476 output_compression: NoCompression 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log Time 2023/11/17-12:26:24.215314) EVENT_LOG_v1 {"time_micros": 1700216784215306, "job": 3, "event": "compaction_finished", "compaction_time_micros": 44437, "compaction_time_cpu_micros": 40923, "output_level": 6, "num_output_files": 1, "total_output_size": 7566988, "num_input_records": 19842, "num_output_records": 1366, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000086.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215520, "job": 3, "event": "table_file_deletion", "file_number": 86} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000080.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215570, "job": 3, "event": "table_file_deletion", "file_number": 80} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000074.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215603, "job": 3, "event": "table_file_deletion", "file_number": 74} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000068.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215641, "job": 3, "event": "table_file_deletion", "file_number": 68} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000062.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215672, "job": 3, "event": "table_file_deletion", "file_number": 62} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000056.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215708, "job": 3, "event": "table_file_deletion", "file_number": 56} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000050.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215739, "job": 3, "event": "table_file_deletion", "file_number": 50} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000044.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215772, "job": 3, "event": "table_file_deletion", "file_number": 44} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000038.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215804, "job": 3, "event": "table_file_deletion", "file_number": 38} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000032.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215831, "job": 3, "event": "table_file_deletion", "file_number": 32} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000026.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215858, "job": 3, "event": "table_file_deletion", "file_number": 26} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000020.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215888, "job": 3, "event": "table_file_deletion", "file_number": 20} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000014.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215952, "job": 3, "event": "table_file_deletion", "file_number": 14} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000008.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784216804, "job": 3, "event": "table_file_deletion", "file_number": 8} update_mkfs generating seed initial monmap epoch 0 fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 last_changed 2023-11-17T12:26:24.222814+0200 created 2023-11-17T12:26:24.222814+0200 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon.node01 1: [v2:10.40.99.12:3300/0,v1:10.40.99.12:6789/0] mon.node02 2: [v2:10.40.99.13:3300/0,v1:10.40.99.13:6789/0] mon.node03 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl.cc:446] Shutdown: canceling all background work 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl.cc:625] Shutdown complete Then we copied the /root/mon-store/store.db folder across on all 3 nodes and tried to start ceph.target service again. The output on node01 is below: d31781fa6b4c quay.io/ceph/ceph "/usr/bin/ceph-mds -..." 55 minutes ago Up 55 minutes ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node01-cjrvjc e385c32651d2 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-10 904f522c4cb5 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-0 033edf99a98e quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-4 70344a6e87a0 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-8 905b782aedcf quay.io/prometheus/prometheus:v2.43.0 "/bin/prometheus --c..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-prometheus-node01 ff191654eb3e quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node01 459c46f4bdb7 quay.io/ceph/ceph "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node01-xlciyx cacfe8abcbbf quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node01 e216ef2af166 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-alertmanager-node01 d3238b2285d1 quay.io/ceph/ceph-grafana:9.4.7 "/bin/sh -c 'grafana..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-grafana-node01 The output on node02 is below: 2aec62685dee quay.io/ceph/ceph "/usr/bin/ceph-mds -..." 54 minutes ago Up 54 minutes ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node02-lyudbp 249b04f32f8c quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-5 a2c96f56b517 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-2 87496d374a29 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-12 55fe47765917 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-9 76171e25dbde quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-7 220472e8c1bf quay.io/ceph/ceph "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node02-gudauu 0c783e73e543 quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node02 4e638003fa2e quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node02 42719d5cfdbf quay.io/ceph/ceph "/usr/bin/ceph-mon -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node02 The output on node03 is below: 7e5879dce643 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-11 d53996ff33b9 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-3 e1ac5a8b87d3 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-1 f4cda871218d quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-13 969e670dc47c quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-6 a49e91a7bb8e quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node03 835c3893a3f4 quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node03 bfa6f5b989ea quay.io/ceph/ceph "/usr/bin/ceph-mon -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node03 # ceph -s (output below): cluster: id: be4304e4-b0d5-11ec-8c6a-2965d4229f37 health: HEALTH_ERR 20 stray daemon(s) not managed by cephadm 3 stray host(s) with 20 daemon(s) not managed by cephadm 1/3 mons down, quorum node02,node03 1/523510 objects unfound (0.000%) 3 nearfull osd(s) 1 osds exist in the crush map but not in the osdmap Low space hindering backfill (add storage if this doesn't resolve itself): 20 pgs backfill_toofull Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 74666/1570530 objects degraded (4.754%), 21 pgs degraded, 21 pgs undersized 3 pool(s) nearfull services: mon: 3 daemons, quorum node02,node03 (age 2d), out of quorum: node01 mgr: node01.xlciyx(active, since 2d), standbys: node02.gudauu osd: 14 osds: 14 up (since 2d), 14 in (since 3d); 21 remapped pgs data: pools: 3 pools, 161 pgs objects: 523.51k objects, 299 GiB usage: 1014 GiB used, 836 GiB / 1.8 TiB avail pgs: 74666/1570530 objects degraded (4.754%) 1/523510 objects unfound (0.000%) 140 active+clean 20 active+undersized+degraded+remapped+backfill_toofull 1. active+recovery_unfound+undersized+degraded+remapped # ceph fs ls (output below): No filesystems enabled It looks like that we have a problem with the orchestrator now (we've lost cephadm orchestrator) and we also cannot see the filesystem. May you please assist since we are not able to mount the filesystem ? Thank you, Manolis Daramas Under the General Data Protection Regulation (GDPR) (EU) 2016/679, Motivian as Data Controller has a legal duty to protect any information collected from you via email. Information contained in this email and any attachments may be privileged or confidential and intended for the exclusive use of the original recipient. If you have received this email by mistake, please advise the sender immediately and delete the email, including emptying your deleted email box. Information included in this email is reserved to named addressee's eyes only. You may not share this message or any of its attachments to anyone. Please note that as the recipient, it is your responsibility to check the email for malicious software. Motivian puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.

2 5

MDS stuck in up:rejoin
by Eric Tittley 05 Dec '23

05 Dec '23

Hi all, For about a week our CephFS has experienced issues with its MDS. Currently the MDS is stuck in "up:rejoin" Issues become apparent when simple commands like "mv foo bar/" hung. I unmounted CephFS offline on the clients, evicted those remaining, and then issued ceph config set mds.0 mds_wipe_sessions true ceph config set mds.1 mds_wipe_sessions true which allowed me to delete the hung requests. I've lost the exact commands I used, but something like rados -p cephfs_metadata ls | grep mds rados rm -p cephfs_metadata mds0_openfiles.0 etc This allowed the MDS to get to "up:rejoin" where it has been stuck ever since which is getting on five days. # ceph mds stat cephfs:1/1 {0=cephfs.ceph00.uvlkrw=up:rejoin} 2 up:standby root@ceph00:/var/log/ceph/a614303a-5eb5-11ed-b492-011f01e12c9a# ceph -s cluster: id: a614303a-5eb5-11ed-b492-011f01e12c9a health: HEALTH_WARN 1 filesystem is degraded 1 pgs not deep-scrubbed in time 2 pool(s) do not have an application enabled 1 daemons have recently crashed services: mon: 3 daemons, quorum ceph00,ceph01,ceph02 (age 57m) mgr: ceph01.lvdgyr(active, since 2h), standbys: ceph00.gpwpgs mds: 1/1 daemons up, 2 standby osd: 91 osds: 90 up (since 78m), 90 in (since 112m) data: volumes: 0/1 healthy, 1 recovering pools: 5 pools, 1539 pgs objects: 138.83M objects, 485 TiB usage: 971 TiB used, 348 TiB / 1.3 PiB avail pgs: 1527 active+clean 12 active+clean+scrubbing+deep io: client: 3.1 MiB/s rd, 3.16k op/s rd, 0 op/s wr # ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) I've tried failing the MDS so it switches. Rebooted a couple of times. I've added more OSDs to the metadata pool and took one out as I thought it might be a bad metadata OSD (The "recently crashed" daemon). The error logs are full of (prefix to all are: Nov 27 14:02:44 ceph00 bash[2145]: debug 2023-11-27T14:02:44.619+0000 7f74e845e700 1 -- [v2:192.168.1.128:6800/2157301677,v1:192.168.1.128:6801/2157301677] --> [v2:192.168.1.133:6896/4289132926,v1:192.168.1.133:6897/4289132926] ) crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x559be00adc00 type=42 osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write_message sending message m=0x559be00adc00 seq=8142643 osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 crc :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 154 + 0 + 30 byte message. envelope type=43 src osd.89 off 0 crc :-1 s=READ_MESSAGE_COMPLETE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x559be01f4480 seq=8142643 from=osd.89 type=43 osd_op_reply(8142873 1.00000000 [getxattr (30) out=30b] v0'0 uv560123 ondisk = 0) v8 osd_op_reply(8142873 1.00000000 [getxattr (30) out=30b] v0'0 uv560123 ondisk = 0) v8 ==== 154+0+30 (crc 0 0 0) 0x559be01f4480 con 0x559be00ad800 osd_op(unknown.0.36244:8142874 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 -- 0x559be2caec00 con 0x559be00ad800 Repeating multiple times a second (and filling /var) Prior to taking one of the cephfs_metadata OSDs offline, these came from communications from ceph00 to the node hosting the suspected bad OSD. Now they are between ceph00 and the host of the replacement metadata OSD. Does anyone have any suggestion on how to get the MDS to switch from "up:rejoin" to "up:active"? Is there any way to debug this, to determine what issue really is? I'm unable to interpret the debug log. Cheers, Eric ________________________________________________________ Dr Eric Tittley Research Computing Officer www.roe.ac.uk/~ert<http://www.roe.ac.uk/~ert> Institute for Astronomy Royal Observatory, Edinburgh The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

2 5

Re: ceph-users Digest, Vol 114, Issue 14
by duluxoz 05 Dec '23

05 Dec '23

Hi Zitat, I'm confused - doesn't k4 m2 mean that you can loose any 2 out of the 6 osds? Cheers Dulux-Oz On 05/12/2023 20:02, ceph-users-request(a)ceph.io wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: EC Profiles & DR (David Rivera) > 2. Re: EC Profiles & DR (duluxoz) > 3. Re: EC Profiles & DR (Eugen Block) > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

3 2

mds slow request with “failed to authpin, subtree is being exported"
by zxcs 05 Dec '23

05 Dec '23

HI, Experts, we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system. and one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path. then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported" then need to restart mds, our question is, does there any dead lock? how can we avoid this and how to fix it without restart mds(it will influence other users) ? Thanks a ton! xz

6 11

the image used size becomes 0 after export/import with snapshot
by Tony Liu 05 Dec '23

05 Dec '23

Hi, I have an image with a snapshot and some changes after snapshot. ``` $ rbd du backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 NAME PROVISIONED USED f0408e1e-06b6-437b-a2b5-70e3751d0a26@snapshot-eb085877-7557-4620-9c01-c5587b857029 10 GiB 2.4 GiB f0408e1e-06b6-437b-a2b5-70e3751d0a26 10 GiB 2.4 GiB <TOTAL> 10 GiB 4.8 GiB ``` If there is no changes after snapshot, the image line will show 0 used. I did export and import. ``` $ rbd export --export-format 2 backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 - | rbd import --export-format 2 - backup/test Exporting image: 100% complete...done. Importing image: 100% complete...done. ``` When check the imported image, the image line shows 0 used. ``` $ rbd du backup/test NAME PROVISIONED USED test@snapshot-eb085877-7557-4620-9c01-c5587b857029 10 GiB 2.4 GiB test 10 GiB 0 B <TOTAL> 10 GiB 2.4 GiB ``` Any clues how that happened? I'd expect the same du as the source. I tried another quick test. It works fine. ``` $ rbd create backup/test-src --size 10G $ sudo rbd map backup/test-src /dev/rbd0 $ echo "hello" | sudo tee /dev/rbd0 hello $ rbd du backup/test-src NAME PROVISIONED USED test-src 10 GiB 4 MiB $ rbd snap create backup/test-src@snap-1 Creating snap: 100% complete...done. $ rbd du backup/test-src NAME PROVISIONED USED test-src@snap-1 10 GiB 4 MiB test-src 10 GiB 0 B <TOTAL> 10 GiB 4 MiB $ echo "world" | sudo tee /dev/rbd0 world $ rbd du backup/test-src NAME PROVISIONED USED test-src@snap-1 10 GiB 4 MiB test-src 10 GiB 4 MiB <TOTAL> 10 GiB 8 MiB $ rbd export --export-format 2 backup/test-src - | rbd import --export-format 2 - backup/test-dst Exporting image: 100% complete...done. Importing image: 100% complete...done. $ rbd du backup/test-dst NAME PROVISIONED USED test-dst@snap-1 10 GiB 4 MiB test-dst 10 GiB 4 MiB <TOTAL> 10 GiB 8 MiB ``` Thanks! Tony

2 2