December 2023 - ceph-users

How to replace a disk with minimal impact on performance

by Michal Strnad

Hi guys! Based on our observation of the impact of the balancer on the performance of the entire cluster, we have drawn conclusions that we would like to discuss with you. - A newly created pool should be balanced before being handed over to the user. This, I believe, is quite evident. - When replacing a disk, it is advisable to exchange it directly for a new one. As soon as the OSD replacement occurs, the balancer should be invoked to realign any improperly placed PGs during the disk outage and disk recovery. Perhaps an even better method is to pause recovery and backfilling before removing the disk, remove the disk itself, promptly add a new one, and then resume recovery and backfilling. It's essential to per form all of this as quickly as possible (using a script). Ad. We are using a community balancer developed by Jonas Jelton because the built-in one does not meet our requirements. What are your thoughts on this? Michal

5 months, 1 week

2
1
0 0

nfs export over RGW issue in Pacific

by Adiga, Anantha

Hi, oot@a001s016:~# cephadm version Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) root@a001s016:~# root@a001s016:~# cephadm shell Inferring fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 Inferring config /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/config Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 root@a001s016:~# ceph version ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) --------------------- But, Cephadm does not show "nfs export create rgw...." nfs export create cephfs <fsname> <clusterid> <binding> [--readonly] [<path>] nfs export rm <clusterid> <binding> nfs export delete <clusterid> <binding> nfs export ls <clusterid> [--detailed] nfs export get <clusterid> <binding> nfs export update --------------------- However, Ceph Dashboard allows to create the export see below: Access Type RW Cluster nfs-1 Daemons nfs-1.0.0.zp3110b001a0101.uckows, nfs-1.1.0.zp3110b001a0102.hhpebb, nfs-1.2.0.zp3110b001a0103.bbkpcb, nfs-1.3.0.zp3110b001a0104.zujkso NFS Protocol NFSv4 Object Gateway User admin Path buc-cluster-inventory Pseudo /rgwnfs_cluster_inventory Squash no_root_squash Storage Backend Object Gateway Transport TCP --------------------- While nfs export is created, pseudo "/rgwnfs_cluster_inventory", cephadm is not listing it # ceph nfs export ls nfs-1 [ "/cluster_inventory", "/oob_crashdump" ] # Anantha

5 months, 2 weeks

2
2
0 0

Difficulty adding / using a non-default RGW placement target & storage class

by Anthony D'Atri

I'm having difficulty adding and using a non-default placement target & storage class and would appreciate insights. Am I going about this incorrectly? Rook does not yet have the ability to do this, so I'm adding it by hand. Following instructions on the net I added a second bucket pool, placement target, and storageclass, and created a user defaulting to the new pg/sc, but I get an error when trying to create a bucket: [rook@rook-ceph-tools-5ff8d58445-gkl5w .aws]$ s5cmd --endpoint-url http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc mb s3://foofoobars ERROR "mb s3://foofoobars": InvalidLocationConstraint: status code: 400, request id: tx0000057b71002881d48ca-0065495d54-1abe555-ceph-objectstore, host id: I found an article suggesting that the placement target and/or storage class should have the api_name prepended, so I tried setting either or both to "ceph-objectstore:HDD-EC" / "ceph-objectstore:GLACIER" with no success. I suspect that I'm missing something subtle -- or that Rook has provisioned these bits in an atypical fashion. Log entry: /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.543+0000 7f6573a9f700 1 ====== starting new request req=0x7f64818ba730 ===== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 0 req 6320538205097380042 0.003000009s s3:create_bucket could not find user default placement id HDD-EC/GLACIER within zonegroup /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 1 ====== req done req=0x7f64818ba730 op status=-2208 http_status=400 latency=0.003000009s ====== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log:2023-11-06T21:40:36.546+0000 7f6570a99700 1 beast: 0x7f64818ba730: 10.233.90.156 - aad [06/Nov/2023:21:40:36.543 +0000] "PUT /foofoobars HTTP/1.1" 400 266 - "aws-sdk-go/1.40.25 (go1.18.3; linux; amd64)" - latency=0.003000009s [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ ceph -v ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Here's the second buckets pool, constrained to HDDs. AFAICT it can share the index and data_extra_pool created for the default / STANDARD pt/sc by Rook. I initially omitted ec_overwrites but enabled it after creation. pool 19 'ceph-objectstore.rgw.buckets.data' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile size 6 min_size 5 crush_rule 10 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 165350 lfor 0/156300/165341 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw pool 21 'ceph-objectstore.rgw.buckets.data.hdd' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile_hdd size 6 min_size 5 crush_rule 11 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 167193 lfor 0/0/164453 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zonegroup get { "id": "d994155c-2a9c-4e37-ae30-64fd2934ff99", "name": "ceph-objectstore", "api_name": "ceph-objectstore", "is_master": "true", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "72035401-a6d9-426b-8c89-9a17e268825f", "zones": [ { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "HDD-EC", "tags": [], "storage_classes": [ "GLACIER" ] }, { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "51fb8875-31ac-40ef-ab21-0ffd4e229f15", "sync_policy": { "groups": [] } } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zone get { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "domain_root": "ceph-objectstore.rgw.meta:root", "control_pool": "ceph-objectstore.rgw.control", "gc_pool": "ceph-objectstore.rgw.log:gc", "lc_pool": "ceph-objectstore.rgw.log:lc", "log_pool": "ceph-objectstore.rgw.log", "intent_log_pool": "ceph-objectstore.rgw.log:intent", "usage_log_pool": "ceph-objectstore.rgw.log:usage", "roles_pool": "ceph-objectstore.rgw.meta:roles", "reshard_pool": "ceph-objectstore.rgw.log:reshard", "user_keys_pool": "ceph-objectstore.rgw.meta:users.keys", "user_email_pool": "ceph-objectstore.rgw.meta:users.email", "user_swift_pool": "ceph-objectstore.rgw.meta:users.swift", "user_uid_pool": "ceph-objectstore.rgw.meta:users.uid", "otp_pool": "ceph-objectstore.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "HDD-EC", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "GLACIER": { "data_pool": "ceph-objectstore.rgw.buckets.data.hdd" }, "STANDARD": {} # <------------- seems like this shouldn't be here? }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } }, { "key": "default-placement", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "ceph-objectstore.rgw.buckets.data" } }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "", "notif_pool": "ceph-objectstore.rgw.log:notif" } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin user info --uid=aad { "user_id": "aad", "display_name": "Anthony", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "aad", "access_key": "xxxx", "secret_key": "yyyyyy" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "HDD-EC", "default_storage_class": "GLACIER", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }

5 months, 2 weeks

2
1
0 0

Moving from ceph-ansible to cephadm and upgrading from pacific to octopus

by wodel youchi

Hi, I have an Openstack platform deployed with Yoga and ceph-ansible pacific on Rocky 8. Now I need to do an upgrade to Openstack zed with octopus on Rocky 9. This is the path of the upgrade I have traced - upgrade my nodes to Rocky 9 keeping Openstack yoga with ceph-ansible pacific. - convert ceph pacific from ceph-ansible to cephadm. - stop Openstack platform yoga - upgrade ceph pacific to octopus - upgrade Openstack yoga to zed. Any thoughts or guide lines to keep in mind and follow regarding ceph convertion and upgrade. Ps : on my ceph I have rbd, rgw and cephfs pools. Regards.

5 months, 2 weeks

1
0
0 0

About delete old bucket lifecycle

by VÔ VI

Hi community, I am have multiple bucket was delete but lifecycle of bucket still exist, how i can delete it with radosgw-admin, because user can't access to bucket for delete lifecycle. User for this bucket does not exist. root@ceph:~# radosgw-admin lc list [ { "bucket": ":r30203:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.8", "started": "Wed, 06 Dec 2023 10:43:55 GMT", "status": "COMPLETE" }, { "bucket": ":r30304:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.13", "started": "Wed, 06 Dec 2023 10:43:54 GMT", "status": "COMPLETE" }, { "bucket": ":ec3204cam04:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.1", "started": "Wed, 06 Dec 2023 10:44:30 GMT", "status": "COMPLETE" }, { "bucket": ":r30105:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.5", "started": "Wed, 06 Dec 2023 10:44:40 GMT", "status": "COMPLETE" }, { "bucket": ":r30303:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.14", "started": "Wed, 06 Dec 2023 10:44:40 GMT", "status": "COMPLETE" }, { "bucket": ":ec3201cam02:f3fec4b6-a248-4f3f-be75-b8055e61233a.56439.2", "started": "Wed, 06 Dec 2023 10:43:56 GMT", "status": "COMPLETE" }, Thanks to the community.

5 months, 2 weeks

2
2
0 0

Assistance Needed with Ceph Cluster Slow Ops Issue

by Peter

Dear all, I am reaching out regarding an issue with our Ceph cluster that has been recurring every six hours. Upon investigating the problem using the "ceph daemon dump_historic_slow_ops" command, I observed that the issue appears to be related to slow operations, specifically getting stuck at "Waiting for RW Locks." The wait times often range from one to two seconds. Our cluster use SAS SSD disks from Samsung for the storage pool in question. While these disks are of high quality and should provide sufficient speed, the problem persists. The slow ops occurrence is consistent every six hours. I would greatly appreciate any insights or suggestions you may have to address and resolve this issue. If there are specific optimizations or configurations that could improve the situation, please advise. below are some output: root@lasas003:~# ceph -v ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) "events": [ { "event": "initiated", "time": "2023-12-06T08:34:18.501644-0800", "duration": 0 }, { "event": "throttled", "time": "2023-12-06T08:34:18.501644-0800", "duration": 3.067e-06 }, { "event": "header_read", "time": "2023-12-06T08:34:18.501647-0800", "duration": 3.5429999999999998e-06 }, { "event": "all_read", "time": "2023-12-06T08:34:18.501650-0800", "duration": 9.3399999999999997e-07 }, { "event": "dispatched", "time": "2023-12-06T08:34:18.501651-0800", "duration": 3.2830000000000002e-06 }, { "event": "queued_for_pg", "time": "2023-12-06T08:34:18.501654-0800", "duration": 1.3819939990000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:19.883648-0800", "duration": 5.7980000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:19.883654-0800", "duration": 4.2484711649999998 }, { "event": "reached_pg", "time": "2023-12-06T08:34:24.132125-0800", "duration": 1.0667e-05 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:24.132136-0800", "duration": 2.1593527840000002 }, { "event": "reached_pg", "time": "2023-12-06T08:34:26.291489-0800", "duration": 3.292e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:26.291492-0800", "duration": 0.43918164700000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:26.730674-0800", "duration": 5.1529999999999996e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:26.730679-0800", "duration": 1.0531516869999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:27.783831-0800", "duration": 5.1329999999999998e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:27.783836-0800", "duration": 1.232525088 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.016361-0800", "duration": 3.844e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.016365-0800", "duration": 0.0051385700000000003 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.021503-0800", "duration": 4.7600000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.021508-0800", "duration": 0.0092808779999999994 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.030789-0800", "duration": 4.0690000000000003e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.030793-0800", "duration": 0.55757725499999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.588370-0800", "duration": 5.5060000000000003e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.588376-0800", "duration": 0.0064168929999999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.594793-0800", "duration": 7.0690000000000004e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.594800-0800", "duration": 0.0026404089999999998 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.597440-0800", "duration": 3.3440000000000001e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.597444-0800", "duration": 0.0051126670000000004 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.602556-0800", "duration": 5.0200000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.602561-0800", "duration": 0.0040569960000000002 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.606618-0800", "duration": 5.0989999999999998e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.606623-0800", "duration": 0.0068874100000000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.613511-0800", "duration": 1.4636e-05 }, { "event": "started", "time": "2023-12-06T08:34:29.613525-0800", "duration": 0.00028943699999999997 }, { "event": "done", "time": "2023-12-06T08:34:29.613815-0800", "duration": 11.112171102 } Thank you in advance for your assistance.

5 months, 2 weeks

2
3
0 0

EC Profiles & DR

by duluxoz

Hi All, Looking for some help/explanation around erasure code pools, etc. I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc. I created an Erasure Code Profile, etc: ~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~ My Crush Map is: ~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 } # rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit } # end crush map ~~~ Finally I create a pool: ~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~ So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs. However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again. So my question is: What did I do wrong with my config? Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it? Thanks in advance Cheers Dulux-Oz

5 months, 2 weeks

11
18
0 0

CLT Meeting minutes 2023-12-06

by Laura Flores

Closing the loop (blocked waiting for Neha's input): how are we using Gibba on a day-to-day basis? Is it only used for checking reef point releases? - To be discussed again next week, as Neha had a conflict [Nizam] http://old.ceph.com/pgcalc is not working anymore, is there any replacement for pgcalc page in the new ceph site? - Last attempt at a fix https://github.com/ceph/ceph.io/issues/265 - Nizam will take a look; the PR needs some CSS work - We may not even need this link due to the autoscaler, etc. - Perhaps add some kind of "banner" to the pgcalc page to make it clear that users should look to the autoscaler Update on 18.2.1 bluestore issue? - Fix has been raised; also not as serious as initially suspected - Fix is in testing -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

5 months, 2 weeks

2
1
0 0

ceph df reports incorrect stats

by Frank Schilder

Dear fellow cephers, we got a problem with ceph df: ceph df reports incorrect USED. It would be great if someone could look at this, if a ceph operator doesn't discover this issue, they might run out of space without noticing. This has been reported before but didn't get much attention: https://www.spinics.net/lists/ceph-users/msg74602.html https://www.spinics.net/lists/ceph-users/msg74630.html The symptom: STORED=USED in output of ceph df. All reports I know of are for octopus clusters, but I suspect newer versions are affected as well. I don't have a reproducer yet (still lacking a test cluster). Here is a correct usage report: ==> logs/health_231203.log <== --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.29 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 20.95 3.7 PiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 42.78 3.3 PiB Here is an incorrect one: ==> logs/health_231204.log <== --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.06 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.82 3.6 PiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.09 3.3 PiB That the first report is correct and not the second is supported by the output of ceph osd df tree, showing a use of 4.6PB in alignment with the first output of ceph df. Note that the date of the ceph osd df tree output is identical to the date of the incorrect ceph df output, hence, ceph osd df tree is *not* affected by this issue: ==> ceph osd df tree 231204 <=== SIZE RAW USE DATA OMAP META AVAIL NAME 12 PiB 4.6 PiB 4.6 PiB 2.2 TiB 19 TiB 7.5 PiB datacenter ContainerSquare 0 B 0 B 0 B 0 B 0 B 0 B room CON-161-A 12 PiB 4.6 PiB 4.6 PiB 2.2 TiB 19 TiB 7.5 PiB room CON-161-A1 In our case, the problem showed up out of nowhere. Here the log snippet for the time window within which the flip happened (compare the lines for con-fs2-data?-pools): ==> logs/health_231203.log <== ceph status/df/pool stats/health detail at 16:30:03: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 14h), 1279 in (since 2w) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.1 PiB / 13 PiB avail pgs: 25035 active+clean 29 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 215 MiB/s rd, 140 MiB/s wr, 2.34k op/s rd, 1.89k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 831 GiB 6.0 TiB 11.84 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.08 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.44 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.60 TOTAL 13 PiB 8.1 PiB 4.9 PiB 5.0 PiB 38.04 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 13 GiB 16.57k 38 GiB 0.03 39 TiB sr-rbd-data-one 2 4096 121 TiB 32.06M 108 TiB 48.08 88 TiB sr-rbd-one-stretch 3 160 262 GiB 68.81k 573 GiB 0.48 39 TiB con-rbd-meta-hpc-one 7 50 12 KiB 45 372 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 24 GiB 6.10k 24 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 137 TiB 35.95M 193 TiB 46.57 166 TiB con-fs2-meta1 12 512 554 GiB 76.76M 2.2 TiB 7.26 6.9 TiB con-fs2-meta2 13 4096 0 B 574.23M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 21.09 3.6 PiB con-fs2-data-ec-ssd 17 256 700 GiB 7.27M 706 GiB 2.44 22 TiB ms-rbd-one 18 256 805 GiB 210.92k 1.4 TiB 1.18 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 42.96 3.3 PiB sr-rbd-data-one-perf 20 4096 6.8 TiB 1.81M 20 TiB 57.09 5.1 TiB device_health_metrics 21 1 1.4 GiB 1.11k 4.2 GiB 0 39 TiB ceph status/df/pool stats/health detail at 16:30:10: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 14h), 1279 in (since 2w) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.1 PiB / 13 PiB avail pgs: 25035 active+clean 29 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 241 MiB/s rd, 174 MiB/s wr, 2.68k op/s rd, 2.34k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 830 GiB 6.0 TiB 11.84 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.08 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.44 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.60 TOTAL 13 PiB 8.1 PiB 4.9 PiB 5.0 PiB 38.04 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 13 GiB 16.57k 13 GiB 0.01 39 TiB sr-rbd-data-one 2 4096 92 TiB 32.06M 92 TiB 44.11 88 TiB sr-rbd-one-stretch 3 160 222 GiB 68.81k 222 GiB 0.19 39 TiB con-rbd-meta-hpc-one 7 50 6.9 KiB 45 6.9 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 23 GiB 6.10k 23 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 135 TiB 35.95M 135 TiB 37.88 166 TiB con-fs2-meta1 12 512 367 GiB 76.76M 367 GiB 1.28 6.9 TiB con-fs2-meta2 13 4096 0 B 574.23M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.82 3.6 PiB con-fs2-data-ec-ssd 17 256 515 GiB 7.27M 515 GiB 1.79 22 TiB ms-rbd-one 18 256 579 GiB 210.92k 579 GiB 0.48 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.09 3.3 PiB sr-rbd-data-one-perf 20 4096 6.9 TiB 1.81M 6.9 TiB 31.29 5.1 TiB device_health_metrics 21 1 1.2 GiB 1.11k 1.2 GiB 0 39 TiB For us, the issue disappeared after taking some OSDs in a second root down. These OSDs were moved there for draining, we use a second crush root for this purpose. Here the log snippet with the time window within which the back-flip to correct reporting happened: ==> logs/health_231205.log <== ceph status/df/pool stats/health detail at 17:42:58: cluster: health: HEALTH_WARN 1 osds down 24 hosts (12 osds) down 1 root (12 osds) down services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1267 up (since 19m), 1268 in (since 0.401448s) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.0 PiB / 13 PiB avail pgs: 25034 active+clean 31 active+clean+scrubbing+deep io: client: 118 MiB/s rd, 789 MiB/s wr, 1.75k op/s rd, 2.14k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 731 GiB 5.9 TiB 11.65 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.36 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.59 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.19 TOTAL 13 PiB 8.0 PiB 4.9 PiB 5.0 PiB 38.32 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 14 GiB 16.94k 14 GiB 0.01 39 TiB sr-rbd-data-one 2 4096 93 TiB 32.32M 93 TiB 44.29 88 TiB sr-rbd-one-stretch 3 160 222 GiB 68.81k 222 GiB 0.19 39 TiB con-rbd-meta-hpc-one 7 50 6.9 KiB 45 6.9 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 23 GiB 6.10k 23 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 135 TiB 36.08M 135 TiB 38.00 165 TiB con-fs2-meta1 12 512 367 GiB 76.81M 367 GiB 1.28 6.9 TiB con-fs2-meta2 13 4096 0 B 572.65M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.83 3.6 PiB con-fs2-data-ec-ssd 17 256 515 GiB 7.27M 515 GiB 1.78 22 TiB ms-rbd-one 18 256 579 GiB 210.92k 579 GiB 0.48 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.16 3.3 PiB sr-rbd-data-one-perf 20 4096 6.9 TiB 1.81M 6.9 TiB 31.07 5.1 TiB device_health_metrics 21 1 1.2 GiB 1.11k 1.2 GiB 0 39 TiB ceph status/df/pool stats/health detail at 17:43:04: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1267 up (since 19m), 1267 in (since 6s) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.0 PiB / 13 PiB avail pgs: 25035 active+clean 30 active+clean+scrubbing+deep io: client: 151 MiB/s rd, 840 MiB/s wr, 2.13k op/s rd, 2.10k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 731 GiB 5.9 TiB 11.65 hdd 13 PiB 7.7 PiB 4.8 PiB 4.8 PiB 38.42 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.59 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.19 TOTAL 13 PiB 8.0 PiB 4.9 PiB 5.0 PiB 38.37 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 14 GiB 16.94k 42 GiB 0.04 39 TiB sr-rbd-data-one 2 4096 122 TiB 32.32M 109 TiB 48.26 88 TiB sr-rbd-one-stretch 3 160 262 GiB 68.81k 573 GiB 0.48 39 TiB con-rbd-meta-hpc-one 7 50 11 KiB 45 368 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 24 GiB 6.10k 24 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 138 TiB 36.08M 193 TiB 46.69 165 TiB con-fs2-meta1 12 512 555 GiB 76.81M 2.2 TiB 7.26 6.9 TiB con-fs2-meta2 13 4096 0 B 572.65M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 21.09 3.6 PiB con-fs2-data-ec-ssd 17 256 700 GiB 7.27M 706 GiB 2.43 22 TiB ms-rbd-one 18 256 805 GiB 210.92k 1.4 TiB 1.18 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 43.01 3.3 PiB sr-rbd-data-one-perf 20 4096 6.8 TiB 1.81M 20 TiB 56.75 5.1 TiB device_health_metrics 21 1 1.4 GiB 1.11k 4.2 GiB 0 39 TiB This observation leads me to suspect that having multiple crush roots might be a reason for this observation. Our crush tree looks like this (OSDs removed), it has 3 different roots (BB, DTU and default): ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -78 106.92188 root BB -99 0 host bb-04 -102 0 host bb-05 -105 0 host bb-06 -325 0 host bb-06-old -108 0 host bb-07 -331 0 host bb-07-old -3 8.91016 host bb-08 -9 8.91016 host bb-09 -18 8.91016 host bb-10 -21 8.91016 host bb-11 -28 8.91016 host bb-12 -34 8.91016 host bb-13 -72 8.91016 host bb-14 -75 8.91016 host bb-15 -111 8.91016 host bb-16 -114 8.91016 host bb-17 -117 0 host bb-18 -142 0 host bb-19 -145 0 host bb-20 -241 0 host bb-21 -246 0 host bb-22 -251 8.91016 host bb-23 -256 8.91016 host bb-24 -151 0 host bb-office -40 14614.77832 root DTU -42 0 region Lyngby -41 14614.77832 region Risoe -50 12843.79590 datacenter ContainerSquare -56 0 room CON-161-A -57 12843.79590 room CON-161-A1 -11 1092.49060 host ceph-08 -13 1074.27673 host ceph-09 -23 1075.67920 host ceph-10 -15 1067.16492 host ceph-11 -25 1080.21912 host ceph-12 -83 1061.17480 host ceph-13 -85 1047.70276 host ceph-14 -87 1079.02820 host ceph-15 -136 1012.55048 host ceph-16 -139 1073.61475 host ceph-17 -261 1125.57202 host ceph-23 -262 1054.32227 host ceph-24 -148 885.49133 datacenter MultiSite -65 86.16304 host ceph-04 -67 101.50623 host ceph-05 -69 104.85805 host ceph-06 -71 96.39923 host ceph-07 -81 97.54230 host ceph-18 -94 98.48271 host ceph-19 -4 97.20181 host ceph-20 -64 99.77657 host ceph-21 -66 103.56137 host ceph-22 -49 885.49133 datacenter ServerRoom -55 885.49133 room SR-113 -65 86.16304 host ceph-04 -67 101.50623 host ceph-05 -69 104.85805 host ceph-06 -71 96.39923 host ceph-07 -81 97.54230 host ceph-18 -94 98.48271 host ceph-19 -4 97.20181 host ceph-20 -64 99.77657 host ceph-21 -66 103.56137 host ceph-22 -1 0 root default Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 months, 2 weeks

2
1
0 0

Re: After hardware failure tried to recover ceph and followed instructions for recovery using OSDS

by Eugen Block

And how about this? osd crush rm osd.14 Zitat von Manolis Daramas <mdaramas(a)motivian.com>: > Hi Eugen, > > I think that it cannot even be removed with ceph osd purge command, > see below: > > root@node01:~# ceph osd purge 14 --force --yes-i-really-mean-it > osd.14 does not exist > > root@node01:~# ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 2.05046 root default > -3 0.68349 host node01 > 0 hdd 0.14650 osd.0 up 1.00000 1.00000 > 4 hdd 0.04880 osd.4 up 1.00000 1.00000 > 8 hdd 0.04880 osd.8 up 1.00000 1.00000 > 10 hdd 0.04880 osd.10 up 1.00000 1.00000 > 14 hdd 0.39059 osd.14 DNE 0 > -5 0.68349 host node02 > 2 hdd 0.14650 osd.2 up 1.00000 1.00000 > 5 hdd 0.04880 osd.5 up 1.00000 1.00000 > 7 hdd 0.04880 osd.7 up 1.00000 1.00000 > 9 hdd 0.04880 osd.9 up 1.00000 1.00000 > 12 hdd 0.39059 osd.12 up 1.00000 1.00000 > -7 0.68349 host node03 > 1 hdd 0.14650 osd.1 up 1.00000 1.00000 > 3 hdd 0.04880 osd.3 up 1.00000 1.00000 > 6 hdd 0.04880 osd.6 up 1.00000 1.00000 > 11 hdd 0.04880 osd.11 up 1.00000 1.00000 > 13 hdd 0.39059 osd.13 up 1.00000 1.00000 > > The output of journalctl with --no-pager option is below: > > -- Logs begin at Thu 2023-11-23 06:16:17 EET, end at Tue 2023-12-05 > 16:01:49 EET. -- > Dec 05 15:23:16 node01 systemd[1]: Started Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: Dec 05 15:23:17 node01 bash[1969]: > debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: terminate called after throwing > an instance of 'std::invalid_argument' > Dec 05 15:23:17 node01 bash[1969]: what(): stoull > Dec 05 15:23:17 node01 bash[1969]: *** Caught signal (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: debug -290> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command assert hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -289> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command abort hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -288> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command leak_some_memory hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -287> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -286> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 1 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -285> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -284> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -283> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -282> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 2 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -281> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -280> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -279> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf reset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -278> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config show hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -277> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config help hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -276> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config set hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -275> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config unset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -274> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -273> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -272> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -271> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command injectargs hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -270> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log flush hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -269> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -268> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log reopen hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -267> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command dump_mempools hook 0x55be2168a068 > Dec 05 15:23:17 node01 bash[1969]: debug -266> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug -265> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug -264> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug -263> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > init /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -262> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > bind_and_listen /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -261> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 0 hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -260> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -259> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command git_version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -258> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command help hook 0x55be208de5b0 > Dec 05 15:23:17 node01 bash[1969]: debug -257> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command get_command_descriptions hook 0x55be208de5a0 > Dec 05 15:23:17 node01 bash[1969]: debug -256> > 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) > entry start > Dec 05 15:23:17 node01 bash[1969]: debug -255> > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug -254> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -253> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -252> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -251> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -250> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -249> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -248> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: > verify_sharding column families from rocksdb: [default] > Dec 05 15:23:17 node01 bash[1969]: debug -247> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug -246> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug -245> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug -244> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug -243> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug -242> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug -241> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug -240> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug -239> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug -238> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug -237> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -236> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -235> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -234> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -233> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug -232> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug -231> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug -230> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug -229> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -228> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -227> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -226> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -225> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -224> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug -223> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -222> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -221> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -220> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -219> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -218> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -217> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -216> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug -215> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug -214> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug -213> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -212> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -211> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -210> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -209> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -208> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -207> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -206> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug -205> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -204> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -203> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -202> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -201> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -200> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -199> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -198> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -197> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -196> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -195> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -194> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -193> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -192> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug -191> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug -190> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -189> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -188> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -187> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -186> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -185> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -184> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -183> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -182> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -181> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -180> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -179> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug -178> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -177> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug -176> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug -175> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -174> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug -173> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -172> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -171> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -170> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -169> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -168> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug -167> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -166> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug -165> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -164> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -163> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -162> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -161> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -160> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -159> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -158> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -157> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -156> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug -155> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -154> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -153> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -152> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -151> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -150> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -149> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -148> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -147> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug -146> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug -145> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug -144> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug -143> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug -142> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -141> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -140> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -139> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug -138> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug -137> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -136> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -135> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -134> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -133> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug -132> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -131> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -130> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug -129> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -128> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -127> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -126> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -125> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -124> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -123> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -122> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -121> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -120> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -119> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -118> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -117> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -116> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -115> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -114> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -113> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -112> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug -111> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug -110> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug -109> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug -108> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -107> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -106> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -105> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -104> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -103> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -102> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -101> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -100> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -99> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -98> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -97> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug -96> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug -95> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -94> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug -93> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug -92> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -91> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -90> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug -89> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug -88> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -87> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -86> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug -85> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug -84> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -83> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug -82> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -81> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -80> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug -79> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -78> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug -77> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -76> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -75> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -74> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -73> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -72> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -71> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -70> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -69> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -68> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug -67> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -66> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug -65> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -64> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -63> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -62> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug -61> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -60> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -59> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug -58> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug -57> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug -56> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -55> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug -54> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug -53> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug -52> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -51> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug -50> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -49> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug -48> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug -47> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug -46> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug -45> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -44> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -43> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -42> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -41> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -40> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -39> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -38> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -37> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -36> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -35> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -34> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -33> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -32> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -31> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -30> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -29> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -28> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -27> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -26> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -25> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -24> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -23> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -22> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -21> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -20> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -19> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -18> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -17> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -16> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -15> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -14> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -13> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -12> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -11> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -10> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -9> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -8> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -7> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -6> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -5> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01(a)-1(???).mds > e0 Unable to load 'last_metadata' > Dec 05 15:23:17 node01 bash[1969]: debug -4> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -3> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -2> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -1> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug 0> > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon > Dec 05 15:23:17 node01 bash[1969]: 0/10 monc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache > Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse > Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter > Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) > Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) > Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for > recent threads --- > Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket > Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 > Dec 05 15:23:17 node01 bash[1969]: max_new 10000 > Dec 05 15:23:17 node01 bash[1969]: log_file > Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: debug -290> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command assert hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -289> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command abort hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -288> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command leak_some_memory hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -287> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -286> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 1 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -285> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -284> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -283> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -282> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 2 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -281> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -280> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -279> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf reset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -278> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config show hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -277> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config help hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -276> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config set hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -275> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config unset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -274> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -273> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -272> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -271> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command injectargs hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -270> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log flush hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -269> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -268> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log reopen hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -267> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command dump_mempools hook 0x55be2168a068 > Dec 05 15:23:17 node01 bash[1969]: debug -266> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug -265> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug -264> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug -263> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > init /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -262> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > bind_and_listen /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -261> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 0 hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -260> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -259> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command git_version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -258> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command help hook 0x55be208de5b0 > Dec 05 15:23:17 node01 bash[1969]: debug -257> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command get_command_descriptions hook 0x55be208de5a0 > Dec 05 15:23:17 node01 bash[1969]: debug -256> > 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) > entry start > Dec 05 15:23:17 node01 bash[1969]: debug -255> > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug -254> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -253> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -252> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -251> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -250> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -249> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -248> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: > verify_sharding column families from rocksdb: [default] > Dec 05 15:23:17 node01 bash[1969]: debug -247> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug -246> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug -245> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug -244> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug -243> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug -242> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug -241> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug -240> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug -239> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug -238> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug -237> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -236> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -235> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -234> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -233> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug -232> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug -231> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug -230> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug -229> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -228> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -227> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -226> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -225> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -224> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug -223> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -222> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -221> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -220> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -219> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -218> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -217> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -216> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug -215> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug -214> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug -213> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -212> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -211> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -210> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -209> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -208> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -207> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -206> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug -205> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -204> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -203> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -202> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -201> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -200> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -199> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -198> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -197> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -196> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -195> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -194> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -193> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -192> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug -191> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug -190> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -189> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -188> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -187> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -186> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -185> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -184> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -183> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -182> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -181> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -180> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -179> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug -178> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -177> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug -176> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug -175> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -174> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug -173> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -172> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -171> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -170> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -169> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -168> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug -167> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -166> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug -165> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -164> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -163> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -162> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -161> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -160> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -159> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -158> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -157> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -156> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug -155> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -154> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -153> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -152> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -151> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -150> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -149> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -148> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -147> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug -146> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug -145> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug -144> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug -143> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug -142> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -141> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -140> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -139> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug -138> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug -137> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -136> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -135> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -134> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -133> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug -132> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -131> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -130> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug -129> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -128> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -127> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -126> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -125> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -124> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -123> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -122> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -121> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -120> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -119> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -118> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -117> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -116> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -115> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -114> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -113> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -112> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug -111> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug -110> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug -109> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug -108> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -107> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -106> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -105> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -104> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -103> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -102> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -101> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -100> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -99> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -98> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -97> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug -96> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug -95> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -94> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug -93> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug -92> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -91> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -90> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug -89> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug -88> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -87> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -86> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug -85> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug -84> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -83> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug -82> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -81> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -80> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug -79> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -78> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug -77> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -76> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -75> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -74> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -73> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -72> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -71> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -70> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -69> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -68> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug -67> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -66> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug -65> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -64> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -63> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -62> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug -61> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -60> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -59> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug -58> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug -57> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug -56> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -55> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug -54> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug -53> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug -52> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -51> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug -50> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -49> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug -48> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug -47> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug -46> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug -45> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -44> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -43> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -42> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -41> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -40> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -39> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -38> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -37> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -36> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -35> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -34> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -33> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -32> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -31> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -30> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -29> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -28> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -27> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -26> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -25> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -24> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -23> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -22> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -21> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -20> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -19> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -18> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -17> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -16> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -15> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -14> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -13> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -12> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -11> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -10> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -9> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -8> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -7> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -6> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -5> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01(a)-1(???).mds > e0 Unable to load 'last_metadata' > Dec 05 15:23:17 node01 bash[1969]: debug -4> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -3> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -2> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -1> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug 0> > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon > Dec 05 15:23:17 node01 bash[1969]: 0/10 monc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache > Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse > Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter > Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) > Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) > Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for > recent threads --- > Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket > Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 > Dec 05 15:23:17 node01 bash[1969]: max_new 10000 > Dec 05 15:23:17 node01 bash[1969]: log_file > /var/lib/ceph/crash/2023-12-05T13:23:17.739483Z_174a5e49-22df-4c2f-be14-4e88cc65b2f2/log > Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- > Dec 05 15:23:17 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: Main > process exited, code=exited, status=134/n/a > Dec 05 15:23:18 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: Failed > with result 'exit-code'. > Dec 05 15:23:29 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: > Scheduled restart job, restart counter is at 1. > Dec 05 15:23:29 node01 systemd[1]: Stopped Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:29 node01 systemd[1]: Started Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 0 load: jerasure load: lrc > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB SUMMARY > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB Session ID: > OQS561SAY9M8P46I122B > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: MANIFEST file: > MANIFEST-000231 size: 308 Bytes > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000232.log size: > 0 ; > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.env: 0x562241625a80 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.info_log: 0x562243b9a360 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_buffer_manager: 0x56224490e990 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000231 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x562243becfb0) > Dec 05 15:23:29 node01 bash[6375]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:29 node01 bash[6375]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:29 node01 bash[6375]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:29 node01 bash[6375]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:29 node01 bash[6375]: index_type: 0 > Dec 05 15:23:29 node01 bash[6375]: data_block_index_type: 0 > Dec 05 15:23:29 node01 bash[6375]: index_shortening: 1 > Dec 05 15:23:29 node01 bash[6375]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:29 node01 bash[6375]: hash_index_allow_collision: 1 > Dec 05 15:23:29 node01 bash[6375]: checksum: 1 > Dec 05 15:23:29 node01 bash[6375]: no_block_cache: 0 > Dec 05 15:23:29 node01 bash[6375]: block_cache: 0x562243c1f090 > Dec 05 15:23:29 node01 bash[6375]: block_cache_name: BinnedLRUCache > Dec 05 15:23:29 node01 bash[6375]: block_cache_options: > Dec 05 15:23:29 node01 bash[6375]: capacity : 536870912 > Dec 05 15:23:29 node01 bash[6375]: num_shard_bits : 4 > Dec 05 15:23:29 node01 bash[6375]: strict_capacity_limit : 0 > Dec 05 15:23:29 node01 bash[6375]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:29 node01 bash[6375]: block_cache_compressed: (nil) > Dec 05 15:23:29 node01 bash[6375]: persistent_cache: (nil) > Dec 05 15:23:29 node01 bash[6375]: block_size: 4096 > Dec 05 15:23:29 node01 bash[6375]: block_size_deviation: 10 > Dec 05 15:23:29 node01 bash[6375]: block_restart_interval: 16 > Dec 05 15:23:29 node01 bash[6375]: index_block_restart_interval: 1 > Dec 05 15:23:29 node01 bash[6375]: metadata_block_size: 4096 > Dec 05 15:23:29 node01 bash[6375]: partition_filters: 0 > Dec 05 15:23:29 node01 bash[6375]: use_delta_encoding: 1 > Dec 05 15:23:29 node01 bash[6375]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:29 node01 bash[6375]: whole_key_filtering: 1 > Dec 05 15:23:29 node01 bash[6375]: verify_compression: 0 > Dec 05 15:23:29 node01 bash[6375]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:29 node01 bash[6375]: format_version: 4 > Dec 05 15:23:29 node01 bash[6375]: enable_index_compression: 1 > Dec 05 15:23:29 node01 bash[6375]: block_align: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.paranoid_file_checks: Dec 05 15:23:29 node01 bash[6375]: > debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:29 > [Text gekürzt]

5 months, 2 weeks

3
6
0 0

2024

2023

2022

2021

2020

2019

ceph-users December 2023