March 2023 - ceph-users - lists.ceph.io

handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
by Arvid Picciani 14 Mar '23

14 Mar '23

since quincy i'm randomly getting authentication issues from clients to osds. symptom is qemu hangs, but when it happens, i can reproduce it using: > ceph tell osd.\* version some - but only some - osds will never respond, but only to clients on _some_ hosts. the client gets stuck in a loop with this error > 2023-03-14T10:09:38.492+0100 7f38f5d95700 1 --2- 10.180.10.36:0/329477069 >> [v2:10.180.10.24:6810/697584,v1:10.180.10.24:6811/697584] conn(0x7f38f0107990 0x7f38f0107d60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) restarting the affected OSD helps for a few hours. in the osd log i see only > 2023-03-14T09:27:27.801+0000 7fb79020a700 10 osd.4 114909 ms_handle_authentication session 0x55880cd58b40 client.admin h as caps osdcap[grant(*)] 'allow *' > 2023-03-14T09:27:27.805+0000 7fb781a3c700 2 osd.4 114909 ms_handle_reset con 0x55880a7fec00 session 0x55880cd58b40 searching for this issue gives me people whos mon is dead, but i dont think "tell" is supposed to go through mon, beyond the initial listing, which succeeds. but here's the full auth log from mon anyway if it helps: 2023-03-14T09:34:48.847+0000 7fcc8a5c7700 10 In get_auth_session_handler for protocol 0 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 start_session entity_name=client.admin global_id=6751719 is_new_global_id=1 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server client.admin: start_session server_challenge 20aa2b96857f41cf 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 start_session entity_name=client.admin global_id=6751722 is_new_global_id=1 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server client.admin: start_session server_challenge 6066dd1200ddc855 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server client.admin: handle_request get_auth_session_key for client.admin 2023-03-14T09:34:48.847+0000 7fcc84dbc700 20 cephx server client.admin: checking key: req.key=92ed7ea281e9ac0c expected_key=92ed7ea281e9ac0c 2023-03-14T09:34:48.847+0000 7fcc84dbc700 20 cephx server client.admin: checking old_ticket: secret_id=0 len=0, old_ticket_may_be_omitted=0 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server client.admin: new global_id 6751719 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx: build_service_ticket_reply encoding 1 tickets with secret REDACTED== 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx: build_service_ticket service auth secret_id 160 ticket_info.ticket.name=client.admin ticket.global_id 6751719 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server client.admin: handle_request get_auth_session_key for client.admin 2023-03-14T09:34:48.847+0000 7fcc865bf700 20 cephx server client.admin: checking key: req.key=3c1f6182caf84073 expected_key=3c1f6182caf84073 2023-03-14T09:34:48.847+0000 7fcc865bf700 20 cephx server client.admin: checking old_ticket: secret_id=0 len=0, old_ticket_may_be_omitted=0 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server client.admin: new global_id 6751722 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx: build_service_ticket_reply encoding 1 tickets with secret REDACTED== 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx: build_service_ticket service auth secret_id 160 ticket_info.ticket.name=client.admin ticket.global_id 6751722 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 start_session entity_name=client.admin global_id=6751725 is_new_global_id=1 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: start_session server_challenge 22fa068f8da1fb28 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: handle_request get_auth_session_key for client.admin 2023-03-14T09:34:48.851+0000 7fcc84dbc700 20 cephx server client.admin: checking key: req.key=fc7fdedb8e669347 expected_key=fc7fdedb8e669347 2023-03-14T09:34:48.851+0000 7fcc84dbc700 20 cephx server client.admin: checking old_ticket: secret_id=0 len=0, old_ticket_may_be_omitted=0 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: new global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket_reply encoding 1 tickets with secret REDACTED== 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket service auth secret_id 160 ticket_info.ticket.name=client.admin ticket.global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: adding key for service mon 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: adding key for service osd 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: adding key for service mgr 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket_reply encoding 3 tickets with secret REDACTED== 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket service mon secret_id 11455 ticket_info.ticket.name=client.admin ticket.global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket service osd secret_id 11455 ticket_info.ticket.name=client.admin ticket.global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket service mgr secret_id 11455 ticket_info.ticket.name=client.admin ticket.global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc865bf700 0 mon.mon-yca4ceph@0(leader) e8 handle_command mon_command({"prefix": "osd ls"} v 0) v1 2023-03-14T09:34:48.851+0000 7fcc865bf700 0 log_channel(audit) log [DBG] : from='client.? 10.180.10.36:0/329477069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch -- +4916093821054

1 0

Mixed mode ssd and hdd issue
by xadhoom76＠gmail.com 14 Mar '23

14 Mar '23

Hi, we have a cluster with 3 nodes . Each node has 4 HDD and 1 SSD We would like to have a pool only on ssd and a pool only on hdd, using class feature. here is the setup # buckets host ceph01s3 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily id -21 class ssd # do not change unnecessarily # weight 34.561 alg straw2 hash 0 # rjenkins1 item osd.0 weight 10.914 item osd.5 weight 10.914 item osd.8 weight 10.914 item osd.9 weight 1.819 } host ceph02s3 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily id -22 class ssd # do not change unnecessarily # weight 34.561 alg straw2 hash 0 # rjenkins1 item osd.1 weight 10.914 item osd.3 weight 10.914 item osd.7 weight 10.914 item osd.10 weight 1.819 } host ceph03s3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily id -23 class ssd # do not change unnecessarily # weight 34.561 alg straw2 hash 0 # rjenkins1 item osd.2 weight 10.914 item osd.4 weight 10.914 item osd.6 weight 10.914 item osd.11 weight 1.819 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily id -24 class ssd # do not change unnecessarily # weight 103.683 alg straw2 hash 0 # rjenkins1 item ceph01s3 weight 34.561 item ceph02s3 weight 34.561 item ceph03s3 weight 34.561 } # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step take default class hdd step set_chooseleaf_tries 5 step set_choose_tries 100 step chooseleaf indep 0 type host step emit } rule erasure2_1 { id 2 type erasure min_size 3 max_size 3 step take default class hdd step set_chooseleaf_tries 5 step set_choose_tries 100 step chooseleaf indep 0 type host step emit } rule erasure-pool.meta { id 3 type erasure min_size 3 max_size 3 step take default class hdd step set_chooseleaf_tries 5 step set_choose_tries 100 step chooseleaf indep 0 type host step emit } rule erasure-pool.data { id 4 type erasure min_size 3 max_size 3 step take default class hdd step set_chooseleaf_tries 5 step set_choose_tries 100 step chooseleaf indep 0 type host step emit } rule replicated_rule_ssd { id 5 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 0 type host step emit } # end crush map pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1669 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 5 'Datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 2749 lfor 0/0/321 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 7 'erasure-pool.data' erasure profile k2m1 size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 128 pgp_num 126 pgp_num_target 128 autoscale_mode on last_change 2780 lfor 0/0/1676 flags hashpspool,ec_overwrites stripe_width 8192 application cephfs pool 8 'erasure-pool.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 344 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 9 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 592 flags hashpspool stripe_width 0 application rgw pool 10 'brescia-ovest.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 595 flags hashpspool stripe_width 0 application rgw pool 11 'brescia-ovest.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 597 flags hashpspool stripe_width 0 application rgw pool 12 'brescia-ovest.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 719 lfor 0/719/717 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 13 'brescia-ovest.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 845 lfor 0/845/843 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 14 'brescia-ovest.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 739 flags hashpspool stripe_width 0 application rgw pool 15 'brescia-ovest.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 849 flags hashpspool stripe_width 0 application rgw pool 17 'ssd_pool' replicated size 3 min_size 2 crush_rule 5 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 2774 lfor 0/0/2653 flags hashpspool stripe_width 0 application rbd rados bench -p ssd_pool 10 write --no-cleanup Object prefix: benchmark_data_ceph01s3.itservicenet.net_268 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 32 16 63.9957 64 0.937457 0.917873 2 16 48 32 63.9936 64 0.882778 0.898646 3 16 64 48 63.9929 64 0.949337 0.903045 4 16 81 65 64.9925 68 0.515819 0.897401 5 16 97 81 64.7919 64 1.00908 0.918797 6 16 114 98 65.3248 68 0.99787 0.922301 7 16 130 114 65.1339 64 0.794492 0.903341 8 16 147 131 65.4909 68 0.770237 0.892833 9 16 173 157 69.7677 104 0.976005 0.878237 10 16 195 179 71.5891 88 0.755363 0.869603 That is very poor ! Why ? Thanks

2 1

bucket.sync-status mdlogs not remove
by Bernie(Chanyeol) Yoon 14 Mar '23

14 Mar '23

hi , all I m using rgw multisite with ceph 17.2.5 deployed with rook. A number of bucket.sync-status mdlogs with names of buckets deleted during maintenance were found. test env ) bash-4.4$ rados -p master.rgw.log ls | grep bucket.sync-status | grep test1 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:5 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:4 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:1 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:7 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:2 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:10 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:6 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:3 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:0 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:9 bucket.sync-status.a788ebed-10a9-48da-8fd4-709323da68e7:test1:8da53b60-0940-46e1-a821-551347d82d2c.16016.2:8 bash-4.4$ radosgw-admin bucket list [ "rook-ceph-bucket-checker-94e835a7-6356-46bf-a90c-591b23b15959", "230314", "cyyoon", "test23031303" ] How to delete the bucket.sync-status mdlog of the deleted bucket in this situation? Should I proceed with mdlog trim ? As these issues accumulated in the prod environment, a large omap issue occurred in the log pool. root@osd-001:~# radosgw-admin log list| grep [DELETED BUCKET NAME] | wc -l 25636 Any other ideas as to what might be causing this, or anything else we could try to help diagnose or fix this? Thanks in advance!

1 0

Undo "radosgw-admin bi purge"
by Robert Sander 13 Mar '23

13 Mar '23

Hi, There is an operation "radosgw-admin bi purge" that removes all bucket index objects for one bucket in the rados gateway. What is the undo operation for this? After this operation the bucket cannot be listed or removed any more. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

4 9

User + Dev Meeting happening this week Thursday!
by Laura Flores 13 Mar '23

13 Mar '23

Hi Ceph Users, The User + Dev Meeting is happening this *Thursday, March 16th at 10am EST *(see extra meeting details below). If you have any topics you'd like to discuss, please add them to the etherpad: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes One of the topics we wish to discuss is whether any users would be willing to help with early Reef testing after the RC comes out. Thanks, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time conversions: UTC: Thursday, March 16, 14:00 UTC Mountain View, CA, US: Thursday, March 16, 7:00 PDT Phoenix, AZ, US: Thursday, March 16, 7:00 MST Denver, CO, US: Thursday, March 16, 8:00 MDT Huntsville, AL, US: Thursday, March 16, 9:00 CDT Raleigh, NC, US: Thursday, March 16, 10:00 EDT London, England: Thursday, March 16, 14:00 GMT Paris, France: Thursday, March 16, 15:00 CET Helsinki, Finland: Thursday, March 16, 16:00 EET Tel Aviv, Israel: Thursday, March 16, 16:00 IST Pune, India: Thursday, March 16, 19:30 IST Brisbane, Australia: Friday, March 17, 0:00 AEST Singapore, Asia: Thursday, March 16, 22:00 +08 Auckland, New Zealand: Friday, March 17, 3:00 NZDT -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Can't install cephadm on HPC
by zyz 13 Mar '23

13 Mar '23

Hi: I encountered a problem when I install cephadm on Huawei Cloud EulerOS. When enter the following command, it raise an error. What should I do? >> ./cephadm add-repo --release quincy << ERROR: Distro hce version 2.0 not supported

2 1

quincy: test cluster on nvme: fast write, slow read
by Arvid Picciani 12 Mar '23

12 Mar '23

Hi, Doing some lab tests to understand why ceph isnt working for us, and here's the first puzzle: setup: A completely fresh quincy cluster, 64 core EPYC 7713, 2 nvme drives > ceph osd crush rule create-replicated osd default osd ssd > ceph osd pool create rbd replicated osd --size 2 > dd if=/dev/rbd0 of=/tmp/testfile status=progress bs=4M count=1000 4194304000 bytes (4.2 GB, 3.9 GiB) copied, 7.0152 s, 598 MB/s > dd of=/dev/rbd0 if=/tmp/testfile status=progress bs=4M count=1000 4194304000 bytes (4.2 GB, 3.9 GiB) copied, 3.82156 s, 1.1 GB/s write performance is 1/3 of raw nvme, which i suppose is expected (not very good tho) but why is read performance so bad? top shows only one core is being utilized at 40% cpu. it can't be network either, since this is all localhost. thanks Arvid -- +4916093821054

1 0

pg wait too long when osd restart
by yite gu 10 Mar '23

10 Mar '23

Hi all, osd_heartbeat_grace = 20 and osd_pool_default_read_lease_ratio = 0.8 by default, so, pg will wait 16s when osd restart in the worst case. This wait time is too long, client i/o can not be unacceptable. I think adjusting the osd_pool_default_read_lease_ratio to lower is a good way. Have any good suggestions about reduce pg wait time？ Best Regard Yite Gu

1 0

restoring ceph cluster from osds
by Ben 09 Mar '23

09 Mar '23

Hi, I ended up with having whole set of osds to get back original ceph cluster. I figured out to make the cluster running. However, it's status is something as below: bash-4.4$ ceph -s cluster: id: 3f271841-6188-47c1-b3fd-90fd4f978c76 health: HEALTH_WARN 7 daemons have recently crashed 4 slow ops, oldest one blocked for 35077 sec, daemons [mon.a,mon.b] have slow ops. services: mon: 3 daemons, quorum a,b,d (age 9h) mgr: b(active, since 14h), standbys: a osd: 4 osds: 0 up, 4 in (since 9h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: All osds are down. I checked the osds logs and attached with this. Please help and I wonder if it's possible to get the cluster back. I have some backup for monitor's data. Till now I haven't restore that in the course. Thanks, Ben

2 5

Trying to throttle global backfill
by Rice, Christian 09 Mar '23

09 Mar '23

I have a large number of misplaced objects, and I have all osd settings to “1” already: sudo ceph tell osd.\* injectargs '--osd_max_backfills=1 --osd_recovery_max_active=1 --osd_recovery_op_priority=1' How can I slow it down even more? The cluster is too large, it’s impacting other network traffic 😉

2 2

radosgw - octopus - 500 Bad file descriptor on upload
by Boris Behrens 09 Mar '23

09 Mar '23

Hi, we've observed 500er errors on uploading files to a single bucket, but the problem went away after around 2 hours. We've checked and saw the following error message: 2023-03-08T17:55:58.778+0000 7f8062f15700 0 WARNING: set_req_state_err err_no=125 resorting to 500 2023-03-08T17:55:58.778+0000 7f8062f15700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Bad file descriptor 2023-03-08T17:55:58.778+0000 7f8062f15700 1 ====== req done req=0x7f81d0189700 op status=-125 http_status=500 latency=65003730017ns ====== 2023-03-08T17:55:58.778+0000 7f8062f15700 1 beast: 0x7f81d0189700: IPADDRESS - - [2023-03-08T17:55:58.778961+0000] "PUT /BUCKET/OBJECT HTTP/1.1" 500 57 - "aws-sdk-php/3.257.11 OS/Linux/5.15.0-60-generic lang/php/8.2.3 GuzzleHttp/7" - It only happened to a single bucket over a period of 1-2 hours (around 300 requests). In the same time we've had >20k PUT requests the were working fine on other buckets. This error also seem to happen to other buckets, but only very sporadically. Did someone encounter this issue or knows what it could be? Cheers Boris

1 0

LRC k6m3l3, rack outage and availability
by steve.bakerx1＠gmail.com 09 Mar '23

09 Mar '23

Hi, currently we are testing LRC codes and I got a cluster setup with 3 racks and 4 hosts in each of those. What I want to achieve is to have a storage efficient erasure code (<=200%) and also availability during a rack outage. In (my) theory, that should have worked with the LRC k6m3l3 having a crush-locality=rack and a crush-failure domain=host. But when I tested it, the PGs of the pool all go in the "down" state. So, when we've got k=6 data chunks and m=3 coding chunks, the data should be reconstructable with 6 of these 9 objects. With l=3, LRC splits these 9 objects in 3 groups of 3 objects and creates one additional locality-chunk per group. We now got 3 groups of 4 objects. These 3 groups get distributed over the 3 racks, the 4 objects of each group get distributed over the 4 hosts of a rack. I thought that on a full rack outage, the 6 remaining k/m chunks on the other 2 racks should still be enough to keep up the availability and the cluster could proceed in a degraded state. But it does not, so I guess my thinking is wrong :) I wonder what's the reason for this, is it maybe some min_size setting ? The default min_size of this pool becomes 7 - I also changed that to 6 (yes, one shouldn't do that in productrion I think) but got the same result. Below I've added some details about the cluster, pool creation and pg dumps. Any ideas ? Can s.o. explain why this does not work or give another solution how to achieve the described specifications? Thx! ############ Ceph version: ############ ceph --version ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) ################# Creation of the pool: ################# ceph osd erasure-code-profile set lrc_individual_profile plugin=lrc k=6 m=3 l=3 crush-failure-domain=host crush-locality=rack crush-root=default ceph osd pool create lrc_individual_pool 1024 1024 erasure lrc_individual_profile ceph osd pool set lrc_individual_pool pg_num 1024 ceph osd pool set lrc_individual_pool pg_num_min 1024 ceph osd pool set lrc_individual_pool pgp_num 1024 ceph osd pool set lrc_individual_pool pg_autoscale_mode warn ceph osd pool set lrc_individual_pool bulk true ################## Resulting pool details: ################## ceph osd pool ls detail pool 72 'lrc_individual_pool' erasure profile lrc_individual_profile size 12 min_size 7 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 140484 flags hashpspool,bulk stripe_width 24576 pg_num_min 1024 ceph osd pool get lrc_individual_pool all size: 12 min_size: 7 pg_num: 1024 pgp_num: 1024 crush_rule: lrc_individual_pool hashpspool: true allow_ec_overwrites: false nodelete: false nopgchange: false nosizechange: false write_fadvise_dontneed: false noscrub: false nodeep-scrub: false use_gmt_hitset: 1 erasure_code_profile: lrc_individual_profile fast_read: 0 pg_autoscale_mode: warn pg_num_min: 1024 bulk: true ################# Resulting crush rule: ################# ceph osd crush rule dump lrc_individual_pool { "rule_id": 1, "rule_name": "lrc_individual_pool", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 12, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_indep", "num": 3, "type": "rack" }, { "op": "chooseleaf_indep", "num": 4, "type": "host" }, { "op": "emit" } ] } ############################ Ceph status after the rack outage: ############################ cluster: id: ... health: HEALTH_WARN 96 osds down 4 hosts (96 osds) down 1 rack (96 osds) down Reduced data availability: 1024 pgs inactive, 1024 pgs down services: mon: 3 daemons, quorum ...,...,... (age 4d) mgr: ...(active, since 4d), standbys: ...,... osd: 288 osds: 192 up (since 116s), 288 in (since 21h) data: pools: 2 pools, 1025 pgs objects: 291 objects, 0 B usage: 199 GiB used, 524 TiB / 524 TiB avail pgs: 99.902% pgs not active 1024 down 1 active+clean ################# Section of pg dump: ################# 72.32 0 0 0 0 0 0 0 0 0 0 down 2023-03-08T09:04:02.992141+0100 0'0 140549:52 [NONE,NONE,NONE,NONE,246,116,170,275,112,41,238,40] 246 [NONE,NONE,NONE,NONE,246,116,170,275,112,41,238,40] 246 0'0 2023-03-08T08:48:49.712787+0100 0'0 2023-03-08T08:48:49.712787+0100 0 72.33 0 0 0 0 0 0 0 0 0 0 down 2023-03-08T09:04:02.988083+0100 0'0 140549:134 [36,263,162,73,NONE,NONE,NONE,NONE,99,155,74,282] 36 [36,263,162,73,NONE,NONE,NONE,NONE,99,155,74,282] 36 0'0 2023-03-08T08:48:49.712787+0100 0'0 2023-03-08T08:48:49.712787+0100 0 72.31 0 0 0 0 0 0 0 0 0 0 down 2023-03-08T09:04:06.225241+0100 0'0 140549:136 [116,259,194,170,84,52,98,198,NONE,NONE,NONE,NONE] 116 [116,259,194,170,84,52,98,198,NONE,NONE,NONE,NONE] 116 0'0 2023-03-08T08:48:49.712787+0100 0'0 2023-03-08T08:48:49.712787+0100 0 ############## Single pg query: ############## ceph pg 72.33 query { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "down", "epoch": 140550, "up": [ 36, 263, 162, 73, 2147483647, 2147483647, 2147483647, 2147483647, 99, 155, 74, 282 ], "acting": [ 36, 263, 162, 73, 2147483647, 2147483647, 2147483647, 2147483647, 99, 155, 74, 282 ], "info": { "pgid": "72.33s0", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 140477, "epoch_pool_created": 140477, "last_epoch_started": 140516, "last_interval_started": 140515, "last_epoch_clean": 140516, "last_interval_clean": 140515, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 140538, "same_interval_since": 140538, "same_primary_since": 140477, "last_scrub": "0'0", "last_scrub_stamp": "2023-03-08T08:48:49.712787+0100", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2023-03-08T08:48:49.712787+0100", "last_clean_scrub_stamp": "2023-03-08T08:48:49.712787+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": 135, "reported_epoch": 140550, "state": "down", "last_fresh": "2023-03-08T09:07:15.104685+0100", "last_change": "2023-03-08T09:04:02.988083+0100", "last_active": "2023-03-08T09:04:02.337985+0100", "last_peered": "2023-03-08T09:04:01.288586+0100", "last_clean": "2023-03-08T09:04:01.288586+0100", "last_became_active": "2023-03-08T08:57:08.464085+0100", "last_became_peered": "2023-03-08T08:57:08.464085+0100", "last_unstale": "2023-03-08T09:07:15.104685+0100", "last_undegraded": "2023-03-08T09:07:15.104685+0100", "last_fullsized": "2023-03-08T09:07:15.104685+0100", "mapping_epoch": 140538, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 140477, "last_epoch_clean": 140516, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "2023-03-08T08:48:49.712787+0100", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2023-03-08T08:48:49.712787+0100", "last_clean_scrub_stamp": "2023-03-08T08:48:49.712787+0100", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 36, 263, 162, 73, 2147483647, 2147483647, 2147483647, 2147483647, 99, 155, 74, 282 ], "acting": [ 36, 263, 162, 73, 2147483647, 2147483647, 2147483647, 2147483647, 99, 155, 74, 282 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 173, 219, 236, 253 ], "up_primary": 36, "acting_primary": 36, "purged_snaps": [] }, "empty": 1, "dne": 0, "incomplete": 0, "last_epoch_started": 140516, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [], "recovery_state": [ { "name": "Started/Primary/Peering/Down", "enter_time": "2023-03-08T09:04:02.988076+0100", "comment": "not enough up instances of this PG to go active" }, { "name": "Started/Primary/Peering", "enter_time": "2023-03-08T09:04:02.987998+0100", "past_intervals": [ { "first": "140515", "last": "140537", "all_participants": [ { "osd": 36, "shard": 0 }, { "osd": 73, "shard": 3 }, { "osd": 74, "shard": 10 }, { "osd": 99, "shard": 8 }, { "osd": 155, "shard": 9 }, { "osd": 162, "shard": 2 }, { "osd": 173, "shard": 7 }, { "osd": 219, "shard": 6 }, { "osd": 236, "shard": 5 }, { "osd": 253, "shard": 4 }, { "osd": 263, "shard": 1 }, { "osd": 282, "shard": 11 } ], "intervals": [ { "first": "140515", "last": "140536", "acting": "36(0),73(3),74(10),99(8),155(9),162(2),173(7),219(6),236(5),253(4),263(1),282(11)" } ] } ], "probing_osds": [ "36(0)", "73(3)", "74(10)", "99(8)", "155(9)", "162(2)", "263(1)", "282(11)" ], "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 173, 219, 236, 253 ], "peering_blocked_by": [ { "osd": 173, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 219, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 236, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 253, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] }, { "name": "Started", "enter_time": "2023-03-08T09:04:02.987942+0100" } ], "agent_state": {} }

2 1

Error deploying Ceph Qunicy using ceph-ansible 7 on Rocky 9
by wodel youchi 09 Mar '23

09 Mar '23

Hi, I am trying to deploy Ceph Quincy using ceph-ansible on Rocky9. I am having some problems and I don't know where to search for the reason. PS : I did the same deployment on Rocky8 using ceph-ansible for the Pacific version on the same hardware and it worked perfectly. I have 03 controllers nodes : mon, mgr, mdss and rgws and 27 osd nodes : with 04 nvme disks (osd) each I am using a 10Gb network with jumbo frames. The deployment starts with no issues, the 03 monitors are created correctly, then the 03 managers are created, after that the OSD are prepared and formatted, until here everything is working fine, but when the "*wait for all osd to be up*" task is launched, which means starting all OSDs containers in all OSD nodes, things go south, the monitors become out of quorum, ceph -s takes a lot of time to respond and not all OSDs are being activated, and the deployment fails at the end. cluster 2023-03-06T12:00:26.431947+0100 mon.controllera (mon.0) 3864 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum controllera,controllerc cluster 2023-03-06T12:00:26.431953+0100 mon.controllera (mon.0) 3865 : cluster [WRN] mon.controllerb (rank 1) addr [v2: 20.1.0.27:3300/0,v1:20.1.0.27:6789/0] is down (out of quorum) The monitor container in 2 of my controllers nodes stays at 100% of cpu utilization. CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 068e4e55f299 ceph-mon-controllera 99.91% 58.12MiB / 376.1GiB 0.02% 0B / 0B 122MB / 85.3MB 28 <----------------- 87730f89420d ceph-mgr-controllera 0.32% 408.2MiB / 376.1GiB 0.11% 0B / 0B 181MB / 0B 35 Could that be a resource problem? the monitor containers do not have enough resources CPU, RAM, ...etc to handle all the OSDs that are being started? If yes, how may I find this? thanks in advance. Regards.

2 1

Difficulty with rbd-mirror on different networks.
by Murilo Morais 08 Mar '23

08 Mar '23

Good evening everyone. I'm having trouble with rbd-mirror. In test environment I have the following scenario: DC1: public_network: 172.20.0.0/24, 192.168.0.0/24 --mon-ip 172.20.0.1 ip: 192.168.0.1 DC2: public_network: 172.21.0.0/24, 192.168.0.0/24 --mon-ip 172.21.0.1 ip 192.168.0.2 If I add the peer as being from the 192.168.0.0/24 network, the mirror does not work, both machines see each other, if I add a machine within the same block as the other (the first public_network network of each), that is, DC1 with 172.21.0.2/24 and DC2 with 172.20.0.2/24, the mirror works. Why can't I make the mirror work on the 192.168.0.0/24 network? Thank you very much in advance.

1 0

Dashboard for Object Servers using wrong hostname
by Wyll Ingersoll 08 Mar '23

08 Mar '23

I have an ochestrated (cephadm) ceph cluster (16.2.11) with 2 radosgw services on 2 separate hosts without HA (i.e. no ingress/haproxy in front). Both of the rgw servers use SSL and have a properly signed certificate. We can access them with standard S3 tools like s3cmd, cyberduck, etc. The problem seems to be that the the Ceph mgr dashboard fails to access the RGW API because it uses the shortname "gw01" instead of the FQDN "gw01.domain.com" when forming the S3 signature which makes the S3 signature check fail and we get the following error: Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","RequestId":"tx00000521ceca28974e94b-006408e' b'f93-454bbb4e-default","HostId":"454bbb4e-default-default"}') It seems that the ceph mgr (which we have restarted several times) uses just the short hostname from the inventory and I don't see how to tell it to use the FQDN. Neither is it possible to configure the RGW to listen on an alternate non-SSL port on the cluster private network since the service spec for RGW only allows to set the rgw_frontend_port and rgw_frontend_type, but not the full frontend spec (which would allow for multiple listeners). When we did have HA (haproxy) ingress configured, we ran into issues with the user clients getting lots of 503 errors due to some interaction between the RGW and the haproxy so we gave up on that config and now talk directly to the RGW over SSL which is working well. Any suggestions? thanks, Wyllys Ingersoll

1 0

Problem with cephadm and deploying 4 ODSs on nvme Storage
by claas.goltz＠contact-software.com 08 Mar '23

08 Mar '23

Hi Community, currently i’m installing an nvme only storage cluster with cephadm from scratch (v.17.2.5). Everything works fine. Each of my nodes (6) has 3 enterprise nvme’s with 7TB capacity. At the beginning I only installed one OSD per nvme, now I want to use four instead of one but I’m struggling with that. Frist of all I set the following option in my cluster: ceph orch apply osd --all-available-devices --unmanaged=true As I understand this option should prevent cephadm to automatically fetch new, available disks and deploying OSD daemons. But that seems not to work. If I delete and purge my OSD and zapping the disk with ceph orch device zap ceph-nvme01 /dev/nvme2n1 –force the disk came available for the cluster and seconds later it deploys the same OSD ID than it has before. I checked that the old OSD was completely removed and the docker container was not started. My next try was to set: ceph orch host label add ceph-nvme01 _no_schedule purge the OSD zapping the disk and following: ceph orch daemon add osd ceph-nvme01:/dev/nvme2n1,osds_per_device=4 removing the _no_schedule flag and again: the old OSD was recreated and not 4. So where is my mistake? Thank you!

3 3

Upgrade problem from 1.6 to 1.7
by Oğuz Yarımtepe 08 Mar '23

08 Mar '23

Hi, I have a cluster with rook operator running the ceph version 1.6 and upgraded first rook operator and then the ceph cluster definition. Everything was fine, every component except from osds are upgraded. Below is the reason of OSD not being upgraded: not updating OSD 1 on node "some-node-name". node no longer exists in the storage spec. if the user wishes to remove OSDs from the node, they must do so manually. Rook will not remove OSDs from nodes that are removed from the storage spec in order to prevent accidental data loss Any idea or anyone had seen it before? Regards.

2 1

s3 lock api get-object-retention
by garcetto 08 Mar '23

08 Mar '23

good morning, i am using ver 17 but cannot have s3 lock get-object-retention working: $ aws --profile=user01 --endpoint-url=http://x.x.x.x:80 s3api list-object-versions --bucket demo-compliance { "ETag": "\"b8d6acf7d330d241a8ef851694365b94\"", "Size": 32, "StorageClass": "STANDARD", "Key": "testme.txt", "VersionId": "M632.ygI0AaMm8PXYXHrPNl3Y2aLOU6", "IsLatest": true, "LastModified": "2023-03-08T10:58:00.257Z", "Owner": { "DisplayName": "user01", "ID": "user01" } $ aws --profile=user01 --endpoint-url=http://x.x.x.x:80 s3api get-object-retention --bucket demo-compliance --key "testme.txt" An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectRetention operation: Unknown thank you.

1 0

user and bucket not sync ( permission denied )
by Guillaume Morin 07 Mar '23

07 Mar '23

Hello, i have an issue about my multisite configuration. pacific 16.2.9 My problem: i have a permission denied on the the master zone when i use the command below. $ radosgw-admin sync status realm 8df19226-a200-48fa-bd43-1491d32c636c (myrealm) zonegroup 29592d75-224d-49b6-bc36-2703efa4f67f (myzonegroup) zone 6cce41f3-a54b-47c2-981f-3b56ca0a4489 (myzone) metadata sync no sync (zone is master) 2023-03-07T22:31:16.466+0100 7f96a3e7a840 0 ERROR: failed to fetch datalog info data sync source: f2b20676-2672-4a92-a7ee-f3eb2efb12c6 (mysecondaryzone) failed to retrieve sync info: (13) Permission denied because on secondary zone (read only) , i see a 403 error about the permission denied from the master node 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 ====== starting new request req=0x7f1fd418c620 ===== 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 req 2604939314198041770 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 ====== req done req=0x7f1fd418c620 op status=0 http_status=403 latency=0.000000000s ====== 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 beast: 0x7f1fd418c620: 10......... - - [07/Mar/2023:00:00:53.309 +0100] "POST /admin/realm/period?period=395f9f13-d941-4ccf-a0cf-6c5d6d6579c2&epoch=76&rgwx-zonegroup=29592d75-224d-49b6-bc36-2703efa4f67f HTTP/1.1" 403 194 - - - latency=0.000000000s 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 ====== starting new request req=0x7f1fd4411620 ===== 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 req 7374970752399537975 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 ====== req done req=0x7f1fd4411620 op status=0 http_status=403 latency=0.000000000s ====== 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 beast: 0x7f1fd4411620: 10......... - - [07/Mar/2023:00:00:53.441 +0100] "POST /admin/log?type=data&notify&source-zone=6cce41f3-a54b-47c2-981f-3b56ca0a4489&rgwx-zonegroup=29592d75-224d-49b6-bc36-2703efa4f67f HTTP/1.1" 403 194 - - - latency=0.000000000s No issue when i use the command to check sync on secondary zone I don't understand because on secondary zone, pull realm and period with a user with flag system and admin works, the sync works for objects but not for users and buckets. When i list user and bucket on secondary zone, there are nothing but i have my objects on pool bucket.data !! i think the 403 was due because my user with flag system doesn't exist on secondary zone but i don't understand why user and bucket are not syncronized ??!! Access key and secret key are set on master zone and secondary zone, endpoint also I have an other cluster with a similary configuration and i don't have any issue Can someone help me ? Sorry for my english Regards Guillaume

1 1

Issue upgrading 17.2.0 to 17.2.5
by aellahib＠gmail.com 07 Mar '23

07 Mar '23

I initially ran the upgrade fine but it failed @ around 40/100 on an osd, so after waiting for along time i thought I'd try restarting it and then restarting the upgrade. I am stuck with the below debug error, I have tested docker pull from other servers and they dont fail for the ceph images but on ceph it does. If i even try to redeploy or add or remove mon damons for example it comes up with the same error related to the images. The error that ceph is giving me is: 2023-03-02T07:22:45.063976-0700 mgr.mgr-node.idvkbw [DBG] _run_cephadm : args = [] 2023-03-02T07:22:45.070342-0700 mgr.mgr-node.idvkbw [DBG] args: --image stop --no-container-init pull 2023-03-02T07:22:45.081086-0700 mgr.mgr-node.idvkbw [DBG] Running command: which python3 2023-03-02T07:22:45.180052-0700 mgr.mgr-node.idvkbw [DBG] Running command: /usr/bin/python3 /var/lib/ceph/5058e342-dac7-11ec-ada3-01065e90228d/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e --image stop --no-container-init pull 2023-03-02T07:22:46.500561-0700 mgr.mgr-node.idvkbw [DBG] code: 1 2023-03-02T07:22:46.500787-0700 mgr.mgr-node.idvkbw [DBG] err: Pulling container image stop... Non-zero exit code 1 from /usr/bin/docker pull stop /usr/bin/docker: stdout Using default tag: latest /usr/bin/docker: stderr Error response from daemon: pull access denied for stop, repository does not exist or may require 'docker login': denied: requested access to the resource is denied ERROR: Failed command: /usr/bin/docker pull stop

4 8

Creating a role for quota management
by anantha.adiga＠intel.com 07 Mar '23

07 Mar '23

Hello, Can you provide details on how to create a role for allowing a set of users to set quota on CephFS pools ?

3 2

Re: Very slow backfilling
by Joffrey 07 Mar '23

07 Mar '23

My Ceph Version is 17.2.5 and all configuration about osd_scrub* are defaults. I tried some updates on osd-max-backfills but no change. I have many HDD with NVME for db and all are connected in a 25G network. Yes, it's the same PG since 4 days. I got a failure on a HDD and get many days of recovery+backfilling last 2 weeks. Perhaps the 'not in time' warning is related to this. 'Jof Le jeu. 2 mars 2023 à 14:25, Anthony D'Atri <aad(a)dreamsnake.net> a écrit : > Run `ceph health detail`. > > Is it the same PG backfilling for a long time, or a different one over > time? > > That it’s remapped makes me think that what you’re seeing is the balancer > doing its job. > > As far as the scrubbing, do you limit the times when scrubbing can happen? > Are these HDDs? EC? > > > On Mar 2, 2023, at 07:20, Joffrey <joff.au(a)gmail.com> wrote: > > > > Hi, > > > > I have many 'not {deep-}scrubbed in time' and a1 PG remapped+backfilling > > and I don't understand why this backfilling is taking so long. > > > > root@hbgt-ceph1-mon3:/# ceph -s > > cluster: > > id: c300532c-51fa-11ec-9a41-0050569c3b55 > > health: HEALTH_WARN > > 15 pgs not deep-scrubbed in time > > 13 pgs not scrubbed in time > > > > services: > > mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 > > (age 36h) > > mgr: hbgt-ceph1-mon2.nteihj(active, since 2d), standbys: > > hbgt-ceph1-mon1.thrnnu, hbgt-ceph1-mon3.gmfzqm > > osd: 60 osds: 60 up (since 13h), 60 in (since 13h); 1 remapped pgs > > rgw: 3 daemons active (3 hosts, 2 zones) > > > > data: > > pools: 13 pools, 289 pgs > > objects: 67.74M objects, 127 TiB > > usage: 272 TiB used, 769 TiB / 1.0 PiB avail > > pgs: 288 active+clean > > 1 active+remapped+backfilling > > > > io: > > client: 3.3 KiB/s rd, 1.5 MiB/s wr, 3 op/s rd, 8 op/s wr > > recovery: 790 KiB/s, 0 objects/s > > > > > > What can I do to understand this slow recovery (is it the backfill > action ?) > > > > Thanks you > > > > 'Jof > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

3 8

Theory about min_size and its implications
by stefan.pinter＠bearingpoint.com 06 Mar '23

06 Mar '23

Hi! it is unclear for us what min_size means besides what it does. i hope someone can clear this up :) scenario: size is 3 and min_size is 2 2 rooms with 100 OSDs each and this crush rule "op": "take", "item": -10, "item_name": "default" "op": "choose_firstn", "num": 2, "type": "room" "op": "chooseleaf_firstn", "num": 2, "type": "host" "op": "emit" so if one room goes down/offline, around 50% of the PGs would be left with only 1 replica making them read-only. if we'd set min_size to 1 and one room goes down, user wouldn't still be able to access all PGs - but what is the problem with only one active PG? someone pointed out "split brain" but I am unsure about this. i think what happens in the worst case is this: only 1 PG is available, client writes changes to this PG, the disk of this 1 PG dies as well - so i guess we'd need to restore the data from the 2 offline PGs in the room that is down and we would have lots of trouble with restoring and also with data inconsistency, right? thank you!

5 8

upgrade problem from 1.6 to 1.7 related with osd
by Oğuz Yarımtepe 06 Mar '23

06 Mar '23

Hi, I have a cluster with rook operator running the ceph version 1.6 and upgraded first rook operator and then the ceph cluster definition. Everything was fine, every component except from osds are upgraded. Below is the reason of OSD not being upgraded: not updating OSD 1 on node "some-node-name". node no longer exists in the storage spec. if the user wishes to remove OSDs from the node, they must do so manually. Rook will not remove OSDs from nodes that are removed from the storage spec in order to prevent accidental data loss Any idea or anyone had seen it before? Regards. -- Oğuz Yarımtepe http://about.me/oguzy

1 0

mds readonly, mds all down
by kreept.sama＠gmail.com 06 Mar '23

06 Mar '23

Hello. We trying to resolve some issue with ceph. Our openshift cluster is blocked and we tried do almost all. Actual state is: MDS_ALL_DOWN: 1 filesystem is offline MDS_DAMAGE: 1 mds daemon damaged FS_DEGRADED: 1 filesystem is degraded MON_DISK_LOW: mon be is low on available space RECENT_CRASH: 1 daemons have recently crashed We try to perform cephfs-journal-tool --rank=gml-okd-cephfs:all event recover_dentries summary cephfs-journal-tool --rank=gml-okd-cephfs:all journal reset cephfs-table-tool gml-okd-cephfs:all reset session ceph mds repaired 0 ceph config rm mds mds_verify_scatter ceph config rm mds mds_debug_scatterstat ceph tell gml-okd-cephfs scrub start / recursive repair force After these commands, mds rises but an error appears: MDS_READ_ONLY: 1 MDSs are read only We also tried to create new fs with new metadata pool, delete and recreate old fs with same name with old\new metadatapool. We got rid of the errors, but the Openshift cluster did not want to work with the old persistence volumes. The pods wrote an error that they could not find it, while it was present and moreover, this volume was associated with pvc. Now we have rolled back the cluster and are trying to remove the mds error. Any ideas what to try? Thanks

2 2

orchestrator issues on ceph 16.2.9
by Adrian Nicolae 06 Mar '23

06 Mar '23

Hi, I have some orchestrator issues on our cluster running 16.2.9 with rgw only services. We first noticed these issues a few weeks ago when adding new hosts to the cluster - the orch was not detecting the new drives to build the osd containers for them. Debugging the mgr logs, I noticed that the mgr was crashing due to the dashboard module. I disabled the dashboard module and the new drives were detected and added to the cluster. Now we have other similar issues : we have a failed drive. The failure was detected , the osd was marked as down and the rebalancing is finished. I want to remove the failed osd from the cluster but it looks like the orch is not working : - I launched the osd removal with 'ceph orch osd rm 92 --force' where 92 is the osd id in question - I checked the progress but nothing happens even after a few days : ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT *92 node10 started 0 False True False* - the osd process is stopped on that host and from the orch side I can see this : ceph orch ps --daemon_type osd --daemon_id 92 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID osd.92 node10 error 11h ago 4w - 4096M <unknown> <unknown> - I have the same long refresh interval on other osds as well. I know it should be 10 minutes or so : NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.93 node09 running (4w) 11h ago 4w 5573M 4096M 16.2.9 3520ead5eb19 d2f658e9e37b osd: *116 osds: 115 up* (since 11d), 115 in (since 11d) 90 hdd 16.37109 1.00000 16 TiB 4.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 24.66 0.97 146 up * 92 hdd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down* 94 hdd 16.37109 1.00000 16 TiB 4.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 24.66 0.97 146 up - I activated debug 20 on the mgr but I can't see any errors or other clues regarding the osd removal. I also switched to the standby manager with 'ceph mgr fail'. The mgr switch works but still nothing happens - It's not only the osd removal thing. I also tried to deploy new rgw services by applying rgw labels on 2 new hosts , we have specs for building rgw containers when detecting the label. Again, nothing happens. I'm planning to upgrade to 16.2.11 to see if this solves the issues but I'm not very confident, I didn't see anything regarding this in the changelogs. Is there anything else I can try to debug this issue ? Thanks.

2 1

Creating a role for allowing users to set quota on CpehFS pools
by ananda a 06 Mar '23

06 Mar '23

Hello, How do I create a role to allow users to set quota on specific Ceph FS pools. ? Thank you

1 0

deep scrub and long backfilling
by xadhoom76＠gmail.com 05 Mar '23

05 Mar '23

ceph 16.2.11, is safe to enable scrub and deep scrub during backfilling ? I have log recovery-backfilling due to a new crushmap , backfilling is going slow and deep scrub interval as expired so I have many pgs not deep-scrubbed in time. Best regards Alessandro

4 3

Role for setting quota on Cephfs pools
by saaa_2001＠yahoo.com 04 Mar '23

04 Mar '23

Hello, I need to provide a credential to set of users to allow them to set quota for Cephfs pools. How do I accomplish it?

1 0

restoring ceph cluster from osds
by Ben 04 Mar '23

04 Mar '23

Hi, I ended up with having whole set of osds to get back original ceph cluster. I figured out to make the cluster running. However, it's status is something as below: bash-4.4$ ceph -s cluster: id: 3f271841-6188-47c1-b3fd-90fd4f978c76 health: HEALTH_WARN 7 daemons have recently crashed 4 slow ops, oldest one blocked for 35077 sec, daemons [mon.a,mon.b] have slow ops. services: mon: 3 daemons, quorum a,b,d (age 9h) mgr: b(active, since 14h), standbys: a osd: 4 osds: 0 up, 4 in (since 9h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: All osds are down. I checked the osds logs following this. Please help and I wonder if it's possible to get the cluster back. I have some backup for monitor's data. Until now I haven't restored that in the course. Thanks, Ben osd0 log: debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 set uid:gid to 167:167 (ceph:ceph) debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process ceph-osd, pid 1 debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 pidfile_write: ignore empty --pid-file debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 100 GiB debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:50.934+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:51.481+0000 7f824d7043c0 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal debug 2023-03-04T04:41:51.482+0000 7f824d7043c0 -1 Falling back to public interface debug 2023-03-04T04:41:51.492+0000 7f824d7043c0 0 load: jerasure load: lrc debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.024+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 0 osd.0:0.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 0 osd.0:1.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 0 osd.0:2.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.324+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.324+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 0 osd.0:3.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.734+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.734+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 0 osd.0:4.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.872+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.872+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 0 osd.0:5.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:53.394+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:53.394+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 0 osd.0:6.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.062+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:54.062+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 0 osd.0:7.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:54.504+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.504+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bdev(0x5570b27f3800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bdev(0x5570b27f3800 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 100 GiB debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs mount debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 debug 2023-03-04T04:41:54.508+0000 7f824d7043c0 1 bluefs mount shared_bdev_used = 0 debug 2023-03-04T04:41:54.508+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: RocksDB version: 6.15.5 debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: Compile date Oct 17 2022 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: DB SUMMARY debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: DB Session ID: TK70G5I4T4HSLQWUMAF4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: CURRENT file: CURRENT debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: IDENTITY file: IDENTITY debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: MANIFEST file: MANIFEST-000407 size: 5232 Bytes debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: SST files in db dir, Total Num: 25, files: 000067.sst 000085.sst 000098.sst 000298.sst 000304.sst 000348.sst 000355.sst 000356.sst 000366.sst debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Write Ahead Log file in db.wal: 000408.log size: 5147 ; debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.error_if_exists: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.create_if_missing: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.paranoid_checks: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.env: 0x5570b2806450 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.fs: Legacy File System debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.info_log: 0x5570b285af80 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_file_opening_threads: 16 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.statistics: (nil) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_fsync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_log_file_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.log_file_time_to_roll: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.keep_log_file_num: 1000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.recycle_log_file_num: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_fallocate: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_mmap_reads: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_mmap_writes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_direct_reads: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.create_missing_column_families: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_log_dir: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_dir: db.wal debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_cache_numshardbits: 6 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.WAL_ttl_seconds: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.WAL_size_limit_MB: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.is_fd_close_on_exec: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.advise_random_on_open: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_write_buffer_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_buffer_manager: 0x5570b28064e0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_adaptive_mutex: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.rate_limiter: (nil) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_recovery_mode: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_thread_tracking: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_pipelined_write: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.unordered_write: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.row_cache: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_filter: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_ingest_behind: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.preserve_deletes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.two_write_queues: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.manual_wal_flush: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.atomic_flush: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.persist_stats_to_disk: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_dbid_to_manifest: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.log_readahead_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.file_checksum_gen_factory: Unknown debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.best_efforts_recovery: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_data_in_errors: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_host_id: __hostname__ debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_jobs: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_compactions: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_subcompactions: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.delayed_write_rate : 16777216 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_total_wal_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_dump_period_sec: 600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_persist_period_sec: 600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_open_files: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.strict_bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_readahead_size: 2097152 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_flushes: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Compression algorithms supported: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZSTD supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kXpressCompression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZlibCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kSnappyCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kLZ4Compression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kBZip2Compression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/db_impl/db_impl_readonly.cc:23] Opening the db in read only mode debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: db/MANIFEST-000407 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_filter: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_filter_factory: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.sst_partitioner_factory: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_factory: SkipListFactory debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_factory: BlockBasedTable debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5570b1916ef0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5570b1988f30 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 4 enable_index_compression: 1 block_align: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_buffer_size: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_number: 4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression: NoCompression debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression: Disabled debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.prefix_extractor: nullptr debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.num_levels: 7 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.window_bits: -14 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.level: 32767 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.strategy: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.parallel_threads: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.enabled: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.target_file_size_base: 67108864 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.target_file_size_multiplier: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.arena_block_size: 33554432 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.disable_auto_compactions: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_properties_collectors: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.inplace_update_support: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.inplace_update_num_locks: 10000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_huge_page_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bloom_locality: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_successive_merges: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.optimize_filters_for_hits: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.paranoid_file_checks: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.force_consistency_checks: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.report_bg_io_stats: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.ttl: 2592000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.periodic_compaction_seconds: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_blob_files: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.min_blob_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_file_size: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_compression_type: NoCompression debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_blob_garbage_collection: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [m-0]: ... ... ... ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 3.23 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L1 1/0 26.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 29.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-1] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-1] ** ** Compaction Stats [p-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 4.97 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L1 1/0 22.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 27.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-2] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-2] ** ** Compaction Stats [O-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 1.89 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 L1 1/0 9.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 9.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-0] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-0] ** ** Compaction Stats [O-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.0 0.00 0.00 1 0.000 0 0 L1 1/0 8.74 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 30.5 69.6 67.4 0.13 0.05 1 0.130 20K 2511 Sum 1/0 8.74 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4155.5 69.4 67.3 0.13 0.05 2 0.065 20K 2511 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-1] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 69.6 67.4 0.13 0.05 1 0.130 20K 2511 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.01 GB write, 0.00 MB/s write, 0.01 GB read, 0.00 MB/s read, 0.1 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-1] ** ** Compaction Stats [O-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 2.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.6 0.00 0.00 1 0.000 0 0 L1 1/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.6 0.00 0.00 1 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-2] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.6 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-2] ** ** Compaction Stats [L] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L1 3/0 11.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 11.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [L] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [L] ** ** Compaction Stats [P] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 L1 1/0 19.96 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.9 113.3 54.8 0.36 0.15 1 0.365 450K 365K Sum 1/0 19.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17800.7 113.2 54.7 0.36 0.15 2 0.182 450K 365K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [P] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.3 54.8 0.36 0.15 1 0.365 450K 365K User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.02 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.4 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [P] ** debug 2023-03-04T14:32:10.864+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:32:40.999+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:33:11.317+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:33:42.245+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:34:13.196+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:34:44.048+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:35:14.100+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:35:44.757+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:36:15.544+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:36:46.148+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map

1 0

Ceph v15.2.14 - Dirty Object issue
by xadhoom76＠gmail.com 04 Mar '23

04 Mar '23

Hi, we have a cluster with this ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 240 GiB 205 GiB 29 GiB 35 GiB 14.43 hddvm 1.6 TiB 1.2 TiB 277 GiB 332 GiB 20.73 TOTAL 1.8 TiB 1.4 TiB 305 GiB 366 GiB 19.91 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR device_health_metrics 1 1 0 B 0 B 0 B 0 0 B 0 B 0 B 0 308 GiB N/A N/A 0 0 B 0 B rbd-pool 2 32 539 B 19 B 520 B 9 539 B 19 B 520 B 0 462 GiB N/A N/A 9 0 B 0 B cephfs.sharedfs.meta 3 32 299 MiB 190 MiB 109 MiB 87.10k 299 MiB 190 MiB 109 MiB 0.03 308 GiB N/A N/A 87.10k 0 B 0 B cephfs.sharedfs.data 4 32 2.2 GiB 2.2 GiB 0 B 121.56k 2.2 GiB 2.2 GiB 0 B 0.23 308 GiB N/A N/A 121.56k 0 B 0 B rbd-pool-proddeb02 5 32 712 MiB 712 MiB 568 B 201 712 MiB 712 MiB 568 B 0.08 308 GiB N/A N/A 201 0 B 0 B So as you can see we have 332GB RAW but data really are 299+2.2G+712M POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 308 GiB rbd-pool 2 32 539 B 9 539 B 0 462 GiB cephfs.sharedfs.meta 3 32 299 MiB 87.10k 299 MiB 0.03 308 GiB cephfs.sharedfs.data 4 32 2.2 GiB 121.56k 2.2 GiB 0.23 308 GiB rbd-pool-proddeb02 5 32 712 MiB 201 712 MiB 0.08 308 GiB How to clean Dirty ? How is that possible ? any cache issue or not committed flush from client ? Best regards Alessandro

3 3

The conditional policy for the List operations does not work as expected for the bucket with tenant.
by Dmitry Kvashnin 04 Mar '23

04 Mar '23

The conditional policy for the List operations does not work as expected for the bucket with tenant. With buckets and users without a tenant, everything is fine. The owner of t1\bucket1 is t1$user1. Bucket Policy: { "Version":"2012-10-17", "Statement":[ { "Sid":"PolicyForUser2Prefix", "Effect":"Allow", "Principal":{"AWS": ["arn:aws:iam::t1:user/user2"] }, "Action": ["s3:List*","s3:Get*"], "Condition" : { "StringLike" : { "s3:prefix": [ "obj*"], "s3:delimiter":["/"] } }, "Resource": [ "arn:aws:s3::t1:bucket1", "arn:aws:s3::t1:bucket1/obj*" ] } ] } t1$user2 can list bucket1\obj1 but can't get bucket1\obj1 Get returns Error403 If the "Condition" section is removed from the policy, t1$user2 can get obj1 but can't list. If I change the condition to "StringLikeIfExists", everything is fine, but t1$user2 can List the root of the bucket1, which is undesirable. Are there any errors in the policy or is it a bug?

1 0

ceph quincy nvme drives displayed in device list sata ssd not displayed
by Chris Brown 03 Mar '23

03 Mar '23

Hello, Setting up first ceph cluster in lab. Rocky 8.6 Ceph quincy Using curl install method Following cephadm deployment steps Everything works as expected except ceph orch device ls --refresh Only displays nvme devices and not the sata ssds on the ceph host. Tried sgdisk --zap-all /dev/sda wipefs -a /dev/sda Adding sata osd manually I get: ceph orch daemon add osd ceph-a:data_devices=/dev/sda Created no osd(s) on host ceph-a; already created? nvme osd gets added without issue. I have looked in the volume log on the node and monitor log on the admin server and have not seen anything that seems like an obvious clue. I can see commands running successfully against /dev/sda in the logs. Ideas? Thanks, cb

1 1

RGW Multisite archive zone bucket removal restriction
by Ondřej Kukla 03 Mar '23

03 Mar '23

Hello, I’m planning a Ceph multisite deployment with two clusters. One being a primary one the clients will be intereacting with and a second one with zone tier type set as an archive one. There is also a one way sync from the primary zone set - I would like to have this cluster as a "backup zone”. The setup works as expected for me, but with one exception. When I delete a bucket on the primary zone the bucket in the archive zone gets also deleted which is a issue as I would like to be able to restore the bucket in case of unintentional deletion on the primary zone. I know that there is the object lock, but I would like to know if there is some other way? I would imagine for example a situation where when the bucket is removed on the primary zone the bucket on the archive zone would be marked as to be removed in some time period like 1day or 1week or something. Your thoughts are much appreciated. Kind regards, Ondrej

1 0

Minimum client version for Quincy
by Massimo Sgaravatto 03 Mar '23

03 Mar '23

Dear all I am going to update a ceph cluster (where I am using only rbd and rgw, i.e. I didn't deploy cephfs) from Octtopus to Quincy Before doing that I would like to understand if some old nautilus clients (that I can't update for several reasons) will still be able to connect In general: I am not able to find this information in the documentation of any ceph release Should I refer to get-require-min-compat-client ? Now in my Octopus cluster I see: [root@ceph-mon-01 ~]# ceph osd get-require-min-compat-client luminous but I have the feeling that this value is simply the one I set a while ago to support the upmap feature Thanks, Massimo

2 2

3 node clusters and a corner case behavior
by Victor Rodriguez 03 Mar '23

03 Mar '23

Hello, Before we start I'm fully aware that this kind of setup is not recommended by any means and I'm familiar with it's implications. I'm just trying to practice extreme situations, just in case... I have a test cluster with: 3 nodes with Proxmox 7.3 + Ceph Quincy 17.2.5 3 monitors + 3 managers in server01, server02 and server03 4 OSD, two in server01, two in server02. No OSD in server03. All OSD are class "ssd". 1 pool with replica=2, min_replica=1. Crush rule uses just ssd class OSD. I do wait for ceph status to be fully OK between each test. A.- If I orderly shutdown server01, it's OSDs get marked down as expected. I/O on the pool works correctly before, during and after the shutdown. B.- If I poweroff server01, it's OSDs do not get marked down. I/O on the pool does not work at all, neither reads nor writes. A small number of slow-ops show in ceph status, something like 7 to 25. After 30 minutes, the server01's OSDs get marked down, I/O on the pool gets restored and slow-ops disappear. C.- Now I create an OSD on server03 with class "noClass". This OSD won't be used by the pool. If I now poweroff server01, it's OSDs get marked down as soon as some I/O is sent to the pool and I/O works correctly. Looks like I am in this exact situation: https://tracker.ceph.com/issues/16910#note-2 Questions: Why does Ceph behave this way in test B? Shouldn't it simply mark the OSDs down like in test A and C? Which config setting(s) set that 30 minute wait time before marking all OSD down? Many thanks in advance! --

1 0

unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin
by Jeremy Hansen 03 Mar '23

03 Mar '23

3/3/23 2:13:53 AM[WRN]unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin I keep seeing this warning in the logs. I’m not really sure what action to take to resolve this issue. Thanks -jeremy

3 2

Re: CephFS Kernel Mount Options Without Mount Helper
by Milind Changire 03 Mar '23

03 Mar '23

(for archival purposes) On Thu, Mar 2, 2023 at 6:04 PM Milind Changire <mchangir(a)redhat.com> wrote: > The docs for the ceph kernel module will be updated appropriately in the > kernel documentation. > Thanks for pointing out your pain point. > > -- > Milind > > > On Thu, Mar 2, 2023 at 1:41 PM Shawn Weeks <sweeks(a)weeksconsulting.us> > wrote: > >> I’m already able to mount ceph without a helper using the built in ceph >> kernel module support. My issue is that the documentation mixes what >> parameters the module itself supports and what requires the helper. >> Everything I’ve discovered so far is from reading the source code and >> piecing things together from stack overflow and ceph forum posts. I was >> hoping there was a better answer and I was just missing a different set of >> documentation. >> >> Sent from my iPhone >> >> On Mar 2, 2023, at 1:48 AM, Milind Changire <mchangir(a)redhat.com> wrote: >> >> >> I think the mount(8) man page section titled "EXTERNAL HELPERS" states it >> clearly: >> ----- >> EXTERNAL HELPERS >> The syntax of external mount helpers is: >> >> /sbin/mount.suffix spec dir [-sfnv] [-N namespace] [-o options] >> [-t type.subtype] >> >> where the suffix is the filesystem type and the -sfnvoN options >> have the same meaning >> as the normal mount options. The -t option is used for filesystems >> with subtypes support (for >> example /sbin/mount.fuse -t fuse.sshfs). >> >> The command mount does not pass the mount options unbindable, >> runbindable, private, rprivate, >> slave, rslave, shared, rshared, auto, noauto, comment, x-*, loop, >> offset and sizelimit to >> the mount.<suffix> helpers. All other options are used in a >> comma-separated list as an argument >> to the -o option. >> ----- >> >> So, if there are mount options other than the basic ones processed by the >> main mount program, they are passed to the mount helper presuming they are >> of interest to the mount helper. Since the main mount program will be >> unable to make sense of ceph mount options like "secret", "secretfile", >> "mon_addr", "conf", "name", "ms_mode", "fs", "nofallback", etc., a mount >> helper will be required to mount a ceph file system. >> >> ----- >> 1. What other filesystem types does your Linux Distro mount currently ? >> 2. What other filesystems of your interest can your Linux Distro (Rocky >> 8/9) mount without a mount helper ? >> >> >> >> On Thu, Mar 2, 2023 at 11:53 AM Shawn Weeks <sweeks(a)weeksconsulting.us> >> wrote: >> >>> Rock 8 and 9 don’t have the helper available in their repos and I have >>> to work with what they include. >>> >>> Thanks >>> Shawn >>> >>> Sent from my iPhone >>> >>> On Mar 1, 2023, at 11:41 PM, Milind Changire <mchangir(a)redhat.com> >>> wrote: >>> >>> >>> Why is it critical to mount the ceph filesystem without the ceph mount >>> helper ? >>> >>> >>> On Thu, Mar 2, 2023 at 8:42 AM Shawn Weeks <sweeks(a)weeksconsulting.us> >>> wrote: >>> >>>> That’s the documentation that assumes you’re going to have the helper. >>>> It lists things like “secretfile” and “fs” that doesn’t work without the >>>> helper. I’ve gone back several versions of that page and none of them spell >>>> out what requires the helper and what’s supported native on the kernel. >>>> >>>> Thanks >>>> Shawn >>>> >>>> Sent from my iPhone >>>> >>>> On Mar 1, 2023, at 8:53 PM, Milind Changire <mchangir(a)redhat.com> >>>> wrote: >>>> >>>> >>>> Check if this doc helps: >>>> https://docs.ceph.com/en/quincy/cephfs/mount-using-kernel-driver/ >>>> >>>> >>>> On Tue, Feb 28, 2023 at 11:09 PM Shawn Weeks <sweeks(a)weeksconsulting.us> >>>> wrote: >>>> >>>>> Even the documentation at >>>>> https://www.kernel.org/doc/html/v5.14/filesystems/ceph.html#mount-options >>>>> is incomplete and doesn’t list options like “secret” and “mds_namespace” >>>>> >>>>> Thanks >>>>> Shawn >>>>> >>>>> > On Feb 28, 2023, at 11:03 AM, Shawn Weeks <sweeks(a)weeksconsulting.us> >>>>> wrote: >>>>> > >>>>> > I’m trying to find documentation for which mount options are >>>>> supported directly by the kernel module. For example in the kernel module >>>>> included in Rocky Linux 8 and 9 the secretfile option isn’t supported even >>>>> though the documentation seems to imply it is. It seems like the >>>>> documentation assumes you’ll always be using the mount.ceph helper and I’m >>>>> trying to find out what options are supported if you don’t have mount.ceph >>>>> helper. >>>>> > >>>>> > Thanks >>>>> > Shawn >>>>> > _______________________________________________ >>>>> > ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>> >>>> >>>> -- >>>> Milind >>>> >>>> >>> >>> -- >>> Milind >>> >>> >> >> -- >> Milind >> >> > > -- > Milind > > -- Milind

1 0

Interruption of rebalancing
by Jeffrey Turmelle 02 Mar '23

02 Mar '23

I have a Nautilus cluster with 7 nodes, 210 HDDs. I recently added the 7th node with 30 OSDs which are currently rebalancing very slowly. I just noticed that the ethernet interface only negotiated a 1Gb connection, even though it has a 10Gb interface. I’m not sure why, but would like to reboot the node to get the interface back to 10Gb. Is it ok to do this? What should I do to prep the cluster for the reboot? Jeffrey Turmelle International Research Institute for Climate & Society <https://iri.columbia.edu/> The Climate School <https://climate.columbia.edu/> at Columbia University <https://columbia.edu/> 845-652-3461

4 4

Very slow backfilling
by Joffrey 02 Mar '23

02 Mar '23

Hi, I have many 'not {deep-}scrubbed in time' and a1 PG remapped+backfilling and I don't understand why this backfilling is taking so long. root@hbgt-ceph1-mon3:/# ceph -s cluster: id: c300532c-51fa-11ec-9a41-0050569c3b55 health: HEALTH_WARN 15 pgs not deep-scrubbed in time 13 pgs not scrubbed in time services: mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 (age 36h) mgr: hbgt-ceph1-mon2.nteihj(active, since 2d), standbys: hbgt-ceph1-mon1.thrnnu, hbgt-ceph1-mon3.gmfzqm osd: 60 osds: 60 up (since 13h), 60 in (since 13h); 1 remapped pgs rgw: 3 daemons active (3 hosts, 2 zones) data: pools: 13 pools, 289 pgs objects: 67.74M objects, 127 TiB usage: 272 TiB used, 769 TiB / 1.0 PiB avail pgs: 288 active+clean 1 active+remapped+backfilling io: client: 3.3 KiB/s rd, 1.5 MiB/s wr, 3 op/s rd, 8 op/s wr recovery: 790 KiB/s, 0 objects/s What can I do to understand this slow recovery (is it the backfill action ?) Thanks you 'Jof

1 0

ceph 16.2.10 - misplaced object after changing crush map only setting hdd class
by xadhoom76＠gmail.com 02 Mar '23

02 Mar '23

Hi to all and thanks for sharing your experience on ceph ! We have an easy setup with 9 osd all hdd and 3 nodes, 3 osd for each node. We started the cluster to test how it works with hdd with default and easy bootstrap . Then we decide to add ssd and create a pool to use only ssd. In order to have pools on hdd and pools on ssd only we edited the crushmap to add class hdd We do not enter anything about ssd till now, nor disk or rules only add the class map to the default rule. So i show you the rules before introducing class hdd # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure2_1 { id 2 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure-pool.meta { id 3 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure-pool.data { id 4 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } And here is the after # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure2_1 { id 2 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure-pool.meta { id 3 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure-pool.data { id 4 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } Just doing this triggered the misplaced of all pgs bind to EC pool. Is that correct ? and why ? Best regards Alessandro Bolgia

2 2

How do I troubleshoot radosgw errors STS?
by mat＠hazmat.dev 02 Mar '23

02 Mar '23

I've setup RadosGW with STS ontop of my ceph cluster. It works great and fine but I'm also trying to setup authentication with an OpenIDConnect provider. I'm have a hard time troubleshooting issues because the radosgw log file doesn't have much information in it. For example when I try to use the `sts:AssumeRoleWithWebIdentity` API it fails with `{'Code': 'AccessDenied', ...}` and all I see is the beat log showing an HTTP 403. Is there a way to enable more verbose logging so I can see what is failing and why I'm getting certain errors with STS, S3, or IAM apis? My ceph.conf looks like this for each node (mildly redacted): ``` [client.radosgw.pve4] host = pve4 keyring = /etc/pve/priv/ceph.client.radosgw.keyring log file = /var/log/ceph/client.radosgw.$host.log rgw_dns_name = s3.lab rgw_frontends = beast endpoint=0.0.0.0:7480 ssl_endpoint=0.0.0.0:443 ssl_certificate=/etc/pve/priv/ceph/s3.lab.crt ssl_private_key=/etc/pve/priv/ceph/s3.lab.key rgw_sts_key = 1111111111111111 rgw_s3_auth_use_sts = true rgw_enable_apis = s3, s3website, admin, sts, iam ```

2 3

PG Sizing Question
by Deep Dish 01 Mar '23

01 Mar '23

Hello Looking to get some official guidance on PG and PGP sizing. Is the goal to maintain approximately 100 PGs per OSD per pool or for the cluster general? Assume the following scenario: Cluster with 80 OSD across 8 nodes; 3 Pools: - Pool1 = Replicated 3x - Pool2 = Replicated 3x - Pool3 = Erasure Coded 6-4 Assuming the well published formula: Let (Target PGs / OSD) = 100 [ (Target PGs / OSD) * (# of OSDs) ] / (Replica Size) - Pool1 = (100*80)/3 = 2666.67 => 4096 - Pool2 = (100*80)/3 = 2666.67 => 4096 - Pool3 = (100*80)/10 = 800 => 1024 Total cluster would have 9216 PGs and PGPs. Are there any implications (performance / monitor / MDS / RGW sizing) with how many PGs are created on the cluster? Looking for validation and / or clarification of the above. Thank you.

2 3

RadosGW multipart fragments not being cleaned up by lifecycle policy on Quincy
by Sean Houghton 01 Mar '23

01 Mar '23

The latest version of quincy seems to be having problems cleaning up multipart fragments from canceled uploads. The bucket is empty: % s3cmd -c .s3cfg ls s3://warp-benchmark % However, it's got 11TB of data and 700k objects. # radosgw-admin bucket stats --bucket=warp-benchmark { "bucket": "warp-benchmark", "num_shards": 10, "tenant": "", "zonegroup": "6be863e8-a9f2-42c9-b114-c8651b1f1afa", "placement_rule": "ssd.ec63", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "aa099b5e-01d5-4394-b287-df99a4d63298.18924.1", "marker": "aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "index_type": "Normal", "owner": "warp_benchmark", "ver": "0#5580404,1#5593184,2#5586262,3#5591427,4#5591937,5#5588120,6#5589760,7#5582923,8#5579062,9#5578699", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0", "mtime": "0.000000", "creation_time": "2023-02-10T21:45:12.721604Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#", "usage": { "rgw.main": { "size": 12047620866048, "size_actual": 12047620866048, "size_utilized": 12047620866048, "size_kb": 11765254752, "size_kb_actual": 11765254752, "size_kb_utilized": 11765254752, "num_objects": 736113 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } A bucket list shows that they are all multipart fragments # radosgw-admin bucket list --bucket=warp-benchmark [ ... (LOTS OF THESE) { "name": "_multipart_(2F3(gCS/1.GagoUCrCRqawswb6.rnd.tg1efLm7-es41Xg3i-Nm6bYjS-c-No79.12", "instance": "", "ver": { "pool": 20, "epoch": 30984 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 16777216, "mtime": "2023-02-16T00:03:01.586472Z", "etag": "e7475bca6a58de35648ca5f25d6653bf", "storage_class": "", "owner": "warp_benchmark", "owner_display_name": "Warp Benchmark", "content_type": "", "accounted_size": 16777216, "user_data": "", "appendable": "false" }, "tag": "_YdopX7yxnVrvg2h35MIQGN3vsPyZx5W", "flags": 0, "pending_map": [], "versioned_epoch": 0 } ] Note that the timestamp is from 2 weeks ago so a lifecycle policy of "cleanup after 1 day" should delete them. cat cleanup-multipart.xml <LifecycleConfiguration> <Rule> <ID>abort-multipart-rule</ID> <Filter> <Prefix></Prefix> </Filter> <Status>Enabled</Status> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> </Rule> </LifecycleConfiguration> % s3cmd dellifecycle s3://warp-benchmark s3://warp-benchmark/: Lifecycle Policy deleted % s3cmd setlifecycle cleanup-multipart.xml s3://warp-benchmark s3://warp-benchmark/: Lifecycle Policy updated A secondary problem is that the lifecycle policy never runs automatically and is stuck in the UNINITIAL state. This problem is for another day of debugging. # radosgw-admin lc list [ { "bucket": ":warp-benchmark:aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] However, it can be started manually # radosgw-admin lc process # radosgw-admin lc list [ { "bucket": ":warp-benchmark:aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "started": "Wed, 01 Mar 2023 17:35:27 GMT", "status": "COMPLETE" } ] This has no effect on the bucket and the bucket stats show the exact same size and object count (output omitted for brevity). Running a gc pass also has no effect # radosgw-admin gc list [] # radosgw-admin gc process # radosgw-admin gc list [] Any ideas?

1 0

Re: How do I troubleshoot radosgw errors STS?
by hazmat 01 Mar '23

01 Mar '23

Also, here is the ceph version: ceph version 17.2.5 (e04241aa9b639588fa6c864845287d2824cb6b55) quincy (stable)

1 0

Next quincy release (17.2.6)
by Yuri Weinstein 01 Mar '23

01 Mar '23

Hello We are planning to start QE validation release next week. If you have PRs that are to be part of it, please let us know by adding "needs-qa" for 'quincy' milestone ASAP. Thx YuriW

3 3