March 2023 - ceph-users - lists.ceph.io

upgrade problem from 1.6 to 1.7 related with osd

by Oğuz Yarımtepe

Hi, I have a cluster with rook operator running the ceph version 1.6 and upgraded first rook operator and then the ceph cluster definition. Everything was fine, every component except from osds are upgraded. Below is the reason of OSD not being upgraded: not updating OSD 1 on node "some-node-name". node no longer exists in the storage spec. if the user wishes to remove OSDs from the node, they must do so manually. Rook will not remove OSDs from nodes that are removed from the storage spec in order to prevent accidental data loss Any idea or anyone had seen it before? Regards. -- Oğuz Yarımtepe http://about.me/oguzy

1 year, 1 month

1
0
0 0

mds readonly, mds all down

by kreept.sama＠gmail.com

Hello. We trying to resolve some issue with ceph. Our openshift cluster is blocked and we tried do almost all. Actual state is: MDS_ALL_DOWN: 1 filesystem is offline MDS_DAMAGE: 1 mds daemon damaged FS_DEGRADED: 1 filesystem is degraded MON_DISK_LOW: mon be is low on available space RECENT_CRASH: 1 daemons have recently crashed We try to perform cephfs-journal-tool --rank=gml-okd-cephfs:all event recover_dentries summary cephfs-journal-tool --rank=gml-okd-cephfs:all journal reset cephfs-table-tool gml-okd-cephfs:all reset session ceph mds repaired 0 ceph config rm mds mds_verify_scatter ceph config rm mds mds_debug_scatterstat ceph tell gml-okd-cephfs scrub start / recursive repair force After these commands, mds rises but an error appears: MDS_READ_ONLY: 1 MDSs are read only We also tried to create new fs with new metadata pool, delete and recreate old fs with same name with old\new metadatapool. We got rid of the errors, but the Openshift cluster did not want to work with the old persistence volumes. The pods wrote an error that they could not find it, while it was present and moreover, this volume was associated with pvc. Now we have rolled back the cluster and are trying to remove the mds error. Any ideas what to try? Thanks

1 year, 1 month

2
2
0 0

orchestrator issues on ceph 16.2.9

by Adrian Nicolae

Hi, I have some orchestrator issues on our cluster running 16.2.9 with rgw only services. We first noticed these issues a few weeks ago when adding new hosts to the cluster - the orch was not detecting the new drives to build the osd containers for them. Debugging the mgr logs, I noticed that the mgr was crashing due to the dashboard module. I disabled the dashboard module and the new drives were detected and added to the cluster. Now we have other similar issues : we have a failed drive. The failure was detected , the osd was marked as down and the rebalancing is finished. I want to remove the failed osd from the cluster but it looks like the orch is not working : - I launched the osd removal with 'ceph orch osd rm 92 --force' where 92 is the osd id in question - I checked the progress but nothing happens even after a few days : ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT *92 node10 started 0 False True False* - the osd process is stopped on that host and from the orch side I can see this : ceph orch ps --daemon_type osd --daemon_id 92 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID osd.92 node10 error 11h ago 4w - 4096M <unknown> <unknown> - I have the same long refresh interval on other osds as well. I know it should be 10 minutes or so : NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.93 node09 running (4w) 11h ago 4w 5573M 4096M 16.2.9 3520ead5eb19 d2f658e9e37b osd: *116 osds: 115 up* (since 11d), 115 in (since 11d) 90 hdd 16.37109 1.00000 16 TiB 4.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 24.66 0.97 146 up * 92 hdd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down* 94 hdd 16.37109 1.00000 16 TiB 4.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 24.66 0.97 146 up - I activated debug 20 on the mgr but I can't see any errors or other clues regarding the osd removal. I also switched to the standby manager with 'ceph mgr fail'. The mgr switch works but still nothing happens - It's not only the osd removal thing. I also tried to deploy new rgw services by applying rgw labels on 2 new hosts , we have specs for building rgw containers when detecting the label. Again, nothing happens. I'm planning to upgrade to 16.2.11 to see if this solves the issues but I'm not very confident, I didn't see anything regarding this in the changelogs. Is there anything else I can try to debug this issue ? Thanks.

1 year, 1 month

2
1
0 0

Creating a role for allowing users to set quota on CpehFS pools

by ananda a

Hello, How do I create a role to allow users to set quota on specific Ceph FS pools. ? Thank you

1 year, 1 month

1
0
0 0

deep scrub and long backfilling

by xadhoom76＠gmail.com

ceph 16.2.11, is safe to enable scrub and deep scrub during backfilling ? I have log recovery-backfilling due to a new crushmap , backfilling is going slow and deep scrub interval as expired so I have many pgs not deep-scrubbed in time. Best regards Alessandro

1 year, 2 months

4
3
0 0

Role for setting quota on Cephfs pools

by saaa_2001＠yahoo.com

Hello, I need to provide a credential to set of users to allow them to set quota for Cephfs pools. How do I accomplish it?

1 year, 2 months

1
0
0 0

restoring ceph cluster from osds

by Ben

Hi, I ended up with having whole set of osds to get back original ceph cluster. I figured out to make the cluster running. However, it's status is something as below: bash-4.4$ ceph -s cluster: id: 3f271841-6188-47c1-b3fd-90fd4f978c76 health: HEALTH_WARN 7 daemons have recently crashed 4 slow ops, oldest one blocked for 35077 sec, daemons [mon.a,mon.b] have slow ops. services: mon: 3 daemons, quorum a,b,d (age 9h) mgr: b(active, since 14h), standbys: a osd: 4 osds: 0 up, 4 in (since 9h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: All osds are down. I checked the osds logs following this. Please help and I wonder if it's possible to get the cluster back. I have some backup for monitor's data. Until now I haven't restored that in the course. Thanks, Ben osd0 log: debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 set uid:gid to 167:167 (ceph:ceph) debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process ceph-osd, pid 1 debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 pidfile_write: ignore empty --pid-file debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 100 GiB debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:50.934+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:51.481+0000 7f824d7043c0 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal debug 2023-03-04T04:41:51.482+0000 7f824d7043c0 -1 Falling back to public interface debug 2023-03-04T04:41:51.492+0000 7f824d7043c0 0 load: jerasure load: lrc debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.024+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 0 osd.0:0.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 0 osd.0:1.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 0 osd.0:2.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.324+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.324+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 0 osd.0:3.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.734+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.734+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 0 osd.0:4.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.872+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.872+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 0 osd.0:5.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:53.394+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:53.394+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 0 osd.0:6.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.062+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:54.062+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 0 osd.0:7.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:54.504+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.504+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bdev(0x5570b27f3800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bdev(0x5570b27f3800 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 100 GiB debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs mount debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 debug 2023-03-04T04:41:54.508+0000 7f824d7043c0 1 bluefs mount shared_bdev_used = 0 debug 2023-03-04T04:41:54.508+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: RocksDB version: 6.15.5 debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: Compile date Oct 17 2022 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: DB SUMMARY debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: DB Session ID: TK70G5I4T4HSLQWUMAF4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: CURRENT file: CURRENT debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: IDENTITY file: IDENTITY debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: MANIFEST file: MANIFEST-000407 size: 5232 Bytes debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: SST files in db dir, Total Num: 25, files: 000067.sst 000085.sst 000098.sst 000298.sst 000304.sst 000348.sst 000355.sst 000356.sst 000366.sst debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Write Ahead Log file in db.wal: 000408.log size: 5147 ; debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.error_if_exists: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.create_if_missing: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.paranoid_checks: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.env: 0x5570b2806450 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.fs: Legacy File System debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.info_log: 0x5570b285af80 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_file_opening_threads: 16 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.statistics: (nil) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_fsync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_log_file_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.log_file_time_to_roll: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.keep_log_file_num: 1000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.recycle_log_file_num: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_fallocate: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_mmap_reads: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_mmap_writes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_direct_reads: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.create_missing_column_families: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_log_dir: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_dir: db.wal debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_cache_numshardbits: 6 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.WAL_ttl_seconds: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.WAL_size_limit_MB: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.is_fd_close_on_exec: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.advise_random_on_open: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_write_buffer_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_buffer_manager: 0x5570b28064e0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_adaptive_mutex: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.rate_limiter: (nil) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_recovery_mode: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_thread_tracking: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_pipelined_write: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.unordered_write: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.row_cache: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_filter: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_ingest_behind: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.preserve_deletes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.two_write_queues: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.manual_wal_flush: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.atomic_flush: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.persist_stats_to_disk: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_dbid_to_manifest: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.log_readahead_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.file_checksum_gen_factory: Unknown debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.best_efforts_recovery: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_data_in_errors: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_host_id: __hostname__ debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_jobs: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_compactions: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_subcompactions: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.delayed_write_rate : 16777216 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_total_wal_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_dump_period_sec: 600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_persist_period_sec: 600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_open_files: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.strict_bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_readahead_size: 2097152 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_flushes: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Compression algorithms supported: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZSTD supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kXpressCompression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZlibCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kSnappyCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kLZ4Compression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kBZip2Compression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/db_impl/db_impl_readonly.cc:23] Opening the db in read only mode debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: db/MANIFEST-000407 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_filter: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_filter_factory: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.sst_partitioner_factory: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_factory: SkipListFactory debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_factory: BlockBasedTable debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5570b1916ef0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5570b1988f30 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 4 enable_index_compression: 1 block_align: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_buffer_size: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_number: 4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression: NoCompression debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression: Disabled debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.prefix_extractor: nullptr debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.num_levels: 7 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.window_bits: -14 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.level: 32767 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.strategy: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.parallel_threads: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.enabled: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.target_file_size_base: 67108864 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.target_file_size_multiplier: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.arena_block_size: 33554432 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.disable_auto_compactions: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_properties_collectors: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.inplace_update_support: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.inplace_update_num_locks: 10000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_huge_page_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bloom_locality: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_successive_merges: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.optimize_filters_for_hits: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.paranoid_file_checks: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.force_consistency_checks: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.report_bg_io_stats: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.ttl: 2592000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.periodic_compaction_seconds: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_blob_files: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.min_blob_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_file_size: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_compression_type: NoCompression debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_blob_garbage_collection: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [m-0]: ... ... ... ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 3.23 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L1 1/0 26.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 29.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-1] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-1] ** ** Compaction Stats [p-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 4.97 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L1 1/0 22.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 27.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-2] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-2] ** ** Compaction Stats [O-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 1.89 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 L1 1/0 9.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 9.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-0] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-0] ** ** Compaction Stats [O-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.0 0.00 0.00 1 0.000 0 0 L1 1/0 8.74 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 30.5 69.6 67.4 0.13 0.05 1 0.130 20K 2511 Sum 1/0 8.74 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4155.5 69.4 67.3 0.13 0.05 2 0.065 20K 2511 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-1] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 69.6 67.4 0.13 0.05 1 0.130 20K 2511 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.01 GB write, 0.00 MB/s write, 0.01 GB read, 0.00 MB/s read, 0.1 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-1] ** ** Compaction Stats [O-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 2.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.6 0.00 0.00 1 0.000 0 0 L1 1/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.6 0.00 0.00 1 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-2] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.6 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-2] ** ** Compaction Stats [L] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L1 3/0 11.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 11.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [L] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [L] ** ** Compaction Stats [P] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 L1 1/0 19.96 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.9 113.3 54.8 0.36 0.15 1 0.365 450K 365K Sum 1/0 19.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17800.7 113.2 54.7 0.36 0.15 2 0.182 450K 365K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [P] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.3 54.8 0.36 0.15 1 0.365 450K 365K User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.02 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.4 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [P] ** debug 2023-03-04T14:32:10.864+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:32:40.999+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:33:11.317+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:33:42.245+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:34:13.196+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:34:44.048+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:35:14.100+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:35:44.757+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:36:15.544+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:36:46.148+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map

1 year, 2 months

1
0
0 0

Ceph v15.2.14 - Dirty Object issue

by xadhoom76＠gmail.com

Hi, we have a cluster with this ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 240 GiB 205 GiB 29 GiB 35 GiB 14.43 hddvm 1.6 TiB 1.2 TiB 277 GiB 332 GiB 20.73 TOTAL 1.8 TiB 1.4 TiB 305 GiB 366 GiB 19.91 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR device_health_metrics 1 1 0 B 0 B 0 B 0 0 B 0 B 0 B 0 308 GiB N/A N/A 0 0 B 0 B rbd-pool 2 32 539 B 19 B 520 B 9 539 B 19 B 520 B 0 462 GiB N/A N/A 9 0 B 0 B cephfs.sharedfs.meta 3 32 299 MiB 190 MiB 109 MiB 87.10k 299 MiB 190 MiB 109 MiB 0.03 308 GiB N/A N/A 87.10k 0 B 0 B cephfs.sharedfs.data 4 32 2.2 GiB 2.2 GiB 0 B 121.56k 2.2 GiB 2.2 GiB 0 B 0.23 308 GiB N/A N/A 121.56k 0 B 0 B rbd-pool-proddeb02 5 32 712 MiB 712 MiB 568 B 201 712 MiB 712 MiB 568 B 0.08 308 GiB N/A N/A 201 0 B 0 B So as you can see we have 332GB RAW but data really are 299+2.2G+712M POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 308 GiB rbd-pool 2 32 539 B 9 539 B 0 462 GiB cephfs.sharedfs.meta 3 32 299 MiB 87.10k 299 MiB 0.03 308 GiB cephfs.sharedfs.data 4 32 2.2 GiB 121.56k 2.2 GiB 0.23 308 GiB rbd-pool-proddeb02 5 32 712 MiB 201 712 MiB 0.08 308 GiB How to clean Dirty ? How is that possible ? any cache issue or not committed flush from client ? Best regards Alessandro

1 year, 2 months

3
3
0 0

The conditional policy for the List operations does not work as expected for the bucket with tenant.

by Dmitry Kvashnin

The conditional policy for the List operations does not work as expected for the bucket with tenant. With buckets and users without a tenant, everything is fine. The owner of t1\bucket1 is t1$user1. Bucket Policy: { "Version":"2012-10-17", "Statement":[ { "Sid":"PolicyForUser2Prefix", "Effect":"Allow", "Principal":{"AWS": ["arn:aws:iam::t1:user/user2"] }, "Action": ["s3:List*","s3:Get*"], "Condition" : { "StringLike" : { "s3:prefix": [ "obj*"], "s3:delimiter":["/"] } }, "Resource": [ "arn:aws:s3::t1:bucket1", "arn:aws:s3::t1:bucket1/obj*" ] } ] } t1$user2 can list bucket1\obj1 but can't get bucket1\obj1 Get returns Error403 If the "Condition" section is removed from the policy, t1$user2 can get obj1 but can't list. If I change the condition to "StringLikeIfExists", everything is fine, but t1$user2 can List the root of the bucket1, which is undesirable. Are there any errors in the policy or is it a bug?

1 year, 2 months

1
0
0 0

ceph quincy nvme drives displayed in device list sata ssd not displayed

by Chris Brown

Hello, Setting up first ceph cluster in lab. Rocky 8.6 Ceph quincy Using curl install method Following cephadm deployment steps Everything works as expected except ceph orch device ls --refresh Only displays nvme devices and not the sata ssds on the ceph host. Tried sgdisk --zap-all /dev/sda wipefs -a /dev/sda Adding sata osd manually I get: ceph orch daemon add osd ceph-a:data_devices=/dev/sda Created no osd(s) on host ceph-a; already created? nvme osd gets added without issue. I have looked in the volume log on the node and monitor log on the admin server and have not seen anything that seems like an obvious clue. I can see commands running successfully against /dev/sda in the logs. Ideas? Thanks, cb

1 year, 2 months

1
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users March 2023