- ceph-users - lists.ceph.io

Role for setting quota on Cephfs pools
by saaa_2001＠yahoo.com 04 Mar '23

04 Mar '23

Hello, I need to provide a credential to set of users to allow them to set quota for Cephfs pools. How do I accomplish it?

1 0

restoring ceph cluster from osds
by Ben 04 Mar '23

04 Mar '23

Hi, I ended up with having whole set of osds to get back original ceph cluster. I figured out to make the cluster running. However, it's status is something as below: bash-4.4$ ceph -s cluster: id: 3f271841-6188-47c1-b3fd-90fd4f978c76 health: HEALTH_WARN 7 daemons have recently crashed 4 slow ops, oldest one blocked for 35077 sec, daemons [mon.a,mon.b] have slow ops. services: mon: 3 daemons, quorum a,b,d (age 9h) mgr: b(active, since 14h), standbys: a osd: 4 osds: 0 up, 4 in (since 9h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: All osds are down. I checked the osds logs following this. Please help and I wonder if it's possible to get the cluster back. I have some backup for monitor's data. Until now I haven't restored that in the course. Thanks, Ben osd0 log: debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 set uid:gid to 167:167 (ceph:ceph) debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process ceph-osd, pid 1 debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 pidfile_write: ignore empty --pid-file debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 100 GiB debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:50.934+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:51.481+0000 7f824d7043c0 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal debug 2023-03-04T04:41:51.482+0000 7f824d7043c0 -1 Falling back to public interface debug 2023-03-04T04:41:51.492+0000 7f824d7043c0 0 load: jerasure load: lrc debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.024+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 0 osd.0:0.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 0 osd.0:1.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 0 osd.0:2.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.324+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.324+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 0 osd.0:3.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.734+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.734+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 0 osd.0:4.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.872+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.872+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 0 osd.0:5.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:53.394+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:53.394+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 0 osd.0:6.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.062+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:54.062+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 0 osd.0:7.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:54.504+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.504+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bdev(0x5570b27f3800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bdev(0x5570b27f3800 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 100 GiB debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs mount debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 debug 2023-03-04T04:41:54.508+0000 7f824d7043c0 1 bluefs mount shared_bdev_used = 0 debug 2023-03-04T04:41:54.508+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: RocksDB version: 6.15.5 debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: Compile date Oct 17 2022 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: DB SUMMARY debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: DB Session ID: TK70G5I4T4HSLQWUMAF4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: CURRENT file: CURRENT debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: IDENTITY file: IDENTITY debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: MANIFEST file: MANIFEST-000407 size: 5232 Bytes debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: SST files in db dir, Total Num: 25, files: 000067.sst 000085.sst 000098.sst 000298.sst 000304.sst 000348.sst 000355.sst 000356.sst 000366.sst debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Write Ahead Log file in db.wal: 000408.log size: 5147 ; debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.error_if_exists: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.create_if_missing: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.paranoid_checks: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.env: 0x5570b2806450 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.fs: Legacy File System debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.info_log: 0x5570b285af80 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_file_opening_threads: 16 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.statistics: (nil) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_fsync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_log_file_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.log_file_time_to_roll: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.keep_log_file_num: 1000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.recycle_log_file_num: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_fallocate: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_mmap_reads: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_mmap_writes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_direct_reads: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.create_missing_column_families: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_log_dir: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_dir: db.wal debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_cache_numshardbits: 6 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.WAL_ttl_seconds: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.WAL_size_limit_MB: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.is_fd_close_on_exec: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.advise_random_on_open: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_write_buffer_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_buffer_manager: 0x5570b28064e0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_adaptive_mutex: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.rate_limiter: (nil) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_recovery_mode: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_thread_tracking: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_pipelined_write: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.unordered_write: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.row_cache: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_filter: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_ingest_behind: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.preserve_deletes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.two_write_queues: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.manual_wal_flush: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.atomic_flush: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.persist_stats_to_disk: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_dbid_to_manifest: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.log_readahead_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.file_checksum_gen_factory: Unknown debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.best_efforts_recovery: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_data_in_errors: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_host_id: __hostname__ debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_jobs: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_compactions: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_subcompactions: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.delayed_write_rate : 16777216 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_total_wal_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_dump_period_sec: 600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_persist_period_sec: 600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_open_files: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.strict_bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_readahead_size: 2097152 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_flushes: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Compression algorithms supported: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZSTD supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kXpressCompression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZlibCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kSnappyCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kLZ4Compression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kBZip2Compression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/db_impl/db_impl_readonly.cc:23] Opening the db in read only mode debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: db/MANIFEST-000407 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_filter: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_filter_factory: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.sst_partitioner_factory: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_factory: SkipListFactory debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_factory: BlockBasedTable debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5570b1916ef0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5570b1988f30 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 4 enable_index_compression: 1 block_align: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_buffer_size: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_number: 4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression: NoCompression debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression: Disabled debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.prefix_extractor: nullptr debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.num_levels: 7 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.window_bits: -14 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.level: 32767 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.strategy: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.parallel_threads: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.enabled: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.target_file_size_base: 67108864 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.target_file_size_multiplier: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.arena_block_size: 33554432 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.disable_auto_compactions: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_properties_collectors: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.inplace_update_support: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.inplace_update_num_locks: 10000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_huge_page_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bloom_locality: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_successive_merges: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.optimize_filters_for_hits: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.paranoid_file_checks: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.force_consistency_checks: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.report_bg_io_stats: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.ttl: 2592000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.periodic_compaction_seconds: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_blob_files: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.min_blob_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_file_size: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_compression_type: NoCompression debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_blob_garbage_collection: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [m-0]: ... ... ... ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 3.23 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L1 1/0 26.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 29.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-1] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-1] ** ** Compaction Stats [p-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 4.97 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L1 1/0 22.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 27.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-2] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-2] ** ** Compaction Stats [O-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 1.89 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 L1 1/0 9.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 9.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-0] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-0] ** ** Compaction Stats [O-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.0 0.00 0.00 1 0.000 0 0 L1 1/0 8.74 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 30.5 69.6 67.4 0.13 0.05 1 0.130 20K 2511 Sum 1/0 8.74 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4155.5 69.4 67.3 0.13 0.05 2 0.065 20K 2511 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-1] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 69.6 67.4 0.13 0.05 1 0.130 20K 2511 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.01 GB write, 0.00 MB/s write, 0.01 GB read, 0.00 MB/s read, 0.1 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-1] ** ** Compaction Stats [O-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 2.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.6 0.00 0.00 1 0.000 0 0 L1 1/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.6 0.00 0.00 1 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-2] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.6 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-2] ** ** Compaction Stats [L] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L1 3/0 11.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 11.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [L] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [L] ** ** Compaction Stats [P] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 L1 1/0 19.96 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.9 113.3 54.8 0.36 0.15 1 0.365 450K 365K Sum 1/0 19.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17800.7 113.2 54.7 0.36 0.15 2 0.182 450K 365K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [P] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.3 54.8 0.36 0.15 1 0.365 450K 365K User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.02 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.4 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [P] ** debug 2023-03-04T14:32:10.864+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:32:40.999+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:33:11.317+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:33:42.245+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:34:13.196+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:34:44.048+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:35:14.100+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:35:44.757+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:36:15.544+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:36:46.148+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map

1 0

Ceph v15.2.14 - Dirty Object issue
by xadhoom76＠gmail.com 04 Mar '23

04 Mar '23

Hi, we have a cluster with this ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 240 GiB 205 GiB 29 GiB 35 GiB 14.43 hddvm 1.6 TiB 1.2 TiB 277 GiB 332 GiB 20.73 TOTAL 1.8 TiB 1.4 TiB 305 GiB 366 GiB 19.91 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR device_health_metrics 1 1 0 B 0 B 0 B 0 0 B 0 B 0 B 0 308 GiB N/A N/A 0 0 B 0 B rbd-pool 2 32 539 B 19 B 520 B 9 539 B 19 B 520 B 0 462 GiB N/A N/A 9 0 B 0 B cephfs.sharedfs.meta 3 32 299 MiB 190 MiB 109 MiB 87.10k 299 MiB 190 MiB 109 MiB 0.03 308 GiB N/A N/A 87.10k 0 B 0 B cephfs.sharedfs.data 4 32 2.2 GiB 2.2 GiB 0 B 121.56k 2.2 GiB 2.2 GiB 0 B 0.23 308 GiB N/A N/A 121.56k 0 B 0 B rbd-pool-proddeb02 5 32 712 MiB 712 MiB 568 B 201 712 MiB 712 MiB 568 B 0.08 308 GiB N/A N/A 201 0 B 0 B So as you can see we have 332GB RAW but data really are 299+2.2G+712M POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 308 GiB rbd-pool 2 32 539 B 9 539 B 0 462 GiB cephfs.sharedfs.meta 3 32 299 MiB 87.10k 299 MiB 0.03 308 GiB cephfs.sharedfs.data 4 32 2.2 GiB 121.56k 2.2 GiB 0.23 308 GiB rbd-pool-proddeb02 5 32 712 MiB 201 712 MiB 0.08 308 GiB How to clean Dirty ? How is that possible ? any cache issue or not committed flush from client ? Best regards Alessandro

3 3

The conditional policy for the List operations does not work as expected for the bucket with tenant.
by Dmitry Kvashnin 03 Mar '23

03 Mar '23

The conditional policy for the List operations does not work as expected for the bucket with tenant. With buckets and users without a tenant, everything is fine. The owner of t1\bucket1 is t1$user1. Bucket Policy: { "Version":"2012-10-17", "Statement":[ { "Sid":"PolicyForUser2Prefix", "Effect":"Allow", "Principal":{"AWS": ["arn:aws:iam::t1:user/user2"] }, "Action": ["s3:List*","s3:Get*"], "Condition" : { "StringLike" : { "s3:prefix": [ "obj*"], "s3:delimiter":["/"] } }, "Resource": [ "arn:aws:s3::t1:bucket1", "arn:aws:s3::t1:bucket1/obj*" ] } ] } t1$user2 can list bucket1\obj1 but can't get bucket1\obj1 Get returns Error403 If the "Condition" section is removed from the policy, t1$user2 can get obj1 but can't list. If I change the condition to "StringLikeIfExists", everything is fine, but t1$user2 can List the root of the bucket1, which is undesirable. Are there any errors in the policy or is it a bug?

1 0

ceph quincy nvme drives displayed in device list sata ssd not displayed
by Chris Brown 03 Mar '23

03 Mar '23

Hello, Setting up first ceph cluster in lab. Rocky 8.6 Ceph quincy Using curl install method Following cephadm deployment steps Everything works as expected except ceph orch device ls --refresh Only displays nvme devices and not the sata ssds on the ceph host. Tried sgdisk --zap-all /dev/sda wipefs -a /dev/sda Adding sata osd manually I get: ceph orch daemon add osd ceph-a:data_devices=/dev/sda Created no osd(s) on host ceph-a; already created? nvme osd gets added without issue. I have looked in the volume log on the node and monitor log on the admin server and have not seen anything that seems like an obvious clue. I can see commands running successfully against /dev/sda in the logs. Ideas? Thanks, cb

1 1

RGW Multisite archive zone bucket removal restriction
by Ondřej Kukla 03 Mar '23

03 Mar '23

Hello, I’m planning a Ceph multisite deployment with two clusters. One being a primary one the clients will be intereacting with and a second one with zone tier type set as an archive one. There is also a one way sync from the primary zone set - I would like to have this cluster as a "backup zone”. The setup works as expected for me, but with one exception. When I delete a bucket on the primary zone the bucket in the archive zone gets also deleted which is a issue as I would like to be able to restore the bucket in case of unintentional deletion on the primary zone. I know that there is the object lock, but I would like to know if there is some other way? I would imagine for example a situation where when the bucket is removed on the primary zone the bucket on the archive zone would be marked as to be removed in some time period like 1day or 1week or something. Your thoughts are much appreciated. Kind regards, Ondrej

1 0

Minimum client version for Quincy
by Massimo Sgaravatto 03 Mar '23

03 Mar '23

Dear all I am going to update a ceph cluster (where I am using only rbd and rgw, i.e. I didn't deploy cephfs) from Octtopus to Quincy Before doing that I would like to understand if some old nautilus clients (that I can't update for several reasons) will still be able to connect In general: I am not able to find this information in the documentation of any ceph release Should I refer to get-require-min-compat-client ? Now in my Octopus cluster I see: [root@ceph-mon-01 ~]# ceph osd get-require-min-compat-client luminous but I have the feeling that this value is simply the one I set a while ago to support the upmap feature Thanks, Massimo

2 2

3 node clusters and a corner case behavior
by Victor Rodriguez 03 Mar '23

03 Mar '23

Hello, Before we start I'm fully aware that this kind of setup is not recommended by any means and I'm familiar with it's implications. I'm just trying to practice extreme situations, just in case... I have a test cluster with: 3 nodes with Proxmox 7.3 + Ceph Quincy 17.2.5 3 monitors + 3 managers in server01, server02 and server03 4 OSD, two in server01, two in server02. No OSD in server03. All OSD are class "ssd". 1 pool with replica=2, min_replica=1. Crush rule uses just ssd class OSD. I do wait for ceph status to be fully OK between each test. A.- If I orderly shutdown server01, it's OSDs get marked down as expected. I/O on the pool works correctly before, during and after the shutdown. B.- If I poweroff server01, it's OSDs do not get marked down. I/O on the pool does not work at all, neither reads nor writes. A small number of slow-ops show in ceph status, something like 7 to 25. After 30 minutes, the server01's OSDs get marked down, I/O on the pool gets restored and slow-ops disappear. C.- Now I create an OSD on server03 with class "noClass". This OSD won't be used by the pool. If I now poweroff server01, it's OSDs get marked down as soon as some I/O is sent to the pool and I/O works correctly. Looks like I am in this exact situation: https://tracker.ceph.com/issues/16910#note-2 Questions: Why does Ceph behave this way in test B? Shouldn't it simply mark the OSDs down like in test A and C? Which config setting(s) set that 30 minute wait time before marking all OSD down? Many thanks in advance! --

1 0

unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin
by Jeremy Hansen 03 Mar '23

03 Mar '23

3/3/23 2:13:53 AM[WRN]unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin I keep seeing this warning in the logs. I’m not really sure what action to take to resolve this issue. Thanks -jeremy

3 2

Re: CephFS Kernel Mount Options Without Mount Helper
by Milind Changire 03 Mar '23

03 Mar '23

(for archival purposes) On Thu, Mar 2, 2023 at 6:04 PM Milind Changire <mchangir(a)redhat.com> wrote: > The docs for the ceph kernel module will be updated appropriately in the > kernel documentation. > Thanks for pointing out your pain point. > > -- > Milind > > > On Thu, Mar 2, 2023 at 1:41 PM Shawn Weeks <sweeks(a)weeksconsulting.us> > wrote: > >> I’m already able to mount ceph without a helper using the built in ceph >> kernel module support. My issue is that the documentation mixes what >> parameters the module itself supports and what requires the helper. >> Everything I’ve discovered so far is from reading the source code and >> piecing things together from stack overflow and ceph forum posts. I was >> hoping there was a better answer and I was just missing a different set of >> documentation. >> >> Sent from my iPhone >> >> On Mar 2, 2023, at 1:48 AM, Milind Changire <mchangir(a)redhat.com> wrote: >> >> >> I think the mount(8) man page section titled "EXTERNAL HELPERS" states it >> clearly: >> ----- >> EXTERNAL HELPERS >> The syntax of external mount helpers is: >> >> /sbin/mount.suffix spec dir [-sfnv] [-N namespace] [-o options] >> [-t type.subtype] >> >> where the suffix is the filesystem type and the -sfnvoN options >> have the same meaning >> as the normal mount options. The -t option is used for filesystems >> with subtypes support (for >> example /sbin/mount.fuse -t fuse.sshfs). >> >> The command mount does not pass the mount options unbindable, >> runbindable, private, rprivate, >> slave, rslave, shared, rshared, auto, noauto, comment, x-*, loop, >> offset and sizelimit to >> the mount.<suffix> helpers. All other options are used in a >> comma-separated list as an argument >> to the -o option. >> ----- >> >> So, if there are mount options other than the basic ones processed by the >> main mount program, they are passed to the mount helper presuming they are >> of interest to the mount helper. Since the main mount program will be >> unable to make sense of ceph mount options like "secret", "secretfile", >> "mon_addr", "conf", "name", "ms_mode", "fs", "nofallback", etc., a mount >> helper will be required to mount a ceph file system. >> >> ----- >> 1. What other filesystem types does your Linux Distro mount currently ? >> 2. What other filesystems of your interest can your Linux Distro (Rocky >> 8/9) mount without a mount helper ? >> >> >> >> On Thu, Mar 2, 2023 at 11:53 AM Shawn Weeks <sweeks(a)weeksconsulting.us> >> wrote: >> >>> Rock 8 and 9 don’t have the helper available in their repos and I have >>> to work with what they include. >>> >>> Thanks >>> Shawn >>> >>> Sent from my iPhone >>> >>> On Mar 1, 2023, at 11:41 PM, Milind Changire <mchangir(a)redhat.com> >>> wrote: >>> >>> >>> Why is it critical to mount the ceph filesystem without the ceph mount >>> helper ? >>> >>> >>> On Thu, Mar 2, 2023 at 8:42 AM Shawn Weeks <sweeks(a)weeksconsulting.us> >>> wrote: >>> >>>> That’s the documentation that assumes you’re going to have the helper. >>>> It lists things like “secretfile” and “fs” that doesn’t work without the >>>> helper. I’ve gone back several versions of that page and none of them spell >>>> out what requires the helper and what’s supported native on the kernel. >>>> >>>> Thanks >>>> Shawn >>>> >>>> Sent from my iPhone >>>> >>>> On Mar 1, 2023, at 8:53 PM, Milind Changire <mchangir(a)redhat.com> >>>> wrote: >>>> >>>> >>>> Check if this doc helps: >>>> https://docs.ceph.com/en/quincy/cephfs/mount-using-kernel-driver/ >>>> >>>> >>>> On Tue, Feb 28, 2023 at 11:09 PM Shawn Weeks <sweeks(a)weeksconsulting.us> >>>> wrote: >>>> >>>>> Even the documentation at >>>>> https://www.kernel.org/doc/html/v5.14/filesystems/ceph.html#mount-options >>>>> is incomplete and doesn’t list options like “secret” and “mds_namespace” >>>>> >>>>> Thanks >>>>> Shawn >>>>> >>>>> > On Feb 28, 2023, at 11:03 AM, Shawn Weeks <sweeks(a)weeksconsulting.us> >>>>> wrote: >>>>> > >>>>> > I’m trying to find documentation for which mount options are >>>>> supported directly by the kernel module. For example in the kernel module >>>>> included in Rocky Linux 8 and 9 the secretfile option isn’t supported even >>>>> though the documentation seems to imply it is. It seems like the >>>>> documentation assumes you’ll always be using the mount.ceph helper and I’m >>>>> trying to find out what options are supported if you don’t have mount.ceph >>>>> helper. >>>>> > >>>>> > Thanks >>>>> > Shawn >>>>> > _______________________________________________ >>>>> > ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>> >>>> >>>> -- >>>> Milind >>>> >>>> >>> >>> -- >>> Milind >>> >>> >> >> -- >> Milind >> >> > > -- > Milind > > -- Milind

1 0