- ceph-users - lists.ceph.io

cephadm bootstrap ignoring --skip-firewalld
by Kozakis, Anestis 17 Oct '24

17 Oct '24

As I mentioned in my earlier e-mail, new to Ceph, and trying to set up automation to deploy, configure, and manage a Ceph cluster. We configure our Firewall rules through SaltStack. I am passing the -skip-firewalld option to the cephadm bootstrap command, but cephadm seems to ignore the option and configures the firewall anyway. I have even reconfigured the options order to be the same as cephadm boostrap -help but it still ignores the option and configures the firewall. This creates issues as it configures the public zone, which we don't want changed. Below is the command we are using (with obvious settings changed/removed). cephadm bootstrap --mon-ip 10.0.0.0 --mgr-id host.domain.name --fsid [fsid] --ssh-private-key id_rsa --ssh-public-key id_rsa.pub --ssh-user [user] --cluster-network 192.168.0.0/25 --allow-fqdn-hostname --config=./ceph.conf --initial-dashboard-user admin --initial-dashboard-password SuperS3cr3tPassw0rd --dashboard-password-noupdate --skip-firewalld --with-centralized-logging --apply-spec=spec.yaml What am I missing? Anestis Kozakis Systems Administrator - Multi-Level Security Solutions P: + 61 2 6122 0205 M: +61 4 88 376 339 anestis.kozakis(a)raytheon.com.au<mailto:anestis.kozakis@raytheon.com.au> Raytheon Australia Cybersecurity and Information Assurance 4 Brindabella Cct Brindabella Business Park Canberra Airport, ACT 2609 www.raytheonaustralia.com.au<http://www.raytheonaustralia.com.au/> LinkedIn<https://au.linkedin.com/company/raytheon-australia> | Twitter<https://twitter.com/RaytheonAU> | Facebook<https://www.facebook.com/RaytheonAustralia> | Instagram<https://www.instagram.com/raytheonaustralia/>

2 3

Ubuntu 24.02 LTS Ceph status warning
by Dominique Ramaekers 16 Oct '24

16 Oct '24

I manage a 4 hosts cluster on Ubuntu 22.04 LTS with ceph installed trough cephad and containers on Docker. Last month, I've migrated to the latest Ceph 19.2. All went great. Last week I've upgraded one of my hosts to Ubuntu 24.04.1 LTS Now I get the following warning in cephadm shell -- ceph status: Failed to apply 1 service(s): osd.all-available-devices failed to probe daemons or devices Outside the ceph shell: Ceph -v results in 'ceph version 19.2.0~git20240301.4c76c50 (4c76c50a73f63ba48ccdf0adccce03b00d1d80c7) squid (dev)' Inside the shell: 'ceph version 19.3.0-5346-gcc481a63 (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)' All osd's, mon's , mgr's and mds's are on 19.2.0` (image id 37996728e013) Do I get the warning because the Ubuntu package of ceph is still on a development version? Or can I have another underling problem? Thanks for the help.

3 5

osd won't start
by Erwin Bogaard 16 Oct '24

16 Oct '24

Hi, we're experiencing issues with a few osd's. They had a crash, but now won't start anymore. Nothing seems wrong with them, but they keep hanging with apparent 100% i/o wait on the machine when starting the osd. This is on ceph 18.2.4 This is the log (edited a bit, as it's too long): 2024-10-16T13:48:05.075+0200 7fc771a10640 0 set uid:gid to 167:167 (ceph:ceph) 2024-10-16T13:48:05.075+0200 7fc771a10640 0 ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process ceph-osd, pid 1223 2024-10-16T13:48:05.076+0200 7fc771a10640 0 pidfile_write: ignore empty --pid-file 2024-10-16T13:48:05.086+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:05.086+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:05.087+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:05.088+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:05.088+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:05.088+0200 7fc771a10640 0 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:05.089+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:05.089+0200 7fc771a10640 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 1.8 TiB 2024-10-16T13:48:05.089+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:05.352+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:05.605+0200 7fc771a10640 0 starting osd.3 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2024-10-16T13:48:05.605+0200 7fc771a10640 -1 Falling back to public interface 2024-10-16T13:48:05.611+0200 7fc771a10640 0 load: jerasure load: lrc 2024-10-16T13:48:05.612+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:05.612+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:05.612+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:05.612+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:05.612+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:05.886+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:05.886+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:05.887+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:05.888+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:05.888+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:06.157+0200 7fc771a10640 0 osd.3:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:06.157+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:06.157+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:06.157+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:06.159+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:06.159+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:06.422+0200 7fc771a10640 0 osd.3:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:06.422+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:06.422+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:06.422+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:06.424+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:06.424+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:06.684+0200 7fc771a10640 0 osd.3:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:06.684+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:06.684+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:06.684+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:06.686+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:06.687+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:06.947+0200 7fc771a10640 0 osd.3:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:06.947+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:06.947+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:06.948+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:06.949+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:06.949+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:07.214+0200 7fc771a10640 0 osd.3:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:07.219+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:07.219+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:07.219+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:07.219+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:07.219+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:07.219+0200 7fc771a10640 0 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:07.220+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:07.220+0200 7fc771a10640 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 1.8 TiB 2024-10-16T13:48:07.220+0200 7fc771a10640 1 bluefs mount 2024-10-16T13:48:07.221+0200 7fc771a10640 1 bluefs _init_alloc shared, id 1, capacity 0x1d1a9000000, block size 0x10000 2024-10-16T13:48:07.335+0200 7fc771a10640 1 bluefs mount shared_bdev_used = 0 2024-10-16T13:48:07.335+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _prepare_db_environment set db_paths to db,1899995385036 db.slow,1899995385036 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: RocksDB version: 7.9.2 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Git sha 0 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Compile date 2024-07-12 14:24:06 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: DB SUMMARY 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: DB Session ID: FNW6XLKJV5A5TZMHCHLU 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: CURRENT file: CURRENT 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: IDENTITY file: IDENTITY 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: MANIFEST file: MANIFEST-020722 size: 59265 Bytes 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: SST files in db dir, Total Num: 109, files: 021121.sst 021122.sst 021123.sst 021124.sst 021125.sst 021126.sst 021127.sst 021128.sst 021129.sst 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Write Ahead Log file in db.wal: 021323.log size: 15835454 ; 021325.log size: 12760055 ; 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Options.error_if_exists: 0 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Options.create_if_missing: 0 ... 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.compaction_readahead_size: 2097152 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.max_background_flushes: -1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Compression algorithms supported: 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kZSTD supported: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kXpressCompression supported: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kBZip2Compression supported: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kLZ4Compression supported: 1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kZlibCompression supported: 1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kLZ4HCCompression supported: 1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kSnappyCompression supported: 1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: [db/db_impl/db_impl_readonly.cc:25] Opening the db in read only mode 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: db/MANIFEST-020722 2024-10-16T13:48:07.346+0200 7fc771a10640 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.compaction_filter: None 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.compaction_filter_factory: None 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.sst_partitioner_factory: None 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.table_factory: BlockBasedTable 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5630304b89a0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 checksum: 4 no_block_cache: 0 block_cache: 0x563030498dd0 block_cache_name: BinnedLRUCache block_cache_options: capacity : 483183820 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: bloomfilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 5 enable_index_compression: 1 block_align: 0 max_auto_readahead_size: 262144 prepopulate_block_cache: 0 initial_auto_readahead_size: 8192 num_file_reads_for_auto_readahead: 2 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.write_buffer_size: 16777216 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.max_write_buffer_number: 64 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.compression: LZ4 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.bottommost_compression: Disabled 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.prefix_extractor: nullptr ... 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.blob_file_starting_level: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 Then is hangs here seemingly indefinitely. Hope someone can shed some light!

2 2

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

SLOW_OPS problems
by Tim Sauerbein 15 Oct '24

15 Oct '24

Dear list, I have a small cluster (Reef 18.2.4) with 7 hosts and 3-4 OSDs each (960GB/1.92TB mixed Intel D3-S4610, Samsung SM883, PM897 SSDs): cluster: id: ecff3ce8-539b-443e-a492-da428f4aa9e9 health: HEALTH_OK services: mon: 5 daemons, quorum titan,mangan,kalium,argon,chromium (age 2w) mgr: mangan(active, since 2w), standbys: titan, argon osd: 22 osds: 22 up (since 2w), 22 in (since 3M) data: pools: 2 pools, 513 pgs objects: 2.76M objects, 7.0 TiB usage: 16 TiB used, 15 TiB / 31 TiB avail pgs: 513 active+clean On that cluster RBD volumes for virtual machines are stored. For a couple of months now the cluster reports slow ops for some OSDs and some PGs as laggy. This happens once or twice a day, sometimes more and sometimes not at all for a few days, at completely random times, independent of when snapshots are deleted and trimmed and independent of the I/O load or load on the hosts. After about 30 seconds, during which the write speed goes to zero on the VMs, everything returns to normal. I cannot reproduce the slow ops manually by creating write load on the cluster. Even writing continuously with 300-400 MB/s full speed for 20 minutes does not create any problems. See attached log file for an example of a typical occurrence. I have also measured write load on the disks during the problems with iostat which just shows how writes stall, see also attached. The OSDs with slow ops are completely random, any of the disks would show up once in a while. Current config (I've tried optimising snaptrim and scrub which didn't help): # ceph config dump WHO MASK LEVEL OPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced bdev_async_discard true global advanced bdev_enable_discard true global advanced public_network 10.0.4.0/24 * mon advanced auth_allow_insecure_global_id_reclaim false mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/mode upmap mgr unknown mgr/pg_autoscaler/autoscale_profile scale-up * osd basic osd_memory_target 4294967296 osd advanced osd_pg_max_concurrent_snap_trims 1 osd advanced osd_scrub_begin_hour 23 osd advanced osd_scrub_end_hour 4 osd advanced osd_scrub_sleep 1.000000 osd advanced osd_snap_trim_priority 1 osd advanced osd_snap_trim_sleep 2.000000 osd.0 basic osd_mclock_max_capacity_iops_ssd 29199.674019 osd.1 basic osd_mclock_max_capacity_iops_ssd 31554.530141 osd.10 basic osd_mclock_max_capacity_iops_ssd 25949.821194 osd.11 basic osd_mclock_max_capacity_iops_ssd 26300.596265 osd.12 basic osd_mclock_max_capacity_iops_ssd 25167.331294 osd.13 basic osd_mclock_max_capacity_iops_ssd 21606.610828 osd.14 basic osd_mclock_max_capacity_iops_ssd 27894.095121 osd.15 basic osd_mclock_max_capacity_iops_ssd 25929.047047 osd.16 basic osd_mclock_max_capacity_iops_ssd 15423.600235 osd.17 basic osd_mclock_max_capacity_iops_ssd 25097.493934 osd.18 basic osd_mclock_max_capacity_iops_ssd 25966.188007 osd.19 basic osd_mclock_max_capacity_iops_ssd 23628.746459 osd.2 basic osd_mclock_max_capacity_iops_ssd 32157.280832 osd.20 basic osd_mclock_max_capacity_iops_ssd 22722.682745 osd.3 basic osd_mclock_max_capacity_iops_ssd 33951.086556 osd.4 basic osd_mclock_max_capacity_iops_ssd 22736.907664 osd.5 basic osd_mclock_max_capacity_iops_ssd 21916.777510 osd.6 basic osd_mclock_max_capacity_iops_ssd 29984.954749 osd.7 basic osd_mclock_max_capacity_iops_ssd 26757.965797 osd.8 basic osd_mclock_max_capacity_iops_ssd 22738.921429 osd.9 basic osd_mclock_max_capacity_iops_ssd 24635.156413 Any help would be much appreciated! Thanks, Tim

9 21

Announcing go-ceph v0.30.0
by John Mulligan 15 Oct '24

15 Oct '24

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.30.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, rgw, and other subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulliga(a)ibm.com / jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

1 0

Ceph User + Dev October Meetup Details
by Laura Flores 15 Oct '24

15 Oct '24

Hi all, Please see the following link for details about the User + Dev October Meetup that will be happening this week Wednesday! https://www.meetup.com/ceph-user-group/events/303960311/ Joseph Mundackal will be presenting a proposal for introducing a new S3-focused user group. Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Question about mounting cephFS on MacOS
by Baijia Ye 15 Oct '24

15 Oct '24

Dear ceph developers, I hope this email finds you well. I am an MSCS student at NYU and I am trying to use ceph as our university's research network storage system. We have built the server with ceph, may I ask how to mount cephFS on MacOS as a client? I hope this is the correct place to ask questions. If not, could you instruct me the write way? Thanks for your help. Best, Baijia Ye

1 0

Membership additions/removals from the Ceph Steering Committee
by Patrick Donnelly 14 Oct '24

14 Oct '24

Greetings, The Ceph Steering Committee [1] (CSC) -- formerly the "CLT" or Ceph Leadership Team -- was formed 3 years ago with the adoption of our new governance model [2]. The CSC is responsible [4] for electing the Ceph Executive Council [3] (CEC), amending the governance model, and deciding on tactical issues /strategic direction for the Ceph project. After voting for the inaugural members of the CEC (Dan van der Ster, Josh Durgin, and Neha Ohja), we have primarily focused on week-to-week tactical issues confronting the Ceph project during our weekly meetings. This was a natural continuation of our former "Ceph Leadership Team" meetings where we discussed any technical issues or planning for the project. We have not had another election for the Ceph Executive Council membership or any other governance amendment since but are overdue for one according to our governance charter. To begin such an election, it seemed prudent to first vote on any membership changes to the CSC due to natural attrition (e.g. folks have moved on to other projects) or expansion (e.g. new team leads, interested community members, etc.). The CSC weekly call has had several individuals present that are all but officially on the committee. This announcement is a call for anyone (developer, community member, etc.) interested in participating in the governance and strategic initiatives of the Ceph project to please put your or someone else's name forward. There is broad interest in the CSC to have representatives from newer components in Ceph present. If you are on the current list of members and would like to resign from the CSC, please let us know. Some members who have not been present in any recent calls have already been proposed to be removed. Any addition or removal to the CSC must be proposed as a motion with a second and then approved by a 2/3 supermajority vote. We hope to have this election for CSC membership in the coming weeks. It will be performed remotely and offline using the Helios election system (also a FOSS project!). It is deemed necessary to finalize this in advance of any other election topic, including the CEC membership. [1] https://docs.ceph.com/en/latest/governance/#ceph-steering-committee [2] https://docs.ceph.com/en/latest/governance/ [3] https://docs.ceph.com/en/latest/governance/#ceph-executive-council [4] https://docs.ceph.com/en/latest/governance/#id2 [5] https://vote.heliosvoting.org/ Cheers, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 2

What is the problem with many PGs per OSD
by Frank Schilder 14 Oct '24

14 Oct '24

Hi all, during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing. As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing. Yet, I'm holding back on an increase of PG count due to these rumors. My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450. I heard in related rumors that some users have 1000+ PGs per OSD without problems. I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do. I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object" or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors. An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500? Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

8 24