- ceph-users - lists.ceph.io

Slow benchmarks for rbd vs. rados bench
by Daniel Vogelbacher 16 Mar '25

16 Mar '25

Hi, during deployment of a small testcluster on two existing servers I got multiple benchmark results I don't understand. In summary, I think the rados benchmark values are okay for my limited setup - I got ~80 MiB/s for write, ~300 MiB/s for reads. But for rbd benchmarks, the values dropped down to ~46 MiB/s for writes and 71 MiB/s for reads. Have I missed something for rbd bench? I don't understand why rbd is so much slower than rados. Hint: the productive cluster should get 3 servers total and each server 6x 12TB SAS3 disks and maybe 2 additional servers for up to 5 monitors. I know Ceph makes more fun with > 3, but budget ist limited :( Here are the results of all benches: # rados bench -p testbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_virt-master4_1711455 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 31 15 59.9956 60 0.430283 0.622336 2 16 51 35 69.9923 80 0.565152 0.635417 3 16 74 58 77.3241 92 0.746055 0.663046 4 16 97 81 80.99 92 0.558241 0.646831 5 16 122 106 84.7893 100 0.705829 0.686034 6 16 149 133 88.6553 108 0.496865 0.674404 7 16 170 154 87.9886 84 0.916755 0.674053 8 16 189 173 86.4887 76 4.24615 0.700781 9 16 215 199 88.4327 104 0.60869 0.702643 10 16 235 219 87.588 80 0.882353 0.696558 11 1 235 234 85.0792 60 0.666018 0.692935 Total time run: 11.8131 Total writes made: 235 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 79.5726 Stddev Bandwidth: 16.1087 Max bandwidth (MB/sec): 108 Min bandwidth (MB/sec): 60 Average IOPS: 19 Stddev IOPS: 4.02718 Max IOPS: 27 Min IOPS: 15 Average Latency(s): 0.708564 Stddev Latency(s): 0.440743 Max latency(s): 4.36566 Min latency(s): 0.318937 # rados bench -p testbench 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 87 71 283.925 284 0.354899 0.187907 2 16 161 145 289.941 296 0.122009 0.194255 3 5 235 230 306.611 340 0.231543 0.198982 Total time run: 3.08835 Total reads made: 235 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 304.37 Average IOPS: 76 Stddev IOPS: 7.37111 Max IOPS: 85 Min IOPS: 71 Average Latency(s): 0.200212 Max latency(s): 1.87814 Min latency(s): 0.017529 # rbd bench --io-type write image01 --pool=testbench bench type write io_size 4096 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 11520 11536.2 45 MiB/s 2 23984 12000.2 47 MiB/s 3 38304 12739.6 50 MiB/s 4 52672 13028.9 51 MiB/s 5 63616 12575.7 49 MiB/s 6 71456 11987.4 47 MiB/s 7 82464 11677.5 46 MiB/s 8 96080 11573.9 45 MiB/s 9 110080 11436.1 45 MiB/s 10 124512 12327.4 48 MiB/s 11 136496 13008.2 51 MiB/s 12 148160 13118.5 51 MiB/s 13 163216 13374 52 MiB/s 14 176848 13527 53 MiB/s 15 191120 13132.7 51 MiB/s 16 203376 13376.3 52 MiB/s 17 212576 12924.8 50 MiB/s 18 221120 11534.9 45 MiB/s 19 228032 10196.2 40 MiB/s 20 241584 10240.5 40 MiB/s 21 254176 10095.6 39 MiB/s elapsed: 22 ops: 262144 ops/sec: 11896.4 bytes/sec: 46 MiB/s ~# rbd bench --io-type read image01 --pool=testbench bench type read io_size 4096 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 18368 18384.4 72 MiB/s 2 36032 18024.3 70 MiB/s 3 56768 18928.4 74 MiB/s 4 75472 18872.4 74 MiB/s 5 93264 18656.4 73 MiB/s 6 109232 18173.1 71 MiB/s 7 128320 18458 72 MiB/s 8 146336 17913.9 70 MiB/s 9 164176 17741.1 69 MiB/s 10 184720 18291.6 71 MiB/s 11 202416 18637.2 73 MiB/s 12 216688 17673.9 69 MiB/s 13 236464 18025.9 70 MiB/s 14 255776 18320.4 72 MiB/s elapsed: 14 ops: 262144 ops/sec: 18255.5 bytes/sec: 71 MiB/s Some cluster details (per server): * SAS3 SEAGATE ST12000NM004J * Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz * Dual NIC Intel E810 (LACP, 2x10GBit Link) * LSI SAS3004 PCI-Express Fusion-MPT SAS-3 * 192 GB ECC RAM Ceph Squid deployed via cephadm. ~# ceph -s cluster: id: 7644057a-00f6-11f0-9a0c-eac00fed9338 health: HEALTH_WARN 1 pool(s) do not have an application enabled services: mon: 2 daemons, quorum virt-master4,virt-master3 (age 24h) mgr: virt-master4.lddrxr(active, since 24h), standbys: virt-master3.akkopo osd: 2 osds: 2 up (since 3h), 2 in (since 3h) data: pools: 3 pools, 65 pgs objects: 3.43k objects, 13 GiB usage: 27 GiB used, 22 TiB / 22 TiB avail pgs: 65 active+clean ~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 21.82808 root default -5 10.91399 host virt-master3 1 hdd 10.91399 osd.1 up 1.00000 1.00000 -3 10.91409 host virt-master4 0 hdd 10.91409 osd.0 up 1.00000 1.00000 # ceph osd pool ls detail pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 2.00 pool 2 'kvm' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 33 lfor 0/0/28 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm snappy compression_mode aggressive application rbd read_balance_score 1.06 pool 3 'testbench' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 446 lfor 0/443/441 flags hashpspool,selfmanaged_snaps stripe_width 0 read_balance_score 1.06 osd.0 direct benchmark with 4M and 4K blocksize: root@virt-master4:~# fio --ioengine=sync --filename=/dev/disk/by-id/scsi-35000c500f8bfda83 --direct=1 --sync=1 --rw=write --bs=4M --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=sync, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=172MiB/s][w=43 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=1585455: Sat Mar 15 14:29:21 2025 write: IOPS=42, BW=170MiB/s (178MB/s)(9.97GiB/60013msec); 0 zone resets clat (usec): min=21993, max=40219, avg=23274.37, stdev=963.46 lat (usec): min=22341, max=40453, avg=23511.77, stdev=961.26 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22414], 20.00th=[22676], | 30.00th=[22938], 40.00th=[22938], 50.00th=[23200], 60.00th=[23200], | 70.00th=[23462], 80.00th=[23462], 90.00th=[24249], 95.00th=[24511], | 99.00th=[24773], 99.50th=[31327], 99.90th=[32637], 99.95th=[32637], | 99.99th=[40109] bw ( KiB/s): min=163840, max=180224, per=99.99%, avg=174166.05, stdev=3764.97, samples=119 iops : min= 40, max= 44, avg=42.52, stdev= 0.92, samples=119 lat (msec) : 50=100.00% cpu : usr=1.00%, sys=2.05%, ctx=7648, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2552,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=9.97GiB (10.7GB), run=60013-60013msec root@virt-master4:~# fio --ioengine=sync --filename=/dev/disk/by-id/scsi-35000c500f8bfda83 --direct=1 --sync=1 --rw=randwrite --bs=4M --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=randwrite, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=sync, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=124MiB/s][w=31 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=1588408: Sat Mar 15 14:37:13 2025 write: IOPS=30, BW=123MiB/s (129MB/s)(7360MiB/60028msec); 0 zone resets clat (usec): min=19173, max=77481, avg=32327.62, stdev=6604.03 lat (usec): min=19453, max=77684, avg=32615.39, stdev=6605.82 clat percentiles (usec): | 1.00th=[21103], 5.00th=[23462], 10.00th=[25035], 20.00th=[26870], | 30.00th=[28443], 40.00th=[30016], 50.00th=[31327], 60.00th=[32637], | 70.00th=[34341], 80.00th=[36439], 90.00th=[41157], 95.00th=[45876], | 99.00th=[51119], 99.50th=[53216], 99.90th=[69731], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min=65536, max=147456, per=99.99%, avg=125542.40, stdev=9567.74, samples=120 iops : min= 16, max= 36, avg=30.65, stdev= 2.34, samples=120 lat (msec) : 20=0.22%, 50=98.32%, 100=1.47% cpu : usr=0.90%, sys=1.53%, ctx=5513, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1840,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=7360MiB (7718MB), run=60028-60028msec root@virt-master4:~# fio --ioengine=io_uring --filename=/dev/disk/by-id/scsi-35000c500f8bfda83 --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=37.6MiB/s][w=9628 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=1591836: Sat Mar 15 14:48:18 2025 write: IOPS=9560, BW=37.3MiB/s (39.2MB/s)(2241MiB/60001msec); 0 zone resets slat (usec): min=3, max=242, avg= 5.75, stdev= 3.14 clat (usec): min=3, max=37068, avg=97.67, stdev=81.74 lat (usec): min=84, max=37086, avg=103.42, stdev=81.88 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 89], 10.00th=[ 90], 20.00th=[ 92], | 30.00th=[ 93], 40.00th=[ 94], 50.00th=[ 95], 60.00th=[ 97], | 70.00th=[ 98], 80.00th=[ 100], 90.00th=[ 104], 95.00th=[ 109], | 99.00th=[ 125], 99.50th=[ 159], 99.90th=[ 416], 99.95th=[ 676], | 99.99th=[ 889] bw ( KiB/s): min=13600, max=39344, per=100.00%, avg=38312.13, stdev=2316.38, samples=119 iops : min= 3400, max= 9836, avg=9578.03, stdev=579.09, samples=119 lat (usec) : 4=0.01%, 20=0.01%, 50=0.01%, 100=80.54%, 250=19.17% lat (usec) : 500=0.21%, 750=0.04%, 1000=0.03% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=3.54%, sys=8.67%, ctx=573655, majf=0, minf=52 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,573639,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=37.3MiB/s (39.2MB/s), 37.3MiB/s-37.3MiB/s (39.2MB/s-39.2MB/s), io=2241MiB (2350MB), run=60001-60001msec -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher

2 2

Adding device class to CRUSH rule without data movement
by Hector Martin 15 Mar '25

15 Mar '25

Hi, I have an old Mimic cluster that I'm doing some cleanup work on and adding SSDs, before upgrading to a newer version. As part of adding SSDs, I need to switch the existing CRUSH rules to only use the HDD device class first. Is there some way of doing this that doesn't result in 100% data movement? Simply replacing `step take default` with `step take default class hdd` for every CRUSH rule seems to completely shuffle the cluster data. I tried manually specifying the bucket IDs for the hdd hierarchy to at least be in the same order as the bucket IDs for the primary hierarchy, hoping having them sort the same would end up with the same data distribution, but that didn't work either. Is there some magic incantation to swap around the CRUSH rules/tree so that it results in exactly the same data distribution after adding the hdd class constraint? The set of potential OSDs should be identical (there are no SSDs yet), so the data movement seems to be some technicality of the CRUSH implementation... perhaps completely switching around the main id and hdd-class id of all the buckets would do it? (I'm a little afraid to mess with the main ids in a production cluster...). This cluster is already having I/O load issues (that's part of why I'm adding SSDs), so I'd really like to avoid a total data shuffle if possible. Thanks, - Hector

3 4

Massive performance issues
by Thomas Schneider 15 Mar '25

15 Mar '25

Hi, we are having massive performance issues with our Ceph cluster, and by now I have no idea how and where to debug further. See attachments for full benchmark logs, relevant excerpts: HDD pool: > read: IOPS=7591, BW=119MiB/s (124MB/s)(34.9GiB/300892msec) SSD pool: > read: IOPS=2007, BW=31.4MiB/s (32.9MB/s)(9422MiB/300334msec) Yes, the SSD pool is in fact slower in both IOPS and data rate than the HDD pool. Of course this is only one particular benchmark scenario, but at least it has some numbers and not just “everything feels slow”. I’m happy to run different benchmarks if required. Cluster information: * 2x10G ethernet to a Cisco Catalyst 3850 on each node * Debian 11 (bullseye), Linux 5.10 * Ceph 18.2.4 (reef), managed with cephadm * All but the oldest node serve as VM hypervisors (KVM) as well * 6 nodes (hostnames appended to compare with OSD information later) - 2x AMD EPYC 7232P, 64G RAM (cirrus, nimbus) - 1x Intel Xeon E3-1230 v3, 32G RAM (pileus, no VMs) - 2x Intel Xeon E5-1620 v4, 64G RAM (lenticular, nacreous) - 1x Intel Xeon Silver 4110, 96G RAM (stratus) * Cluster usage: mostly RBD, some RGW and CephFS * OSDs: - 19 Hitachi Ultrastar 7K4000 2TB - 1 WD Ultrastar DC HC310 4TB - 3 Hitachi Deskstar 7k2000 2TB - 13 Hitachi Ultrastar A7K3000 2TB - 8 Micron 5210 ION 2TB - 4 Micron 5300 MAX - Mixed Use 2TB - 5 Samsung Datacenter PM893 2TB You can find various information that I deemed useful below. Please ask if you would like further information. As I said, I don’t really know where to look and what to do, so I would really appreciate any pointers on how to debug this and improve performance. Of course I have already seen the pg warnings, but I am really not sure what to adjust in which direction, especially the contradiction between the MANY_OBJECTS_PER_PG and POOL_TOO_MANY_PGS warnings for the default.rgw.ec.data pool (there are too many objects per pg, so I should probably increase pg_num, however it recommends to scale down from 128 to 32?!). Thanks, Thomas # ceph -s cluster: id: 91688ac0-1b4f-43e3-913d-a844338d9325 health: HEALTH_WARN 1 pools have many more objects per pg than average 3 pools have too many placement groups services: mon: 5 daemons, quorum nimbus,lenticular,stratus,cirrus,nacreous (age 5d) mgr: lenticular(active, since 3M), standbys: nimbus, stratus, nacreous, cirrus mds: 1/1 daemons up, 1 standby osd: 53 osds: 53 up (since 2w), 53 in (since 18M) rgw: 5 daemons active (5 hosts, 1 zones) data: volumes: 1/1 healthy pools: 15 pools, 2977 pgs objects: 7.20M objects, 15 TiB usage: 42 TiB used, 55 TiB / 97 TiB avail pgs: 2975 active+clean 1 active+clean+scrubbing 1 active+clean+scrubbing+deep io: client: 6.5 MiB/s rd, 1.3 MiB/s wr, 158 op/s rd, 63 op/s wr # ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average; 3 pools have too many placement groups [WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than average pool default.rgw.ec.data objects per pg (27197) is more than 11.2524 times cluster average (2417) [WRN] POOL_TOO_MANY_PGS: 3 pools have too many placement groups Pool default.rgw.ec.data has 128 placement groups, should have 32 Pool templates has 512 placement groups, should have 256 Pool rbd.ec has 1024 placement groups, should have 256 # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 67 TiB 29 TiB 39 TiB 39 TiB 57.58 ssd 30 TiB 26 TiB 3.7 TiB 3.7 TiB 12.37 TOTAL 97 TiB 55 TiB 42 TiB 42 TiB 43.74 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL templates 1 512 4.4 TiB 1.27M 13 TiB 37.46 7.4 TiB .rgw.root 9 32 1.4 KiB 4 768 KiB 0 7.4 TiB default.rgw.control 10 32 0 B 8 0 B 0 7.4 TiB default.rgw.meta 11 32 7.0 KiB 38 5.1 MiB 0 7.4 TiB default.rgw.log 12 32 3.5 KiB 208 5.7 MiB 0 7.4 TiB default.rgw.ec.data 14 128 1.3 TiB 3.48M 2.7 TiB 10.95 13 TiB default.rgw.buckets.index 15 32 57 MiB 53 171 MiB 0 7.4 TiB default.rgw.buckets.data 16 32 2.5 GiB 17.85k 8.3 GiB 0.04 7.4 TiB default.rgw.buckets.non-ec 17 32 1.6 KiB 1 197 KiB 0 7.4 TiB rbd.ssd 19 512 872 GiB 259.98k 2.6 TiB 9.46 8.1 TiB .mgr 20 1 640 MiB 161 1.9 GiB 0 7.4 TiB cephfs_metadata 21 32 404 MiB 35.40k 1.2 GiB 0 8.1 TiB cephfs_data 22 32 42 GiB 28.09k 128 GiB 0.56 7.4 TiB rbd 23 512 12 GiB 6.57k 39 GiB 0.17 7.4 TiB rbd.ec 24 1024 8 TiB 2.10M 16 TiB 42.48 11 TiB The pool "templates" is named as such for historical reasons. It is used as RBD VM image storage. # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .rgw.root 256.0k 3.0 68931G 0.0000 1.0 32 off False default.rgw.control 0 3.0 68931G 0.0000 1.0 32 off False default.rgw.meta 1740k 3.0 68931G 0.0000 1.0 32 off False default.rgw.log 1932k 3.0 68931G 0.0000 1.0 32 off False default.rgw.ec.data 1678G 1.6666666269302368 68931G 0.0406 1.0 128 warn False default.rgw.buckets.index 58292k 3.0 68931G 0.0000 1.0 32 off False default.rgw.buckets.data 2828M 3.0 68931G 0.0001 1.0 32 off False default.rgw.buckets.non-ec 67159 3.0 68931G 0.0000 1.0 32 off False .mgr 639.7M 3.0 68931G 0.0000 1.0 1 off False cephfs_metadata 405.4M 3.0 30404G 0.0000 4.0 32 off False cephfs_data 43733M 3.0 68931G 0.0019 1.0 32 off False templates 4541G 3.0 68931G 0.1977 1.0 512 warn True rbd.ssd 870.4G 3.0 30404G 0.0859 1.0 512 warn True rbd 13336M 3.0 68931G 0.0006 1.0 512 off True rbd.ec 8401G 2.0 68931G 0.2438 1.0 1024 warn True # ceph osd pool ls detail pool 1 'templates' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode warn last_change 7495052 lfor 0/2976944/7469772 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application rbd read_balance_score 1.48 pool 9 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467477 flags hashpspool stripe_width 0 application rgw read_balance_score 3.38 pool 10 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467478 flags hashpspool stripe_width 0 application rgw read_balance_score 3.37 pool 11 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467479 flags hashpspool stripe_width 0 application rgw read_balance_score 3.36 pool 12 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467480 flags hashpspool stripe_width 0 application rgw read_balance_score 4.52 pool 14 'default.rgw.ec.data' erasure profile rgw-video size 5 min_size 4 crush_rule 3 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 7494999 lfor 0/5467738/7467533 flags hashpspool stripe_width 12288 application rgw pool 15 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467482 flags hashpspool stripe_width 0 application rgw read_balance_score 4.52 pool 16 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467483 flags hashpspool stripe_width 0 application rgw read_balance_score 3.39 pool 17 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467484 flags hashpspool stripe_width 0 application rgw read_balance_score 3.38 pool 19 'rbd.ssd' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode warn last_change 7495053 lfor 0/141997/7467030 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application rbd read_balance_score 1.43 pool 20 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 7480977 flags hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth read_balance_score 37.50 pool 21 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467487 lfor 0/0/96365 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.59 pool 22 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467488 lfor 0/0/96365 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs read_balance_score 4.52 pool 23 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode off last_change 7467489 lfor 0/0/884114 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application rbd read_balance_score 1.41 pool 24 'rbd.ec' erasure profile rbd size 4 min_size 3 crush_rule 5 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 7495054 lfor 0/2366959/2366961 flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 8192 application rbd # ceph osd utilization avg 192.66 stddev 75.9851 (expected baseline 13.7486) min osd.48 with 88 pgs (0.456762 * mean) max osd.10 with 500 pgs (2.59524 * mean) # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -13 95.71759 - 97 TiB 42 TiB 42 TiB 4.2 GiB 204 GiB 55 TiB 43.74 1.00 - root default -16 14.15759 - 15 TiB 8.2 TiB 8.2 TiB 713 MiB 29 GiB 6.4 TiB 56.36 1.29 - host cirrus 2 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.0 TiB 85 MiB 4.9 GiB 784 GiB 57.94 1.32 223 up osd.2 5 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 50 MiB 3.2 GiB 821 GiB 55.95 1.28 223 up osd.5 8 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 88 MiB 2.9 GiB 804 GiB 56.86 1.30 229 up osd.8 11 hdd 1.76970 1.00000 1.8 TiB 963 GiB 959 GiB 236 MiB 3.6 GiB 900 GiB 51.66 1.18 240 up osd.11 13 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 61 MiB 4.7 GiB 823 GiB 55.81 1.28 220 up osd.13 15 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 52 MiB 3.1 GiB 762 GiB 59.13 1.35 229 up osd.15 18 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.0 TiB 85 MiB 2.7 GiB 787 GiB 57.76 1.32 228 up osd.18 19 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 54 MiB 3.4 GiB 824 GiB 55.79 1.28 219 up osd.19 -18 15.93417 - 16 TiB 6.0 TiB 6.0 TiB 663 MiB 23 GiB 10 TiB 37.15 0.85 - host lenticular 0 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 88 MiB 3.2 GiB 796 GiB 57.26 1.31 229 up osd.0 4 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 42 MiB 6.0 GiB 753 GiB 59.57 1.36 231 up osd.4 6 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 3.5 GiB 758 GiB 59.31 1.36 237 up osd.6 10 hdd 3.63869 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 127 MiB 5.9 GiB 1.7 TiB 52.08 1.19 500 up osd.10 36 ssd 1.74660 1.00000 1.7 TiB 229 GiB 228 GiB 92 MiB 959 MiB 1.5 TiB 12.79 0.29 103 up osd.36 37 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 97 MiB 1006 MiB 1.5 TiB 12.49 0.29 98 up osd.37 38 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 61 MiB 1.0 GiB 1.5 TiB 12.46 0.28 98 up osd.38 39 ssd 1.74660 1.00000 1.7 TiB 221 GiB 219 GiB 102 MiB 1.4 GiB 1.5 TiB 12.35 0.28 98 up osd.39 -17 22.89058 - 23 TiB 9.5 TiB 9.4 TiB 1.1 GiB 45 GiB 14 TiB 40.59 0.93 - host nacreous 1 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 124 MiB 6.0 GiB 818 GiB 56.07 1.28 229 up osd.1 3 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 3.1 GiB 780 GiB 58.11 1.33 222 up osd.3 7 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 48 MiB 3.1 GiB 773 GiB 58.48 1.34 237 up osd.7 9 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 50 MiB 4.9 GiB 766 GiB 58.88 1.35 223 up osd.9 14 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 93 MiB 3.3 GiB 812 GiB 56.43 1.29 224 up osd.14 16 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 43 MiB 5.0 GiB 792 GiB 57.51 1.31 235 up osd.16 17 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 68 MiB 4.2 GiB 777 GiB 58.31 1.33 227 up osd.17 20 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 66 MiB 5.9 GiB 780 GiB 58.15 1.33 231 up osd.20 48 ssd 1.74660 1.00000 1.7 TiB 213 GiB 212 GiB 94 MiB 1.3 GiB 1.5 TiB 11.92 0.27 88 up osd.48 49 ssd 1.74660 1.00000 1.7 TiB 215 GiB 212 GiB 94 MiB 2.5 GiB 1.5 TiB 12.01 0.27 90 up osd.49 50 ssd 1.74660 1.00000 1.7 TiB 214 GiB 212 GiB 95 MiB 2.1 GiB 1.5 TiB 11.99 0.27 92 up osd.50 51 ssd 1.74660 1.00000 1.7 TiB 217 GiB 215 GiB 144 MiB 1.9 GiB 1.5 TiB 12.13 0.28 94 up osd.51 52 ssd 1.74660 1.00000 1.7 TiB 214 GiB 212 GiB 144 MiB 2.0 GiB 1.5 TiB 11.95 0.27 94 up osd.52 -19 6.98639 - 7.0 TiB 895 GiB 889 GiB 345 MiB 5.6 GiB 6.1 TiB 12.51 0.29 - host nimbus 44 ssd 1.74660 1.00000 1.7 TiB 227 GiB 224 GiB 148 MiB 2.3 GiB 1.5 TiB 12.67 0.29 98 up osd.44 45 ssd 1.74660 1.00000 1.7 TiB 222 GiB 221 GiB 68 MiB 1021 MiB 1.5 TiB 12.39 0.28 93 up osd.45 46 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 64 MiB 972 MiB 1.5 TiB 12.48 0.29 97 up osd.46 47 ssd 1.74660 1.00000 1.7 TiB 224 GiB 222 GiB 64 MiB 1.3 GiB 1.5 TiB 12.52 0.29 95 up osd.47 -15 21.19368 - 22 TiB 9.3 TiB 9.3 TiB 848 MiB 56 GiB 12 TiB 43.35 0.99 - host pileus 12 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 54 MiB 8.7 GiB 790 GiB 57.61 1.32 231 up osd.12 21 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 46 MiB 6.1 GiB 765 GiB 58.94 1.35 236 up osd.21 22 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 60 MiB 6.2 GiB 810 GiB 56.52 1.29 228 up osd.22 23 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 83 MiB 8.0 GiB 779 GiB 58.16 1.33 230 up osd.23 24 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 44 MiB 5.5 GiB 777 GiB 58.29 1.33 229 up osd.24 25 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 73 MiB 5.6 GiB 799 GiB 57.09 1.31 230 up osd.25 26 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 63 MiB 5.4 GiB 797 GiB 57.22 1.31 239 up osd.26 27 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 99 MiB 7.1 GiB 720 GiB 61.36 1.40 247 up osd.27 40 ssd 1.74660 1.00000 1.7 TiB 220 GiB 219 GiB 66 MiB 1013 MiB 1.5 TiB 12.31 0.28 96 up osd.40 41 ssd 1.74660 1.00000 1.7 TiB 221 GiB 220 GiB 63 MiB 981 MiB 1.5 TiB 12.36 0.28 97 up osd.41 42 ssd 1.74660 1.00000 1.7 TiB 228 GiB 227 GiB 98 MiB 999 MiB 1.5 TiB 12.73 0.29 101 up osd.42 43 ssd 1.74660 1.00000 1.7 TiB 227 GiB 226 GiB 99 MiB 866 MiB 1.5 TiB 12.67 0.29 100 up osd.43 -14 14.55518 - 15 TiB 8.6 TiB 8.5 TiB 582 MiB 45 GiB 6.0 TiB 59.02 1.35 - host stratus 28 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 5.5 GiB 756 GiB 59.41 1.36 239 up osd.28 29 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 44 MiB 4.7 GiB 759 GiB 59.27 1.35 230 up osd.29 30 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 68 MiB 5.0 GiB 753 GiB 59.57 1.36 232 up osd.30 31 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 64 MiB 6.2 GiB 778 GiB 58.26 1.33 229 up osd.31 32 hdd 1.81940 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 85 MiB 5.0 GiB 797 GiB 57.24 1.31 226 up osd.32 33 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 112 MiB 4.1 GiB 749 GiB 59.81 1.37 235 up osd.33 34 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 46 MiB 8.5 GiB 770 GiB 58.66 1.34 238 up osd.34 35 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 109 MiB 6.2 GiB 747 GiB 59.92 1.37 244 up osd.35 TOTAL 97 TiB 42 TiB 42 TiB 4.2 GiB 204 GiB 55 TiB 43.74 MIN/MAX VAR: 0.27/1.40 STDDEV: 21.24 # ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.040070", "last_optimize_started": "Thu Mar 13 14:59:05 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } -- Fachschaft I/1 Mathematik/Physik/Informatik der RWTH Aachen Thomas Schneider Campus Mitte: Augustinerbach 2a, 52062 Aachen Telefon: +49 241 80 94506 Informatikzentrum: Ahornstraße 55, Raum 4U17, 52074 Aachen Telefon: +49 241 80 26741 https://www.fsmpi.rwth-aachen.de # fio --ioengine=libaio --direct=1 --bs=16384 --iodepth=128 --rw=randread --norandommap --size=20G --numjobs=1 --runtime=300 --time_based --name=/dev/rbd/rbd/test /dev/rbd/rbd/test: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128 fio-3.25 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=35.0MiB/s][r=2241 IOPS][eta 00m:00s] /dev/rbd/rbd/test: (groupid=0, jobs=1): err= 0: pid=3832471: Thu Jan 30 19:55:46 2025 read: IOPS=7591, BW=119MiB/s (124MB/s)(34.9GiB/300892msec) slat (usec): min=2, max=665, avg= 8.93, stdev= 5.57 clat (usec): min=65, max=5330.5k, avg=16850.13, stdev=184056.88 lat (usec): min=82, max=5330.5k, avg=16859.23, stdev=184056.97 clat percentiles (usec): | 1.00th=[ 147], 5.00th=[ 180], 10.00th=[ 202], | 20.00th=[ 237], 30.00th=[ 269], 40.00th=[ 297], | 50.00th=[ 330], 60.00th=[ 367], 70.00th=[ 420], | 80.00th=[ 523], 90.00th=[ 1876], 95.00th=[ 6128], | 99.00th=[ 362808], 99.50th=[1098908], 99.90th=[3506439], | 99.95th=[4143973], 99.99th=[4731175] bw ( KiB/s): min= 768, max=478432, per=100.00%, avg=121819.01, stdev=95962.09, samples=600 iops : min= 48, max=29902, avg=7613.70, stdev=5997.62, samples=600 lat (usec) : 100=0.01%, 250=24.52%, 500=53.85%, 750=8.61%, 1000=1.56% lat (msec) : 2=1.84%, 4=3.46%, 10=2.34%, 20=1.12%, 50=0.91% lat (msec) : 100=0.40%, 250=0.31%, 500=0.13%, 750=0.18%, 1000=0.21% lat (msec) : 2000=0.34%, >=2000=0.20% cpu : usr=2.33%, sys=10.06%, ctx=1687235, majf=0, minf=523 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=2284212,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=119MiB/s (124MB/s), 119MiB/s-119MiB/s (124MB/s-124MB/s), io=34.9GiB (37.4GB), run=300892-300892msec

4 4

[RGW] Full replication gives stale recovering shard
by Gilles Mocellin 14 Mar '25

14 Mar '25

Hello Cephers, Since I didn't have any progress on my side, I share again my problem, hoping for some clues. --- Since I was not confident in my replication status, I've done a radosgw sync init one after the other, in both of my zones. Since then, there art stale recovering shards. Incremental replication seems OK. Cluster A sync status: realm 421c9bbd-83cc-4d85-a25a-ca847f225bfe (R) zonegroup 2c5ccf52-88be-4b2f-b4b9-c55f73bbacd1 (ZG) zone 3e3b0a2b-1864-4923-93a6-237a13b51594 (Z1) current time 2025-02-06T19:03:10Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: aefd4003-1866-4b16-b1b3-2f308075cd1c (Z2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [65,66,77] oldest incremental change not applied: 2025-02-06T20:02:53.441838+0100 [66] 111 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101, 102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] Cluster B sync status: realm 421c9bbd-83cc-4d85-a25a-ca847f225bfe (R) zonegroup 2c5ccf52-88be-4b2f-b4b9-c55f73bbacd1 (ZG) zone aefd4003-1866-4b16-b1b3-2f308075cd1c (Z2) current time 2025-02-06T19:03:37Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 3e3b0a2b-1864-4923-93a6-237a13b51594 (Z1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 7 shards behind shards: [58,59,60,62,64,66,67] oldest incremental change not applied: 2025-02-06T20:03:27.831651+0100 [60] 81 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,57,58,59,60,61,62,63,64,65,66,67,68,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123 ,124,126,127] Restarting full sync does nothing, except eventually adding recovering shards ! I also have some large omap objects in Z1.rgw.log pool. Where I see a lot of objects with names beginning by bucket.sync-status.*. What can I do to remove those recovering shards ? Some trimming (bilog ?), deleting objects in Z1.rgw.log pool ? I have some errors in radosgw-admin sync error list, but not really usefull : failed to sync bucket instance: (125) Operation canceled => A racing condition exists between writes to the same RADOS object. I also tried to rewrite all objects in buckets listed in thoses errors, with minimal size=0. Nothing changed. Thank you for reading. Regards, -- Gilles

1 1

Archive Sync Module does not add a Delete Marker when object is deleted
by motaharesdq＠gmail.com 14 Mar '25

14 Mar '25

Hello everyone, I was testing out the Archive sync module with 2 zones, one as the master zone and another one as a read-only archive tier zone. The sync policy is set with a standard hierarchy of Group/Flow/Pipe. Came across this scenario: overwriting an object in master zone does result in adding a version and changing latest on the archive zone, but deleting the object in the master zone does not add a delete marker in secondary, and the object remains as latest in the archive zone. This way even setting lifecycle wouldn't help deleting that, since it's not a non-current version. Is this the standard behaviour of archive module? And how should you delete something permanently? Kind Regards

1 0

How to (permanently) disable msgr v1 on Ceph?
by Stefan Kooman 14 Mar '25

14 Mar '25

Hi, For new clusters one of the first things I do is to disable messenger v1: ceph config set mon ms_bind_msgr1 false However, that is not enough, as a restart of the monitors will leave v1 enabled in the monmap. Why is that? That looks like a bug to me. Therefore I explicitly set the addrs and port of the monitors, i.e.: ceph mon set-addrs stefan-ceph-test01 $ip:3300 Be careful not to make a typo, that will ruin your day, see [1]. This immediately disables v1 and a new monmap is created. All good, however ... When a monitor is removed, say mon1, and added again, the new mon is added with both a v1 and a v2 address. I don't want that, I want to disable legacy v1 permanently. How do I do that? Adding: [mon] ms_bind_msgr1 = false to /etc/ceph/ceph.conf does not help either. When I performed a test this morning on a fresh 18.2.4 cluster (remove a mon, add it back again) to reproduce this issue the v1 was enabled again on two monitors (a third was left with only v2). Repeating the "ceph mon set-addrs" command removes the v1 binding again. But I would like to avoid messing around with this, triggering monmaps / elections, especially when making an error with this brings your cluster to a halt [1]. In what Ceph release will v1 be disabled by default on new clusters? Gr. Stefan [1] https://tracker.ceph.com/issues/54744

4 8

Re: ceph orch command not working anymore on squid (19.2.1)
by Eugen Block 13 Mar '25

13 Mar '25

(Don't drop the list in your responses) There's a lot going on in the cephadm.log, it's not clear which of these are relevant to the issue. This one looks bad though: ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not appear to be an IPv4 or IPv6 address Did you build this cluster with 19.2.1 or did you upgrade? Do you have the output from 'ceph orch host ls' lying around (I know, currently it's not working)? Are you using IPv6? I haven't seen this issue yet. Zitat von Moritz Baumann <mo(a)mo.homeip.net>: > Am 09.03.2025 um 15:22 schrieb Moritz Baumann: >> I dont want to sound stupid, but how do I get the log? >> >> Am 09.03.2025 um 11:11 schrieb Eugen Block: >>> ceph versions >> > > > ceph01[22]:~# cephadm shell -- ceph log last cephadm > Inferring fsid 4cfb1138-d2a5-11ef-87eb-1c34da408e52 > Inferring config > /var/lib/ceph/4cfb1138-d2a5-11ef-87eb-1c34da408e52/mon.ceph01/config > Not using image > 'f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d' > as it's not in list of non-dangling images with ceph=True label > 2025-02-25T08:42:54.590667+0000 mgr.ceph01.ombogc (mgr.180813) 1706 > : cephadm [INF] maintenance mode request for ceph05.inf.ethz.ch has > SET the noout group > 2025-02-25T08:43:38.380838+0000 mgr.ceph01.ombogc (mgr.180813) 1729 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:44:40.343420+0000 mgr.ceph01.ombogc (mgr.180813) 1762 > : cephadm [INF] Adjusting osd_memory_target on ceph04.inf.ethz.ch to > 33423M > 2025-02-25T08:44:40.860933+0000 mgr.ceph01.ombogc (mgr.180813) 1764 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:45:41.993564+0000 mgr.ceph01.ombogc (mgr.180813) 1797 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:46:42.758717+0000 mgr.ceph01.ombogc (mgr.180813) 1830 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:47:44.190946+0000 mgr.ceph01.ombogc (mgr.180813) 1864 > : cephadm [ERR] Unable to reach remote host ceph05.inf.ethz.ch. SSH > connection closed > Traceback (most recent call last): > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 245, in _execute_command > r = await conn.run(str(rcmd), input=stdin) > File "/lib/python3.9/site-packages/asyncssh/connection.py", line > 4250, in run > process = await self.create_process(*args, **kwargs) # type: ignore > File "/lib/python3.9/site-packages/asyncssh/connection.py", line > 4128, in create_process > chan, process = await self.create_session( > File "/lib/python3.9/site-packages/asyncssh/connection.py", line > 4018, in create_session > chan = SSHClientChannel(self, self._loop, encoding, errors, > File "/lib/python3.9/site-packages/asyncssh/channel.py", line > 1085, in __init__ > super().__init__(conn, loop, encoding, errors, window, max_pktsize) > File "/lib/python3.9/site-packages/asyncssh/channel.py", line 140, > in __init__ > self._recv_chan: Optional[int] = conn.add_channel(self) > File "/lib/python3.9/site-packages/asyncssh/connection.py", line > 1307, in add_channel > raise ChannelOpenError(OPEN_CONNECT_FAILED, > asyncssh.misc.ChannelOpenError: SSH connection closed > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper > return OrchResult(f(*args, **kwargs)) > File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper > return func(*args, **kwargs) > File "/usr/share/ceph/mgr/cephadm/module.py", line 2224, in > exit_host_maintenance > outs, errs, _code = self.wait_async( > File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async > return self.event_loop.get_result(coro, timeout) > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result > return future.result(timeout) > File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result > return self.__get_result() > File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in > __get_result > raise self._exception > File "/usr/share/ceph/mgr/cephadm/serve.py", line 1669, in _run_cephadm > python = await self.mgr.ssh._check_execute_command(host, cmd, addr=addr) > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 300, in > _check_execute_command > out, err, code = await self._execute_command(host, cmd, stdin, > addr, log_command) > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 253, in _execute_command > raise HostConnectionError(f'Unable to reach remote host {host}. > {str(e)}', host, address) > cephadm.ssh.HostConnectionError: Unable to reach remote host > ceph05.inf.ethz.ch. SSH connection closed > 2025-02-25T08:47:44.762770+0000 mgr.ceph01.ombogc (mgr.180813) 1865 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:47:55.631018+0000 mgr.ceph01.ombogc (mgr.180813) 1874 > : cephadm [INF] exit maintenance request has UNSET for the noout > group on host ceph05.inf.ethz.ch > 2025-02-25T08:48:47.623644+0000 mgr.ceph01.ombogc (mgr.180813) 1901 > : cephadm [INF] Adjusting osd_memory_target on ceph05.inf.ethz.ch to > 32911M > 2025-02-25T08:48:47.827026+0000 mgr.ceph01.ombogc (mgr.180813) 1903 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:49:21.108932+0000 mgr.ceph01.ombogc (mgr.180813) 1925 > : cephadm [INF] Schedule redeploy daemon grafana.ceph05 > 2025-02-25T08:49:21.292879+0000 mgr.ceph01.ombogc (mgr.180813) 1926 > : cephadm [INF] Regenerating cephadm self-signed grafana TLS > certificates > 2025-02-25T08:49:21.328348+0000 mgr.ceph01.ombogc (mgr.180813) 1929 > : cephadm [INF] Deploying daemon grafana.ceph05 on ceph05.inf.ethz.ch > 2025-02-25T09:07:00.824841+0000 mgr.ceph01.ombogc (mgr.180813) 2461 > : cephadm [INF] Detected new or changed devices on ceph05.inf.ethz.ch > 2025-02-25T09:07:01.376245+0000 mgr.ceph01.ombogc (mgr.180813) 2462 > : cephadm [INF] Detected new or changed devices on ceph04.inf.ethz.ch > 2025-02-28T06:18:29.439683+0000 mgr.ceph01.ombogc (mgr.180813) > 137418 : cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE > ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not > appear to be an IPv4 or IPv6 address') > Traceback (most recent call last): > File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, > in communicate > req.parse_request() > File "/lib/python3.9/site-packages/cheroot/server.py", line 714, > in parse_request > success = self.read_request_line() > File "/lib/python3.9/site-packages/cheroot/server.py", line 815, > in read_request_line > scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) > File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit > _check_bracketed_host(bracketed_host) > File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host > ip = ipaddress.ip_address(hostname) # Throws Value Error if not > IPv6 or IPv4 > File "/lib64/python3.9/ipaddress.py", line 53, in ip_address > raise ValueError(f'{address!r} does not appear to be an IPv4 or > IPv6 address') > ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not > appear to be an IPv4 or IPv6 address > > 2025-02-28T06:18:29.441452+0000 mgr.ceph01.ombogc (mgr.180813) > 137419 : cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE > ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not > appear to be an IPv4 or IPv6 address') > Traceback (most recent call last): > File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, > in communicate > req.parse_request() > File "/lib/python3.9/site-packages/cheroot/server.py", line 714, > in parse_request > success = self.read_request_line() > File "/lib/python3.9/site-packages/cheroot/server.py", line 815, > in read_request_line > scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) > File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit > _check_bracketed_host(bracketed_host) > File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host > ip = ipaddress.ip_address(hostname) # Throws Value Error if not > IPv6 or IPv4 > File "/lib64/python3.9/ipaddress.py", line 53, in ip_address > raise ValueError(f'{address!r} does not appear to be an IPv4 or > IPv6 address') > ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not > appear to be an IPv4 or IPv6 address > > 2025-03-07T06:17:28.282574+0000 mgr.ceph01.ombogc (mgr.180813) > 447472 : cephadm [ERR] [07/Mar/2025:06:17:28] ENGINE > ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not > appear to be an IPv4 or IPv6 address') > Traceback (most recent call last): > File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, > in communicate > req.parse_request() > File "/lib/python3.9/site-packages/cheroot/server.py", line 714, > in parse_request > success = self.read_request_line() > File "/lib/python3.9/site-packages/cheroot/server.py", line 815, > in read_request_line > scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) > File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit > _check_bracketed_host(bracketed_host) > File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host > ip = ipaddress.ip_address(hostname) # Throws Value Error if not > IPv6 or IPv4 > File "/lib64/python3.9/ipaddress.py", line 53, in ip_address > raise ValueError(f'{address!r} does not appear to be an IPv4 or > IPv6 address') > ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not > appear to be an IPv4 or IPv6 address > > 2025-03-09T07:36:00.522250+0000 mgr.ceph01.ombogc (mgr.180813) > 536135 : cephadm [ERR] ALERT: Cannot stop active Mgr daemon, Please > switch active Mgrs with 'ceph mgr fail ceph01.ombogc' > Note: Warnings can be bypassed with the --force flag > Traceback (most recent call last): > File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper > return OrchResult(f(*args, **kwargs)) > File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper > return func(*args, **kwargs) > File "/usr/share/ceph/mgr/cephadm/module.py", line 2165, in > enter_host_maintenance > raise OrchestratorError( > orchestrator._interface.OrchestratorError: ALERT: Cannot stop active > Mgr daemon, Please switch active Mgrs with 'ceph mgr fail > ceph01.ombogc' > Note: Warnings can be bypassed with the --force flag

3 10

ceph orch command not working anymore on squid (19.2.1)
by Moritz Baumann 13 Mar '25

13 Mar '25

Hi, we are running a Ceph cluster (on almalinux-9 installed with cephadm) and after I failed the active ceph mgr with "ceph mgr fail "ceph01.xxxxx" to do a "ceph orch host maintenance enter ceph01" I got a: Error ENOENT: Module not found I then did a: ceph01[0]:~# ceph orch set backend "" ceph01[0]:~# ceph orch set backend "cephadm" Error ENOENT: Module not found ceph01[2]:~# ceph orch host ls Error ENOENT: No orchestrator configured (try `ceph orch set backend`) I do I get my orchestrator back to up and running? Best Moritz

3 6

Ceph with 3 nodes and hybrid storage policy: how to configure OSDs with different HDD and SSD sizes
by Daniel Vogelbacher 12 Mar '25

12 Mar '25

Hi, I want to setup a 3-node Ceph cluster with fault domain configured to "host". Each node should be equipped with: 6x SAS3 HDD 12TB 1x SAS3 SSD 7TB (should be extended to 2x7 later) The ceph configuration should be size=3, min_size=2. All nodes are connected with 2x10Gbit (LACP). I want to use different CRUSH rules for different pools. CephFS and low priority/IO VMs stored on RBD should use only HDD drives with default replication CRUSH rule. For high priority VMs, I want to create another RBD data pool which uses a modified CRUSH replication rule: |# Hybrid storage policy rule hybrid { ruleset 2 type replicated step take ssd step chooseleaf firstn 1 type host step emit step take hdd step chooseleaf firstn -1 type host step emit } | |For pools using this hybrid rule, PGs are stored on one SSD (primary) and two HDD (secondary) devices. But these have different sizes in my hardware setup. What happens with the remaining disk space (12-7=5) 5GB on the secondary devices? Is it just unusable, or will ceph use it for other pools with default replication? In any case, I don't bother about these 5GB, just want to know how it works. For the above setup, can you recommend any important configuration settings and should I modify the OSD weighting? Thanks. |-- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher

3 7

RGW multisite metadata sync issue
by Vahideh Alinouri 11 Mar '25

11 Mar '25

Hi guys, My Ceph release is Quincy 17.2.5. I need to change the master zone to decommission the old one and upgrade all zones. I have separated the client traffic and sync traffic in RGWs, meaning there are separate RGW daemons handling the sync process. I encountered an issue when trying to sync one of the zones in the zonegroup. The data sync is proceeding fine, but I have an issue with the metadata sync. It gets stuck behind on a shard. Here is the output from radosgw-admin sync status: metadata sync syncing full sync: 1/64 shards full sync: 135 entries to sync incremental sync: 63/64 shards metadata is behind on 1 shard behind shards: [0] In the RGW log, I see this error: 2024-12-15T21:30:59.641+0000 7f6dff472700 1 beast: 0x7f6d2f1cf730: 172.19.66.112 - s3-cdn-user [15/Dec/2024:21:30:59.641 +0000] "GET /admin/log/?type=data&id=56&marker=00000000000000000000%3A00000000000000204086&extra-info=true&rgwx-zonegroup=7c01d60f-88c6-4192-baf7-d725260bf05d HTTP/1.1" 200 44 - - - latency=0.000000000s 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 I’ve tried the following steps: - Changed the PG number of the metadata pool to force a rebalance, but everything was fine. - Ran metadata sync init and tried to run it again. - Restarted RGW services in both the zone and the master zone. - Created a user in the master zone to ensure metadata sync works, which was successful. - Checked OSD logs but didn’t see any specific errors. - Attempted to list metadata in the pool using rados ls -p s3-cdn-dc07.rgw.meta, but got an empty result. - Compared the code for listing OMAP keys between Quincy and Squid versions; there were no specific changes. I’m looking for any advice or suggestions to resolve this issue.

3 6