Hi all, we have a new Ceph cluster and have some very bad/strange performance behavior. I really don't understand what I'm doing wrong here and would be more than happy if anyone has an idea. Even a hint on what to look at would be helpful. Some Information: Machines (8 Nodes) per Node: - CPU 2x Intel(R) Xeon(R) Gold 6258R CPU @ 2.70GHz (28 Cores) - 384 GB RAM - 20x Dell Ent NVMe AGN RI U.2 7.68TB (for OSDs) - 4x 25G LACP Backend - 2x 25G LACP Frontend - OS: - Ubuntu 22.04 - Kernel: 5.15.0 - Ceph: - Version 18.2.4 - 160 osds - 4096 PGs for the VM pool I took some fio benchmarks from the Proxmox Ceph Performance Paper: https://www.proxmox.com/images/download/pve/docs/Proxmox-VE_Ceph-Benchmark-2... The First test should have about 1500 IOPS (Proxmox Paper: 1806). We only get 170. root@ceph001:/mnt# fio --ioengine=psync --filename=test_fio --size=9G --time_based --name=fio --group_reporting --runtime=60 --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.28 Starting 1 process fio: Laying out IO file (1 file / 9216MiB) Jobs: 1 (f=1): [W(1)][100.0%][w=680KiB/s][w=170 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=174797: Wed Jan 22 20:19:19 2025 write: IOPS=202, BW=811KiB/s (831kB/s)(47.5MiB/60003msec); 0 zone resets clat (usec): min=2185, max=20081, avg=4925.43, stdev=931.63 lat (usec): min=2186, max=20082, avg=4926.19, stdev=931.63 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 3818], 10.00th=[ 3982], 20.00th=[ 4293], | 30.00th=[ 4490], 40.00th=[ 4686], 50.00th=[ 4817], 60.00th=[ 5014], | 70.00th=[ 5211], 80.00th=[ 5407], 90.00th=[ 5800], 95.00th=[ 6063], | 99.00th=[ 8586], 99.50th=[ 9503], 99.90th=[12256], 99.95th=[13304], | 99.99th=[19006] bw ( KiB/s): min= 672, max= 1000, per=100.00%, avg=813.11, stdev=73.18, samples=119 iops : min= 168, max= 250, avg=203.28, stdev=18.29, samples=119 lat (msec) : 4=10.24%, 10=89.43%, 20=0.32%, 50=0.01% cpu : usr=0.25%, sys=2.57%, ctx=36503, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,

=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,12167,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=811KiB/s (831kB/s), 811KiB/s-811KiB/s (831kB/s-831kB/s), io=47.5MiB (49.8MB), run=60003-60003msec Disk stats (read/write): rbd0: ios=0/24296, merge=0/2, ticks=0/56351, in_queue=56351, util=99.97% Bandwith and IOPs with more IO depth look ok form me: fio --filename=/mnt/testingfio1 --size=50GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=150 --numjobs=1 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1 iops-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256 fio-3.28 iops-test-job: (groupid=0, jobs=1): err= 0: pid=146931: Wed Jan 22 19:43:14 2025 read: IOPS=20.0k, BW=78.0MiB/s (81.8MB/s)(11.4GiB/150006msec) slat (nsec): min=1245, max=7415.7k, avg=22636.97, stdev=224525.03 clat (usec): min=238, max=32714, avg=5620.53, stdev=2255.97 lat (usec): min=243, max=32721, avg=5643.32, stdev=2258.28 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2311], 10.00th=[ 2671], 20.00th=[ 3654], | 30.00th=[ 4146], 40.00th=[ 4752], 50.00th=[ 5342], 60.00th=[ 6128], | 70.00th=[ 6915], 80.00th=[ 7635], 90.00th=[ 8717], 95.00th=[ 9896], | 99.00th=[10683], 99.50th=[10945], 99.90th=[11863], 99.95th=[12649], | 99.99th=[14615] bw ( KiB/s): min=63254, max=98432, per=100.00%, avg=79914.04, stdev=6290.69, samples=299 iops : min=15813, max=24608, avg=19978.36, stdev=1572.70, samples=299 write: IOPS=19.9k, BW=77.9MiB/s (81.7MB/s)(11.4GiB/150006msec); 0 zone resets slat (nsec): min=1349, max=8871.7k, avg=23250.80, stdev=225370.53 clat (usec): min=629, max=81108, avg=7160.58, stdev=2338.33 lat (usec): min=633, max=81114, avg=7183.98, stdev=2348.98 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3982], 10.00th=[ 4293], 20.00th=[ 5014], | 30.00th=[ 5735], 40.00th=[ 6325], 50.00th=[ 6980], 60.00th=[ 7570], | 70.00th=[ 8225], 80.00th=[ 9110], 90.00th=[10421], 95.00th=[11207], | 99.00th=[13435], 99.50th=[14353], 99.90th=[16581], 99.95th=[17957], | 99.99th=[21365] bw ( KiB/s): min=61755, max=98813, per=100.00%, avg=79877.64, stdev=6336.82, samples=299 iops : min=15438, max=24703, avg=19969.22, stdev=1584.22, samples=299 lat (usec) : 250=0.01%, 500=0.03%, 750=0.07%, 1000=0.08% clat (usec): min=238, max=32714, avg=5620.53, stdev=2255.97 [141/1761] lat (usec): min=243, max=32721, avg=5643.32, stdev=2258.28 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2311], 10.00th=[ 2671], 20.00th=[ 3654], | 30.00th=[ 4146], 40.00th=[ 4752], 50.00th=[ 5342], 60.00th=[ 6128], | 70.00th=[ 6915], 80.00th=[ 7635], 90.00th=[ 8717], 95.00th=[ 9896], | 99.00th=[10683], 99.50th=[10945], 99.90th=[11863], 99.95th=[12649], | 99.99th=[14615] bw ( KiB/s): min=63254, max=98432, per=100.00%, avg=79914.04, stdev=6290.69, samples=299 iops : min=15813, max=24608, avg=19978.36, stdev=1572.70, samples=299 write: IOPS=19.9k, BW=77.9MiB/s (81.7MB/s)(11.4GiB/150006msec); 0 zone resets slat (nsec): min=1349, max=8871.7k, avg=23250.80, stdev=225370.53 clat (usec): min=629, max=81108, avg=7160.58, stdev=2338.33 lat (usec): min=633, max=81114, avg=7183.98, stdev=2348.98 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3982], 10.00th=[ 4293], 20.00th=[ 5014], | 30.00th=[ 5735], 40.00th=[ 6325], 50.00th=[ 6980], 60.00th=[ 7570], | 70.00th=[ 8225], 80.00th=[ 9110], 90.00th=[10421], 95.00th=[11207], | 99.00th=[13435], 99.50th=[14353], 99.90th=[16581], 99.95th=[17957], | 99.99th=[21365] bw ( KiB/s): min=61755, max=98813, per=100.00%, avg=79877.64, stdev=6336.82, samples=299 iops : min=15438, max=24703, avg=19969.22, stdev=1584.22, samples=299 lat (usec) : 250=0.01%, 500=0.03%, 750=0.07%, 1000=0.08% lat (msec) : 2=0.65%, 4=15.15%, 10=75.45%, 20=8.55%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=7.36%, sys=18.55%, ctx=155494, majf=0, minf=9200 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%,

=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.1% issued rwts: total=2993949,2992429,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs): READ: bw=78.0MiB/s (81.8MB/s), 78.0MiB/s-78.0MiB/s (81.8MB/s-81.8MB/s), io=11.4GiB (12.3GB), run=150006-150006msec WRITE: bw=77.9MiB/s (81.7MB/s), 77.9MiB/s-77.9MiB/s (81.7MB/s-81.7MB/s), io=11.4GiB (12.3GB), run=150006-150006msec Disk stats (read/write): rbd0: ios=2989470/2987987, merge=0/1, ticks=9760844/13043854, in_queue=22804699, util=100.00% We have 4096 PG on the tested pool. root@ceph001:/mnt# ceph -s cluster: id: health: HEALTH_OK services: mon: 5 daemons, quorum ceph001,ceph002,ceph003,ceph005,ceph006 (age 52m) mgr: ceph002.hgppdu(active, since 2d), standbys: ceph001.ooznoq osd: 160 osds: 160 up (since 5w), 160 in (since 5M) rgw: 2 daemons active (2 hosts, 1 zones) data: pools: 11 pools, 8449 pgs objects: 7.80M objects, 19 TiB usage: 56 TiB used, 1.0 PiB / 1.1 PiB avail root@ceph001:~# ceph config get osd osd_memory_target 4294967296 root@ceph001:~# ceph config get osd WHO MASK LEVEL OPTION VALUE ... osd advanced osd_memory_target_autotune true ... We would like to use the cluster with OpenStack Cinder. But the tests were made directly on the cluster nodes with rbd. The metrics on VMs are similar. Thanks in advance. Jan