I completely agree that the test I did is not suitable for testing ceph performance. I merely did the same command as the OP and obtained very different results. I suspect the performance difference is much more due to things like network, OS config, memory constraints, etc. But that needs a rigorous set of tests to be done on the infrastructure before looking at ceph itself. For example, we found that many 10g NICs don't handle full-sized jumbo frames (9000) without a performance hit. We got a huge performance improvement by finding the exact optimal MTU (5139 on a cluster I am looking at - at 5140 it falls off a cliff). Just one example. Chris On 25/03/2025 13:35, Konold, Martin wrote:
Am 2025-03-20 15:15, schrieb Chris Palmer:
Hi,
* Ceph cluster 19.2.1 with 3 nodes, 4 x SATA disks with shared NVMe DB/WAL, single 10g NICs * Promox 8.3.5 cluster with 2 nodes (separate nodes to Ceph), single 10g NICs , single 1g NICs for corosync * Test VM was using KRBD R3 pool on HDD, iothread=1, aio=io_uring, cache=writeback
"cache=writeback" measures basically the memory bandwidth of the kvm-host not the Ceph performance. This gets even worse with --size=1G instead of more realistic 10G.
Please remove the writeback cache and increase the filesize to at least 10G in order to measure Ceph rbd performance
Last but not least in order to make the benchmark independent of previous runs you shall please drop the caches.
Overview Results for blocksize=4K =================================
NVMe with cache=none (10 GB Fibre) | read: IOPS = 12.9k, BW = 50.3 MiB/s NVMe with cache=writeback (10 GB Fibre)| read: IOPS = 58.3k, BW = 228.0 MiB/s HDD with cache=none (10 GB Fibre) | read: IOPS = 0.34k, BW = 1.3 MiB/s HDD with cache=writeback (10 GB Fibre)| read: IOPS = 53.8k, BW = 210.0 MiB/s
Detailed reports ================ NVME with cache=none
# echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=39.0MiB/s][r=9981 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=443550: Tue Mar 25 13:25:10 2025 read: IOPS=12.9k, BW=50.3MiB/s (52.7MB/s)(3017MiB/60001msec) slat (usec): min=2, max=13134, avg=307.13, stdev=154.43 clat (usec): min=2, max=28147, avg=4662.15, stdev=1048.17 lat (usec): min=384, max=28432, avg=4969.28, stdev=1106.45 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3261], 10.00th=[ 3589], 20.00th=[ 4080], | 30.00th=[ 4293], 40.00th=[ 4424], 50.00th=[ 4555], 60.00th=[ 4621], | 70.00th=[ 4817], 80.00th=[ 5080], 90.00th=[ 5932], 95.00th=[ 6325], | 99.00th=[ 8225], 99.50th=[ 9634], 99.90th=[12911], 99.95th=[14353], | 99.99th=[17957] bw ( KiB/s): min=26816, max=58136, per=100.00%, avg=51622.66, stdev=1587.61, samples=476 iops : min= 6704, max=14534, avg=12905.66, stdev=396.90, samples=476 lat (usec) : 4=0.01%, 10=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=0.03%, 4=18.10%, 10=81.47%, 20=0.40%, 50=0.01% cpu : usr=1.52%, sys=8.43%, ctx=726528, majf=0, minf=109 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=772457,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=50.3MiB/s (52.7MB/s), 50.3MiB/s-50.3MiB/s (52.7MB/s-52.7MB/s), io=3017MiB (3164MB), run=60001-60001msec
Disk stats (read/write): sdc: ios=725474/0, sectors=5803792/0, merge=0/0, ticks=223185/0, in_queue=223185, util=99.89%
NVMe with cache=writeback # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 3 (f=3): [r(3),_(1)][65.9%][r=31.9MiB/s][r=8165 IOPS][eta 00m:31s] registry-read: (groupid=0, jobs=4): err= 0: pid=442595: Tue Mar 25 13:19:11 2025 read: IOPS=58.3k, BW=228MiB/s (239MB/s)(13.3GiB/60001msec) slat (usec): min=2, max=5185, avg=58.95, stdev=131.23 clat (nsec): min=1440, max=15294k, avg=908789.46, stdev=1764896.35 lat (usec): min=3, max=15367, avg=967.74, stdev=1880.61 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 58], 10.00th=[ 67], 20.00th=[ 117], | 30.00th=[ 133], 40.00th=[ 165], 50.00th=[ 196], 60.00th=[ 239], | 70.00th=[ 347], 80.00th=[ 873], 90.00th=[ 4146], 95.00th=[ 6063], | 99.00th=[ 6652], 99.50th=[ 6980], 99.90th=[ 8717], 99.95th=[ 9503], | 99.99th=[11076] bw ( KiB/s): min=203632, max=602000, per=100.00%, avg=387963.51, stdev=30229.81, samples=420 iops : min=50908, max=150500, avg=96990.60, stdev=7557.47, samples=420 lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=15.15%, 250=46.78%, 500=10.58%, 750=5.17%, 1000=6.62% lat (msec) : 2=3.90%, 4=1.46%, 10=10.29%, 20=0.03% cpu : usr=2.96%, sys=15.72%, ctx=1029435, majf=0, minf=108 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=3499461,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=13.3GiB (14.3GB), run=60001-60001msec
Disk stats (read/write): sdb: ios=1028680/0, sectors=8229440/0, merge=0/0, ticks=180767/0, in_queue=180766, util=99.60%
HDD with cache=none # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=1033KiB/s][r=258 IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=442579: Tue Mar 25 13:15:51 2025 read: IOPS=335, BW=1340KiB/s (1373kB/s)(78.6MiB/60011msec) slat (usec): min=160, max=361240, avg=11930.20, stdev=15819.52 clat (usec): min=4, max=957176, avg=178747.09, stdev=89162.82 lat (msec): min=5, max=977, avg=190.68, stdev=93.50 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 101], 10.00th=[ 109], 20.00th=[ 121], | 30.00th=[ 130], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 163], | 70.00th=[ 180], 80.00th=[ 218], 90.00th=[ 300], 95.00th=[ 372], | 99.00th=[ 514], 99.50th=[ 558], 99.90th=[ 726], 99.95th=[ 802], | 99.99th=[ 885] bw ( KiB/s): min= 248, max= 2160, per=99.89%, avg=1339.50, stdev=117.47, samples=476 iops : min= 62, max= 540, avg=334.87, stdev=29.37, samples=476 lat (usec) : 10=0.02% lat (msec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=4.87%, 250=79.56% lat (msec) : 500=14.36%, 750=1.08%, 1000=0.08% cpu : usr=0.06%, sys=0.27%, ctx=20110, majf=0, minf=110 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=20110,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=1340KiB/s (1373kB/s), 1340KiB/s-1340KiB/s (1373kB/s-1373kB/s), io=78.6MiB (82.4MB), run=60011-60011msec
Disk stats (read/write): sdd: ios=20096/0, sectors=160768/0, merge=0/0, ticks=238953/0, in_queue=238952, util=99.89%
HDD with cache=writeback # echo 3 > /proc/sys/vm/drop_caches ; fio --name=registry-read --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --size=10G --runtime=60 --group_reporting --iodepth=16 registry-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.36 Starting 4 processes Jobs: 4 (f=4): [r(4)][100.0%][r=210MiB/s][r=53.7k IOPS][eta 00m:00s] registry-read: (groupid=0, jobs=4): err= 0: pid=943730: Thu Mar 20 14:51:33 2025 read: IOPS=53.8k, BW=210MiB/s (220MB/s)(12.3GiB/60001msec) slat (usec): min=26, max=4995, avg=71.31, stdev=21.64 clat (usec): min=3, max=8707, avg=1116.26, stdev=141.55 lat (usec): min=79, max=8769, avg=1187.56, stdev=148.56 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 979], 10.00th=[ 1004], 20.00th=[ 1029], | 30.00th=[ 1045], 40.00th=[ 1074], 50.00th=[ 1090], 60.00th=[ 1106], | 70.00th=[ 1139], 80.00th=[ 1188], 90.00th=[ 1254], 95.00th=[ 1336], | 99.00th=[ 1582], 99.50th=[ 1811], 99.90th=[ 2474], 99.95th=[ 2802], | 99.99th=[ 3982] bw ( KiB/s): min=167800, max=230352, per=100.00%, avg=215502.18, stdev=2114.67, samples=476 iops : min=41950, max=57588, avg=53875.55, stdev=528.67, samples=476 lat (usec) : 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=10.02% lat (msec) : 2=89.68%, 4=0.28%, 10=0.01% cpu : usr=4.83%, sys=37.00%, ctx=3232089, majf=0, minf=101 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=3230027,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=210MiB/s (220MB/s), 210MiB/s-210MiB/s (220MB/s-220MB/s), io=12.3GiB (13.2GB), run=60001-60001msec
Disk stats (read/write): sdd: ios=3224017/2, sectors=25792136/3, merge=0/0, ticks=168114/14, in_queue=168141, util=99.03%
This was hdd (3/2 replication).