Thanks a lot! That was the part i did wrong :) Cheers, Mario On Mon, Dec 9, 2024 at 5:38 PM <darren@soothill.com> wrote:
Hi Mario,
You are not testing like for like.
Either increase the numjobs on the FIO command to 16 or reduce the io-threads on the rbd bench command to 1 or add on iodepth=16 to the FIO command.
Currently with the RBD bench you have 16 concurrent IO’s running at the same time but you are restricting FIO to 1.
Darren
On 9 Dec 2024, at 15:31, Ml Ml <mliebherr99@googlemail.com> wrote:
Hi all,
I have a question regarding the performance differences between rbd bench and a fio test on a Ceph RBD. During testing of my Ceph environment, I noticed that rbd bench reports significantly higher write speeds compared to fio, which is run on an XFS filesystem mounted on the same RBD.
Here are the test details:
fio test: ------------ fio --name=ceph_bench --rw=write --bs=4M --direct=1 --ioengine=libaio --size=4G --numjobs=1 --filename=/mnt/ceph10/backup-cluster1/testfile
Results: Write speed: ~70.3 MiB/s (73.7 MB/s) IOPS: ~17 Latency: ~56.4 ms (average)
rbd bench: -------------- rbd bench --io-type write --io-size 4M --io-threads 16 --io-total 12G backup-proxmox/cluster1-new --keyring /etc/ceph/ceph10/ceph.client.admin.keyring --conf /etc/ceph/ceph10/ceph.conf
Results: Write speed: ~150–180 MiB/s (sustained after initial peak) IOPS: ~43–46 Threads: 16
ceph tell osd.X bench: --------------------------- osd.0: IOPS: 20.03, Throughput: 80.11 MB/s osd.1: IOPS: 23.08, Throughput: 92.33 MB/s osd.2: IOPS: 22.01, Throughput: 88.02 MB/s osd.3: IOPS: 17.65, Throughput: 70.61 MB/s osd.4: IOPS: 20.15, Throughput: 80.59 MB/s osd.5: IOPS: 19.71, Throughput: 78.82 MB/s osd.6: IOPS: 18.96, Throughput: 75.84 MB/s osd.7: IOPS: 20.13, Throughput: 80.52 MB/s osd.8: IOPS: 16.45, Throughput: 65.79 MB/s (hm) osd.9: IOPS: 31.16, Throughput: 124.65 MB/s osd.10: IOPS: 23.19, Throughput: 92.76 MB/s osd.12: IOPS: 19.10, Throughput: 76.38 MB/s osd.13: IOPS: 28.02, Throughput: 112.09 MB/s osd.14: IOPS: 19.07, Throughput: 76.27 MB/s osd.15: IOPS: 21.40, Throughput: 85.59 MB/s osd.16: IOPS: 20.24, Throughput: 80.95 MB/s
I understand that rbd bench is designed to test the raw performance of the RBD layer, while fio runs at the filesystem level. However, the large discrepancy between the two results has left me curious. What could be causing this difference? Are there any specific factors related to caching, journaling, or XFS itself that could explain the significantly lower performance in the fio test?
Any insights or recommendations for further troubleshooting would be greatly appreciated!
Thanks in advance! Best regards, Mario _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io