CEPH performance all Flash lower than local
Hello all I tried doing some fio test on local disk(NVME) and ceph rbd. Why ceph is having low IO whereas it’s also on all NVME. What to tune to reach equal amount of IO? root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 6.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 8441.37| 3588.71| |SEQ1M Q1 T1 | 3074.86| 1172.46| |RND4K Q32T16| 723.65| 733.76| |. IOPS | 176671.80| 179141.74| |. latency us| 2892.49| 2839.37| |RND4K Q1 T1 | 71.05| 57.88| |. IOPS | 17347.13| 14131.57| |. latency us| 56.13| 66.40| ### When vm moved to ceph storage... root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 9.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 1681.40| 889.89| |SEQ1M Q1 T1 | 310.74| 852.11| |RND4K Q32T16| 403.04| 274.23| |. IOPS | 98397.32| 66951.49| |. latency us| 5196.98| 7637.10| |RND4K Q1 T1 | 4.69| 45.18| |. IOPS | 1146.17| 11029.39| |. latency us| 869.47| 87.50| Regards Dev
Hi Keep in mind that when you compare a local NVMe with RBD you are adding to the mix the following: - Network connectivity (link speed and bandwidth, switching latency, ...) - All the lines of code that make Ceph (both client and server side) - The ability to process the lines of code above as fast as possible (CPU dependent) So you are comparing two things that are radically different in terms of features and architecture. The Ceph community is working hard to make Ceph always better and you can take a look at the Crimson project <https://ceph.io/en/news/crimson/> that aims at improving the OSD side when it comes to latency and performance for flash based Ceph clusters. Best JC
On Jul 23, 2025, at 15:52, Devender Singh <devender@netskrt.io> wrote:
root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 6.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 8441.37| 3588.71| |SEQ1M Q1 T1 | 3074.86| 1172.46| |RND4K Q32T16| 723.65| 733.76| |. IOPS | 176671.80| 179141.74| |. latency us| 2892.49| 2839.37| |RND4K Q1 T1 | 71.05| 57.88| |. IOPS | 17347.13| 14131.57| |. latency us| 56.13| 66.40|
Hello Thanks for your reply… Just to clear.. currently no load, nothing running on ceph… the fit test was done on a fresh installed vm to have a fit test output for reference before deploying any further vms.. Below is the bench result, yes its replicated pool… # rados bench -p vms-os-r3-images 30 write -t 32 -b 1M hints = 1 Maintaining 32 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 30 seconds or 0 objects Object prefix: benchmark_data_host08n.van2.netskr_995764 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 32 1376 1344 1343.81 1344 0.0125439 0.0235756 2 32 2985 2953 1476.31 1609 0.0153368 0.0215803 3 32 4545 4513 1504.15 1560 0.0176087 0.0211638 4 32 6151 6119 1529.57 1606 0.013467 0.0208516 5 32 7749 7717 1543.22 1598 0.0229487 0.0206937 6 32 9323 9291 1548.32 1574 0.0218796 0.0206146 7 32 10937 10905 1557.67 1614 0.0247086 0.0205051 8 32 12556 12524 1565.3 1619 0.0174251 0.0204056 9 32 14157 14125 1569.24 1601 0.0142258 0.0203625 10 32 15736 15704 1570.2 1579 0.019351 0.0203552 11 32 17286 17254 1568.34 1550 0.0197798 0.0203718 12 32 18715 18683 1556.71 1429 0.0241973 0.0205236 13 32 20255 20223 1555.41 1540 0.0226729 0.0205478 14 32 21849 21817 1558.15 1594 0.0372476 0.0205132 15 32 23458 23426 1561.52 1609 0.0238018 0.0204644 16 32 25078 25046 1565.16 1620 0.0183276 0.0204308 17 32 26676 26644 1567.08 1598 0.0198966 0.0204023 18 32 28286 28254 1569.46 1610 0.0121277 0.0203729 19 32 29864 29832 1569.9 1578 0.0213983 0.0203617 2025-07-24T18:05:45.399110+0000 min lat: 0.00366107 max lat: 0.220254 avg lat: 0.0203759 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 31 31423 31392 1569.39 1560 0.0165359 0.0203759 21 32 33027 32995 1570.98 1603 0.00897043 0.02035 22 32 34607 34575 1571.38 1580 0.0188498 0.0203498 23 32 36195 36163 1572.1 1588 0.00892525 0.0203346 24 32 37782 37750 1572.71 1587 0.012956 0.020331 25 32 39404 39372 1574.67 1622 0.0168824 0.0203063 26 32 41011 40979 1575.91 1607 0.0177068 0.0202896 27 32 42599 42567 1576.27 1588 0.0196947 0.0202878 28 32 44194 44162 1576.93 1595 0.0179365 0.020279 29 32 45752 45720 1576.27 1558 0.0256928 0.0202886 30 18 47359 47341 1577.75 1621 0.0159732 0.0202704 Total time run: 30.0114 Total writes made: 47359 Write size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 1578.03 Stddev Bandwidth: 57.5748 Max bandwidth (MB/sec): 1622 Min bandwidth (MB/sec): 1344 Average IOPS: 1578 Stddev IOPS: 57.5748 Max IOPS: 1622 Min IOPS: 1344 Average Latency(s): 0.0202697 Stddev Latency(s): 0.00834519 Max latency(s): 0.234282 Min latency(s): 0.00366107 Cleaning up (deleting benchmark objects) Removed 47359 objects Clean up completed and total clean up time :2.91541 Regards Dev
On Jul 23, 2025, at 4:40 PM, Jean-Charles Lopez <jelopez@redhat.com> wrote:
it comes to latency and performance for flash based Cep
On Jul 24, 2025, at 2:11 PM, Devender Singh <devender@netskrt.io> wrote:
Just to clear.. currently no load, nothing running on ceph… the fit [fio?] test was done on a fresh installed vm to have a fit test output for reference before deploying any further vms..
VMs often have iops / throughput throttles applied to mitigate noisy neighbors, so it’s good to check if that might be in play.
I tried doing some fio test on local disk(NVME) and ceph rbd. Why ceph is having low IO whereas it’s also on all NVME. What to tune to reach equal amount of IO?
root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 6.3GiB/64.4GiB
You should really make sure the size of the disk object you make tests on are larger than available ram. I don't know what options your fio-cdm script gives, but 1G is super easy for a machine to cache in ram, so that could skew the tests totally. Apart from that, it will ALWAYS be slower to read data across network (even if network to localhost), and if the RBD is on a replicated pool, then of course it will need to wait for acks on the replicas before allowing the client to know the write has been done, which again slows it down more. Finally, the SEQ1Q tests sounds a lot like as if it was like running "dd" against ceph which is basically the worst possible kind of test for a cluster network storage system. Firstly, it is basically something very very few clients need optimizing for, linear dd with no queuing means "write 1 meg, wait for ack, do nothing until ack comes, then write 1 meg more and go back to waiting". This says more or less nothing about a storage cluster which ties in to the second part, what to use a cluster for. Most clusters are used to handle many clients doing cephfs, or many clients each mounting one or more RBD images, or many RGW frontends each serving many parallel client sessions. So even if each other those clients would be doing the linear "dance" of "write, wait, wait, write, wait, wait, write", the cluster in total would obviously have a much larger throughput, because some other client will of course be reading and writing to other disks while first client is waiting for acks. -- May the most significant bit of your life be positive.
fio test on local disk(NVME) and ceph rbd.
I would suggest trying rados bench as well. This will show the basic Ceph object performance level. If you have good rados performance with a 1M object size (which is usually the case in my experience), then you can look at what is happening at the RBD or VM level. Here is an example 30-second write test: rados bench -p volumes 30 write -t 32 -b 1M On Wed, Jul 23, 2025 at 4:53 PM Devender Singh <devender@netskrt.io> wrote:
Hello all
I tried doing some fio test on local disk(NVME) and ceph rbd. Why ceph is having low IO whereas it’s also on all NVME. What to tune to reach equal amount of IO?
root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 6.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 8441.37| 3588.71| |SEQ1M Q1 T1 | 3074.86| 1172.46| |RND4K Q32T16| 723.65| 733.76| |. IOPS | 176671.80| 179141.74| |. latency us| 2892.49| 2839.37| |RND4K Q1 T1 | 71.05| 57.88| |. IOPS | 17347.13| 14131.57| |. latency us| 56.13| 66.40|
### When vm moved to ceph storage... root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 9.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 1681.40| 889.89| |SEQ1M Q1 T1 | 310.74| 852.11| |RND4K Q32T16| 403.04| 274.23| |. IOPS | 98397.32| 66951.49| |. latency us| 5196.98| 7637.10| |RND4K Q1 T1 | 4.69| 45.18| |. IOPS | 1146.17| 11029.39| |. latency us| 869.47| 87.50|
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Devender Singh
-
Janne Johansson
-
Jean-Charles Lopez
-
Mark Lehrer