Re: 4k IOPS: miserable performance in All-SSD cluster
In my experience, ceph will add around 1ms even if only on localhost. If this is in the client code or on the OSD's, I dont really know. I don't even know the precise reason, but the latency is there nevertheless. Perhaps you can find the reason here among the tradeoffs ceph and similar systems have to make to ensure consistency even if a partition can happen at any time: https://en.wikipedia.org/wiki/PACELC_theorem With size=3, a write will go first to the primary OSD for the PG, (0,1ms), then from there to two more PGs (in parallell), so 0,2ms more total RTT. Then back to the client, 0,1ms. That is, very roughly, 1,4ms even if storage latency is 0 which it never is even for ssds. If you set size=1, you can skip the step where the primary OSD replicates to the 2 replicas, but you still have cephs internal latency as well as the network latency to reach the primary OSD for whatever PG the object will belong to which could be on any server. So expect a small improvement but not too much. With that said, a single thread will not exceed 1000 iops ever in a typical setup. /Peter Den 2024-11-26 kl. 21:09, skrev Martin Gerhard Loschwitz:
that would mean 2-3ms latency between hosts hanging above each other in the same rack connected to the same switches.
Ping shows 0,2ms of latency though for all three affected clusters.
So roughly 5000 iops. We can certainly add Ceph latency to that, but that would mean Ceph eats 99% of the available performance, wouldn't it?
Also, that wouldn't explain why we're seeing a bit of improvement with size=1 for a specific pool but not a massive improvement, given that at least half of the latency is taken out of the equation in that case.
Best regards Martin
Peter Linder <peter.linder@fiberdirekt.se> schrieb am Di. 26. Nov. 2024 um 20:52:
With qd=1 (queue depth?) and a single thread, this isn't totally unreasonable.
Ceph will have an internal latency of around 1ms or so, add some network to that and an operation can take 2-3ms. With a single operation in flight all the time, this means 333-500 operations per second. With hdds, even fewer.
What happens if you try again with many more threads?
Den 2024-11-25 kl. 15:22, skrev Martin Gerhard Loschwitz: > Folks, > > I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. > > In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. > > I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? > > I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! > > Best regards > Martin > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
In my experience, ceph will add around 1ms even if only on localhost. If this is in the client code or on the OSD's, I dont really know. I don't even know the precise reason, but the latency is there nevertheless. Perhaps you can find the reason here among the tradeoffs ceph and similar systems have to make to ensure consistency even if a partition can happen at any time:
https://en.wikipedia.org/wiki/PACELC_theorem
With size=3, a write will go first to the primary OSD for the PG, (0,1ms), then from there to two more PGs (in parallell), so 0,2ms more total RTT. Then back to the client, 0,1ms. That is, very roughly, 1,4ms even if storage latency is 0 which it never is even for ssds.
If you set size=1, you can skip the step where the primary OSD replicates to the 2 replicas, but you still have cephs internal latency as well as the network latency to reach the primary OSD for whatever PG the object will belong to which could be on any server. So expect a small improvement but not too much.
With that said, a single thread will not exceed 1000 iops ever in a typical setup.
Do you have an idea how did this progress over the versions last few years? I thought they were addressing this type of performance issue. I can remember that when moving from direct disk access to using lvm there were also people complianing about added latency.
Not really. I'm assuming that they have been working hard at it and I remember hearing something about a more recent rocksdb version shaving off significant time. It would also depend on your CPU and memory speed. I wouldn't be all surprised if latency is lower today, but I havent really measured it lately. /Peter Den 2024-11-26 kl. 22:03, skrev Marc:
In my experience, ceph will add around 1ms even if only on localhost. If this is in the client code or on the OSD's, I dont really know. I don't even know the precise reason, but the latency is there nevertheless. Perhaps you can find the reason here among the tradeoffs ceph and similar systems have to make to ensure consistency even if a partition can happen at any time:
https://en.wikipedia.org/wiki/PACELC_theorem
With size=3, a write will go first to the primary OSD for the PG, (0,1ms), then from there to two more PGs (in parallell), so 0,2ms more total RTT. Then back to the client, 0,1ms. That is, very roughly, 1,4ms even if storage latency is 0 which it never is even for ssds.
If you set size=1, you can skip the step where the primary OSD replicates to the 2 replicas, but you still have cephs internal latency as well as the network latency to reach the primary OSD for whatever PG the object will belong to which could be on any server. So expect a small improvement but not too much.
With that said, a single thread will not exceed 1000 iops ever in a typical setup.
Do you have an idea how did this progress over the versions last few years? I thought they were addressing this type of performance issue. I can remember that when moving from direct disk access to using lvm there were also people complianing about added latency.
Here’s a benchmark of another setup I did a few months back, with NVME flash drives and a Mellanox EVPN fabric (Spectrum ASIC) between the nodes (no RDMA). 3 hosts and 24 drives in total. root@test01:~# fio --ioengine=libaio --filename=/dev/sdb --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=6966KiB/s][w=1741 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=115698: Tue May 28 16:54:38 2024 write: IOPS=1804, BW=7218KiB/s (7391kB/s)(423MiB/60001msec); 0 zone resets slat (nsec): min=2872, max=92926, avg=5026.65, stdev=2710.03 clat (usec): min=419, max=4486, avg=548.34, stdev=54.66 lat (usec): min=461, max=4490, avg=553.37, stdev=55.02 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 502], 10.00th=[ 510], 20.00th=[ 523], | 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 545], 60.00th=[ 553], | 70.00th=[ 562], 80.00th=[ 570], 90.00th=[ 586], 95.00th=[ 594], | 99.00th=[ 660], 99.50th=[ 758], 99.90th=[ 1156], 99.95th=[ 1287], | 99.99th=[ 2606] bw ( KiB/s): min= 6664, max= 8072, per=100.00%, avg=7225.95, stdev=268.19, samples=119 iops : min= 1666, max= 2018, avg=1806.49, stdev=67.05, samples=119 lat (usec) : 500=4.95%, 750=94.52%, 1000=0.38% lat (msec) : 2=0.13%, 4=0.02%, 10=0.01% cpu : usr=0.57%, sys=1.46%, ctx=108317, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,108275,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=7218KiB/s (7391kB/s), 7218KiB/s-7218KiB/s (7391kB/s-7391kB/s), io=423MiB (443MB), run=60001-60001msec Disk stats (read/write): sdb: ios=80/108093, merge=0/0, ticks=21/59172, in_queue=59193, util=99.96% This was in an instance inside VMware, so there was iSCSI involved in the data path in addition to the normal Ceph replication, with Ceph being mostly out-of-the-box and standard. I wouldn’t believe 40 (or 400 in the SSD cluster) would be a bad value had I not seen substantially better values in the past. And even the 1000 would be a very substantial improvement compared to what I see now. Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH P +49 2433 5253130 <tel:+49 2433 5253130> M +49 176 61832178 <https://mysig.io/4ngY23j0> A Schmiedegasse 24a, 41836 Hückelhoven, Deutschland R HRB 21985, Amtsgericht Mönchengladbach <https://mysig.io/b4g0y3rz> <https://mysignature.io/editor?utm_source=expiredpixel> True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.
Hi Martin, I think what Peter suggests is that you should try with --numjobs=128 and --iodepth=16 to see what your hardware is really capable of with this very small I/O workload. Regards, Frédéric. ________________________________ De : Martin Gerhard Loschwitz <martin.loschwitz@true-west.com> Envoyé : mardi 26 novembre 2024 22:08 À : Peter Linder Cc: ceph-users@ceph.io Objet : [ceph-users] Re: 4k IOPS: miserable performance in All-SSD cluster Here’s a benchmark of another setup I did a few months back, with NVME flash drives and a Mellanox EVPN fabric (Spectrum ASIC) between the nodes (no RDMA). 3 hosts and 24 drives in total. root@test01:~# fio --ioengine=libaio --filename=/dev/sdb --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=6966KiB/s][w=1741 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=115698: Tue May 28 16:54:38 2024 write: IOPS=1804, BW=7218KiB/s (7391kB/s)(423MiB/60001msec); 0 zone resets slat (nsec): min=2872, max=92926, avg=5026.65, stdev=2710.03 clat (usec): min=419, max=4486, avg=548.34, stdev=54.66 lat (usec): min=461, max=4490, avg=553.37, stdev=55.02 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 502], 10.00th=[ 510], 20.00th=[ 523], | 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 545], 60.00th=[ 553], | 70.00th=[ 562], 80.00th=[ 570], 90.00th=[ 586], 95.00th=[ 594], | 99.00th=[ 660], 99.50th=[ 758], 99.90th=[ 1156], 99.95th=[ 1287], | 99.99th=[ 2606] bw ( KiB/s): min= 6664, max= 8072, per=100.00%, avg=7225.95, stdev=268.19, samples=119 iops : min= 1666, max= 2018, avg=1806.49, stdev=67.05, samples=119 lat (usec) : 500=4.95%, 750=94.52%, 1000=0.38% lat (msec) : 2=0.13%, 4=0.02%, 10=0.01% cpu : usr=0.57%, sys=1.46%, ctx=108317, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,108275,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=7218KiB/s (7391kB/s), 7218KiB/s-7218KiB/s (7391kB/s-7391kB/s), io=423MiB (443MB), run=60001-60001msec Disk stats (read/write): sdb: ios=80/108093, merge=0/0, ticks=21/59172, in_queue=59193, util=99.96% This was in an instance inside VMware, so there was iSCSI involved in the data path in addition to the normal Ceph replication, with Ceph being mostly out-of-the-box and standard. I wouldn’t believe 40 (or 400 in the SSD cluster) would be a bad value had I not seen substantially better values in the past. And even the 1000 would be a very substantial improvement compared to what I see now. Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH P +49 2433 5253130 <tel:+49 2433 5253130> M +49 176 61832178 <https://mysig.io/4ngY23j0> A Schmiedegasse 24a, 41836 Hückelhoven, Deutschland R HRB 21985, Amtsgericht Mönchengladbach <https://mysig.io/b4g0y3rz> <https://mysignature.io/editor?utm_source=expiredpixel> True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That is indeed a lot nicer hardware and 1804 iops is faster, but still lower than a usd thumb drive. The thing with ceph is that is scales out really really well, but scaling up is harder. That is, if you run like 500 of these tests at the same time, then you can see what it can do. Some guy did 1TB per second: https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/ Den 2024-11-26 kl. 22:06, skrev Martin Gerhard Loschwitz:
Here’s a benchmark of another setup I did a few months back, with NVME flash drives and a Mellanox EVPN fabric (Spectrum ASIC) between the nodes (no RDMA). 3 hosts and 24 drives in total.
root@test01:~# fio --ioengine=libaio --filename=/dev/sdb --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=6966KiB/s][w=1741 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=115698: Tue May 28 16:54:38 2024 write: IOPS=1804, BW=7218KiB/s (7391kB/s)(423MiB/60001msec); 0 zone resets slat (nsec): min=2872, max=92926, avg=5026.65, stdev=2710.03 clat (usec): min=419, max=4486, avg=548.34, stdev=54.66 lat (usec): min=461, max=4490, avg=553.37, stdev=55.02 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 502], 10.00th=[ 510], 20.00th=[ 523], | 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 545], 60.00th=[ 553], | 70.00th=[ 562], 80.00th=[ 570], 90.00th=[ 586], 95.00th=[ 594], | 99.00th=[ 660], 99.50th=[ 758], 99.90th=[ 1156], 99.95th=[ 1287], | 99.99th=[ 2606] bw ( KiB/s): min= 6664, max= 8072, per=100.00%, avg=7225.95, stdev=268.19, samples=119 iops : min= 1666, max= 2018, avg=1806.49, stdev=67.05, samples=119 lat (usec) : 500=4.95%, 750=94.52%, 1000=0.38% lat (msec) : 2=0.13%, 4=0.02%, 10=0.01% cpu : usr=0.57%, sys=1.46%, ctx=108317, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,108275,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): WRITE: bw=7218KiB/s (7391kB/s), 7218KiB/s-7218KiB/s (7391kB/s-7391kB/s), io=423MiB (443MB), run=60001-60001msec
Disk stats (read/write): sdb: ios=80/108093, merge=0/0, ticks=21/59172, in_queue=59193, util=99.96%
This was in an instance inside VMware, so there was iSCSI involved in the data path in addition to the normal Ceph replication, with Ceph being mostly out-of-the-box and standard.
I wouldn’t believe 40 (or 400 in the SSD cluster) would be a bad value had I not seen substantially better values in the past. And even the 1000 would be a very substantial improvement compared to what I see now.
Best regards Martin
-- True West IT Services GmbH Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH P +49 2433 5253130 <tel:+49 2433 5253130> M +49 176 61832178 <https://mysig.io/4ngY23j0> ASchmiedegasse 24a, 41836 Hückelhoven, Deutschland R HRB 21985, Amtsgericht Mönchengladbach <https://mysig.io/b4g0y3rz>
<https://mysignature.io/editor?utm_source=expiredpixel>
True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.
participants (4)
-
Frédéric Nass
-
Marc
-
Martin Gerhard Loschwitz
-
Peter Linder