Ceph RBD, MySQL write IOPs - what is possible?
I've been using MySQL on Ceph forever, and have been down this road before but it's been a couple of years so I wanted to see if there is anything new here. So the TL:DR version of this email - is there a good way to improve 16K write IOPs with a small number of threads? The OSDs themselves are idle so is this just a weakness in the algorithms or do ceph clients need some profiling? Or "other"? Basically, this is one of the worst possible Ceph workloads so it is fun to try to push the limits. I also happen have a MySQL instance that is reaching the write IOPs limit so this is also a last-ditch effort to keep it on Ceph. This cluster is as straightforward as it gets... 6 servers with 10 SSDs each, 100 Gb networking. I'm using size=3. During operations, the OSDs are more or less idle so I don't suspect any hardware limitations. MySQL has no parallelism so the number of threads and effective queue depth stay pretty low. Therefore, as a proxy for MySQL I use rados bench with 16K writes and 8 threads. The RBD actually gets about 2x this level - still not so great. I get about 2000 IOPs with this test: # rados bench -p volumes 10 write -t 8 -b 16K hints = 1 Maintaining 8 concurrent writes of 16384 bytes to objects of size 16384 for up to 10 seconds or 0 objects Object prefix: benchmark_data_fstosinfra-5_3652583 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 8 2050 2042 31.9004 31.9062 0.00247633 0.00390848 2 8 4306 4298 33.5728 35.25 0.00278488 0.00371784 3 8 6607 6599 34.3645 35.9531 0.00277546 0.00363139 4 7 8951 8944 34.9323 36.6406 0.00414908 0.00357249 5 8 11292 11284 35.257 36.5625 0.00291434 0.00353997 6 8 13588 13580 35.3588 35.875 0.00306094 0.00353084 7 7 15933 15926 35.5432 36.6562 0.00308388 0.0035123 8 8 18361 18353 35.8399 37.9219 0.00314996 0.00348327 9 8 20629 20621 35.7947 35.4375 0.00352998 0.0034877 10 5 23010 23005 35.9397 37.25 0.00395566 0.00347376 Total time run: 10.003 Total writes made: 23010 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 35.9423 Stddev Bandwidth: 1.63433 Max bandwidth (MB/sec): 37.9219 Min bandwidth (MB/sec): 31.9062 Average IOPS: 2300 Stddev IOPS: 104.597 Max IOPS: 2427 Min IOPS: 2042 Average Latency(s): 0.0034737 Stddev Latency(s): 0.00163661 Max latency(s): 0.115932 Min latency(s): 0.00179735 Cleaning up (deleting benchmark objects) Removed 23010 objects Clean up completed and total clean up time :7.44664 Are there any good options to improve this? It seems like the client side is the bottleneck since the OSD servers are at like 15% utilization. Thanks, Mark
Please describe: * server RAM and CPU * osd_memory_target * OSD drive model
On Jun 7, 2024, at 11:32, Mark Lehrer <lehrer@gmail.com> wrote:
I've been using MySQL on Ceph forever, and have been down this road before but it's been a couple of years so I wanted to see if there is anything new here.
So the TL:DR version of this email - is there a good way to improve 16K write IOPs with a small number of threads? The OSDs themselves are idle so is this just a weakness in the algorithms or do ceph clients need some profiling? Or "other"?
Basically, this is one of the worst possible Ceph workloads so it is fun to try to push the limits. I also happen have a MySQL instance that is reaching the write IOPs limit so this is also a last-ditch effort to keep it on Ceph.
This cluster is as straightforward as it gets... 6 servers with 10 SSDs each, 100 Gb networking. I'm using size=3. During operations, the OSDs are more or less idle so I don't suspect any hardware limitations.
MySQL has no parallelism so the number of threads and effective queue depth stay pretty low. Therefore, as a proxy for MySQL I use rados bench with 16K writes and 8 threads. The RBD actually gets about 2x this level - still not so great.
I get about 2000 IOPs with this test:
# rados bench -p volumes 10 write -t 8 -b 16K hints = 1 Maintaining 8 concurrent writes of 16384 bytes to objects of size 16384 for up to 10 seconds or 0 objects Object prefix: benchmark_data_fstosinfra-5_3652583 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 8 2050 2042 31.9004 31.9062 0.00247633 0.00390848 2 8 4306 4298 33.5728 35.25 0.00278488 0.00371784 3 8 6607 6599 34.3645 35.9531 0.00277546 0.00363139 4 7 8951 8944 34.9323 36.6406 0.00414908 0.00357249 5 8 11292 11284 35.257 36.5625 0.00291434 0.00353997 6 8 13588 13580 35.3588 35.875 0.00306094 0.00353084 7 7 15933 15926 35.5432 36.6562 0.00308388 0.0035123 8 8 18361 18353 35.8399 37.9219 0.00314996 0.00348327 9 8 20629 20621 35.7947 35.4375 0.00352998 0.0034877 10 5 23010 23005 35.9397 37.25 0.00395566 0.00347376 Total time run: 10.003 Total writes made: 23010 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 35.9423 Stddev Bandwidth: 1.63433 Max bandwidth (MB/sec): 37.9219 Min bandwidth (MB/sec): 31.9062 Average IOPS: 2300 Stddev IOPS: 104.597 Max IOPS: 2427 Min IOPS: 2042 Average Latency(s): 0.0034737 Stddev Latency(s): 0.00163661 Max latency(s): 0.115932 Min latency(s): 0.00179735 Cleaning up (deleting benchmark objects) Removed 23010 objects Clean up completed and total clean up time :7.44664
Are there any good options to improve this? It seems like the client side is the bottleneck since the OSD servers are at like 15% utilization.
Thanks, Mark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
server RAM and CPU * osd_memory_target * OSD drive model
Thanks for the reply. The servers have dual Xeon Gold 6154 CPUs with 384 GB. The drives are older, first gen NVMe - WDC SN620. osd_memory_target is at the default. Mellanox CX5 and SN2700 hardware. The test client is a similar machine with no drives. The CPUs are 80% idle during the test. The OSDs (according to iostat) hover around 50% util during the test and are close to 0 at other times. I did find it interesting that the wareq-sz option in iostat is around 5 during the test - I was expecting 16. Is there a way to tweak this in bluestore? These drives are terrible at under 8K I/O. Not that it really matters since we're not I/O bound at all. I can also increase threads from 8 to 32 and the iops are roughly quadruple so that's good at least. Single thread writes are about 250 iops and like 3.7MB/sec. So sad. The rados bench process is also under 50% CPU utilization of a single core. This seems like a thead/semaphore kind of issue if I had to guess. It's tricky to debug when there is no obvious bottleneck. Thanks, Mark On Fri, Jun 7, 2024 at 9:47 AM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Please describe:
* server RAM and CPU * osd_memory_target * OSD drive model
On Jun 7, 2024, at 11:32, Mark Lehrer <lehrer@gmail.com> wrote:
I've been using MySQL on Ceph forever, and have been down this road before but it's been a couple of years so I wanted to see if there is anything new here.
So the TL:DR version of this email - is there a good way to improve 16K write IOPs with a small number of threads? The OSDs themselves are idle so is this just a weakness in the algorithms or do ceph clients need some profiling? Or "other"?
Basically, this is one of the worst possible Ceph workloads so it is fun to try to push the limits. I also happen have a MySQL instance that is reaching the write IOPs limit so this is also a last-ditch effort to keep it on Ceph.
This cluster is as straightforward as it gets... 6 servers with 10 SSDs each, 100 Gb networking. I'm using size=3. During operations, the OSDs are more or less idle so I don't suspect any hardware limitations.
MySQL has no parallelism so the number of threads and effective queue depth stay pretty low. Therefore, as a proxy for MySQL I use rados bench with 16K writes and 8 threads. The RBD actually gets about 2x this level - still not so great.
I get about 2000 IOPs with this test:
# rados bench -p volumes 10 write -t 8 -b 16K hints = 1 Maintaining 8 concurrent writes of 16384 bytes to objects of size 16384 for up to 10 seconds or 0 objects Object prefix: benchmark_data_fstosinfra-5_3652583 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 8 2050 2042 31.9004 31.9062 0.00247633 0.00390848 2 8 4306 4298 33.5728 35.25 0.00278488 0.00371784 3 8 6607 6599 34.3645 35.9531 0.00277546 0.00363139 4 7 8951 8944 34.9323 36.6406 0.00414908 0.00357249 5 8 11292 11284 35.257 36.5625 0.00291434 0.00353997 6 8 13588 13580 35.3588 35.875 0.00306094 0.00353084 7 7 15933 15926 35.5432 36.6562 0.00308388 0.0035123 8 8 18361 18353 35.8399 37.9219 0.00314996 0.00348327 9 8 20629 20621 35.7947 35.4375 0.00352998 0.0034877 10 5 23010 23005 35.9397 37.25 0.00395566 0.00347376 Total time run: 10.003 Total writes made: 23010 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 35.9423 Stddev Bandwidth: 1.63433 Max bandwidth (MB/sec): 37.9219 Min bandwidth (MB/sec): 31.9062 Average IOPS: 2300 Stddev IOPS: 104.597 Max IOPS: 2427 Min IOPS: 2042 Average Latency(s): 0.0034737 Stddev Latency(s): 0.00163661 Max latency(s): 0.115932 Min latency(s): 0.00179735 Cleaning up (deleting benchmark objects) Removed 23010 objects Clean up completed and total clean up time :7.44664
Are there any good options to improve this? It seems like the client side is the bottleneck since the OSD servers are at like 15% utilization.
Thanks, Mark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Jun 7, 2024, at 13:20, Mark Lehrer <lehrer@gmail.com> wrote:
server RAM and CPU * osd_memory_target * OSD drive model
Thanks for the reply. The servers have dual Xeon Gold 6154 CPUs with 384 GB
So roughly 7 vcores / HTs per OSD? Your Ceph is a recent release?
The drives are older, first gen NVMe - WDC SN620.
Those appear to be a former SanDisk product, lower performers than more recent drives, how much a factor that is I can't say. Which specific SKU? There appear to be low and standard endurance SKUs, 3.84 or 1.92 T, 3.2T or 1.6T respectively. What is the lifetime used like on them? Less than 80%? If you really want to eliminate uncertainties: * Ensure they're updated to the latest firmware * In rolling fashion, destroy the OSDs, secure-erase each OSD, redeploy the OSDs
osd_memory_target is at the default. Mellanox CX5 and SN2700 hardware. The test client is a similar machine with no drives.
This is via RBD? Do you have the client RBD cache on or off?
The CPUs are 80% idle during the test.
Do you have the server BMC/BIOS profile set to performance? Deep C-states disabled via TuneD or other means?
The OSDs (according to iostat)
Careful, iostat's metrics are of limited utility on SSDs, especially NVMe.
I did find it interesting that the wareq-sz option in iostat is around 5 during the test - I was expecting 16. Is there a way to tweak this in bluestore?
Not my area of expertise, but I once tried to make OSDs with a >4KB BlueStore block size, they crashed at startup. 4096 is hardcoded in various places. Quality SSD firmware will coalesce writes to NAND. If your firmware surfaces host vs NAND writes, you might capture deltas over, say, a week of workload and calculate the WAF.
These drives are terrible at under 8K I/O. Not that it really matters since we're not I/O bound at all.
I/O bound can be tricky, be careful with that assumption, there are multiple facets. I can't find anything specific, but that makes me suspect that internally the IU isn't the usual 4KB, perhaps to save a few bucks on DRAM.
I can also increase threads from 8 to 32 and the iops are roughly quadruple so that's good at least. Single thread writes are about 250 iops and like 3.7MB/sec. So sad.
Assuming that the pool you're writing to spans all 60 OSDs, what is your PG count on that pool? Are there multiple pools in the cluster? As reported by `ceph osd df`, on average how many PG replicas are on each OSD?
The rados bench process is also under 50% CPU utilization of a single core. This seems like a thead/semaphore kind of issue if I had to guess. It's tricky to debug when there is no obvious bottleneck.
rados bench is a good smoke test, but fio may better represent the E2E experience.
Thanks, Mark
On Fri, Jun 7, 2024 at 9:47 AM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Please describe:
* server RAM and CPU * osd_memory_target * OSD drive model
On Jun 7, 2024, at 11:32, Mark Lehrer <lehrer@gmail.com> wrote:
I've been using MySQL on Ceph forever, and have been down this road before but it's been a couple of years so I wanted to see if there is anything new here.
So the TL:DR version of this email - is there a good way to improve 16K write IOPs with a small number of threads? The OSDs themselves are idle so is this just a weakness in the algorithms or do ceph clients need some profiling? Or "other"?
Basically, this is one of the worst possible Ceph workloads so it is fun to try to push the limits. I also happen have a MySQL instance that is reaching the write IOPs limit so this is also a last-ditch effort to keep it on Ceph.
This cluster is as straightforward as it gets... 6 servers with 10 SSDs each, 100 Gb networking. I'm using size=3. During operations, the OSDs are more or less idle so I don't suspect any hardware limitations.
MySQL has no parallelism so the number of threads and effective queue depth stay pretty low. Therefore, as a proxy for MySQL I use rados bench with 16K writes and 8 threads. The RBD actually gets about 2x this level - still not so great.
I get about 2000 IOPs with this test:
# rados bench -p volumes 10 write -t 8 -b 16K hints = 1 Maintaining 8 concurrent writes of 16384 bytes to objects of size 16384 for up to 10 seconds or 0 objects Object prefix: benchmark_data_fstosinfra-5_3652583 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 8 2050 2042 31.9004 31.9062 0.00247633 0.00390848 2 8 4306 4298 33.5728 35.25 0.00278488 0.00371784 3 8 6607 6599 34.3645 35.9531 0.00277546 0.00363139 4 7 8951 8944 34.9323 36.6406 0.00414908 0.00357249 5 8 11292 11284 35.257 36.5625 0.00291434 0.00353997 6 8 13588 13580 35.3588 35.875 0.00306094 0.00353084 7 7 15933 15926 35.5432 36.6562 0.00308388 0.0035123 8 8 18361 18353 35.8399 37.9219 0.00314996 0.00348327 9 8 20629 20621 35.7947 35.4375 0.00352998 0.0034877 10 5 23010 23005 35.9397 37.25 0.00395566 0.00347376 Total time run: 10.003 Total writes made: 23010 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 35.9423 Stddev Bandwidth: 1.63433 Max bandwidth (MB/sec): 37.9219 Min bandwidth (MB/sec): 31.9062 Average IOPS: 2300 Stddev IOPS: 104.597 Max IOPS: 2427 Min IOPS: 2042 Average Latency(s): 0.0034737 Stddev Latency(s): 0.00163661 Max latency(s): 0.115932 Min latency(s): 0.00179735 Cleaning up (deleting benchmark objects) Removed 23010 objects Clean up completed and total clean up time :7.44664
Are there any good options to improve this? It seems like the client side is the bottleneck since the OSD servers are at like 15% utilization.
Thanks, Mark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Jun 7, 2024 at 1:22 PM Mark Lehrer <lehrer@gmail.com> wrote:
server RAM and CPU * osd_memory_target * OSD drive model
Thanks for the reply. The servers have dual Xeon Gold 6154 CPUs with 384 GB. The drives are older, first gen NVMe - WDC SN620. osd_memory_target is at the default. Mellanox CX5 and SN2700 hardware. The test client is a similar machine with no drives.
The CPUs are 80% idle during the test. The OSDs (according to iostat) hover around 50% util during the test and are close to 0 at other times.
...
I get about 2000 IOPs with this test:
# rados bench -p volumes 10 write -t 8 -b 16K hints = 1 Maintaining 8 concurrent writes of 16384 bytes to objects of size 16384 for up to 10 seconds or 0 objects Object prefix: benchmark_data_fstosinfra-5_3652583 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 8 2050 2042 31.9004 31.9062 0.00247633 0.00390848 2 8 4306 4298 33.5728 35.25 0.00278488 0.00371784 3 8 6607 6599 34.3645 35.9531 0.00277546 0.00363139 4 7 8951 8944 34.9323 36.6406 0.00414908 0.00357249 5 8 11292 11284 35.257 36.5625 0.00291434 0.00353997 6 8 13588 13580 35.3588 35.875 0.00306094 0.00353084 7 7 15933 15926 35.5432 36.6562 0.00308388 0.0035123 8 8 18361 18353 35.8399 37.9219 0.00314996 0.00348327 9 8 20629 20621 35.7947 35.4375 0.00352998 0.0034877 10 5 23010 23005 35.9397 37.25 0.00395566 0.00347376 Total time run: 10.003 Total writes made: 23010 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 35.9423 Stddev Bandwidth: 1.63433 Max bandwidth (MB/sec): 37.9219 Min bandwidth (MB/sec): 31.9062 Average IOPS: 2300 Stddev IOPS: 104.597 Max IOPS: 2427 Min IOPS: 2042 Average Latency(s): 0.0034737 Stddev Latency(s): 0.00163661 Max latency(s): 0.115932 Min latency(s): 0.00179735 Cleaning up (deleting benchmark objects) Removed 23010 objects Clean up completed and total clean up time :7.44664
Not the most helpful response, but on a (admittedly well-tuned) cluster of 3x Intel Atom (C3758) nodes and 10Gbe networking and 2x S4510 (SATA) SSD per node, I get this with the same rados bench run: Total time run: 10.0015 Total writes made: 35931 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 56.1335 Stddev Bandwidth: 0.883058 Max bandwidth (MB/sec): 58.0625 Min bandwidth (MB/sec): 55.0469 Average IOPS: 3592 Stddev IOPS: 56.5157 Max IOPS: 3716 Min IOPS: 3523 Average Latency(s): 0.00222274 Stddev Latency(s): 0.000894184 Max latency(s): 0.016538 Min latency(s): 0.00117819 Given that you have Xeon class processors, it would seem something is very, very wrong here with your configuration. Anthony is asking a lot of the right questions, below and I would recommend following up with all of them. Cheers, Tyler
...
I get about 2000 IOPs with this test:
# rados bench -p volumes 10 write -t 8 -b 16K hints = 1 Maintaining 8 concurrent writes of 16384 bytes to objects of size 16384 for up to 10 seconds or 0 objects Object prefix: benchmark_data_fstosinfra-5_3652583 sec Cur ops started finished avg MB/s cur MB/s last lat(s)
I read quite some time ago here that it is possible to do some local caching for an rbd. I think this would increase the iops considerably. But you have to search for this can't really remember how it is called / done.
Not the most helpful response, but on a (admittedly well-tuned)
Actually this was the most helpful since you ran the same rados bench command. I'm trying to stay away from rbd & qemu issues and just test rados bench on a non-virtualized client. I have a test instance newer drives, CPUs, and Ceph code, I'll see what that looks like. Maged's comments were quite useful as far as iops per thread. It seems like Ceph still hasn't adjusted to SSD performance. This kind of feels like MongoDB before the Wired Tiger engine... slow performance but with all the system resources close to idle due to threads being blocked. Thanks, Mark
If they can do 1 TB/s with a single 16K write thread, that will be quite impressive :D Otherwise not really applicable. Ceph scaling has always been good. More seriously, would you mind sending a link to this? Thanks! Mark On Mon, Jun 10, 2024 at 12:01 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Eh? cf. Mark and Dan's 1TB/s presentation.
On Jun 10, 2024, at 13:58, Mark Lehrer <lehrer@gmail.com> wrote:
It seems like Ceph still hasn't adjusted to SSD performance.
Hi, don’t expect solution on group, just direction. Here is link to the blog post https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/ on youtube is presentation from nyc ceph days View performance from the client's perspective, run the measurement tools from inside the virtual machine. This approach will provide you the performance as experienced by the client. The most commonly used tool for performance measurement is fio. I strongly recommend using fio for your evaluation. also use ioping to measure latency. While fio will provide IOPS/ and latency metrics during load, ioping offers view of latency behavior when the machine is not under heavy load. Based on my previous experiences (not only mine, but also my team), many performance issues were related to network configurations or problems around the network infrastructure. As example we encountered a situation where a change made by the network team to the spine switches caused disk latency to increase from 3ms to 80-120ms. Other example which almost burn me was issue with one spine cards, which was not fully broken, monitoring not discovered it, tests shows everything is ok but on ceph we had many, many issues like flapping osd’s, like half of osds form 500 goes down, latency spikes time to time. Card had problems time to time but not during tests :) and of course AMD nodes before I discovered iommu=pt for kernel params. Belive me this c-states and power management on nodes are important. You already received very good advices from others, not much to add, look on your network drivers, rx queue, tx queue. for your information this cluster was not fine tuned, also e2e enc. is enabled 6 node cluster, all nvme 8x nvme per node, 512gb ram, 4x25GB lacp for public and another 4x25GB for cluster net. (malleanox cards) # rados bench -p test 10 write -t 8 -b 16K Rados bench results: Total time run: 10.0003 Total writes made: 113195 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 176.862 Stddev Bandwidth: 27.047 Max bandwidth (MB/sec): 195.828 Min bandwidth (MB/sec): 107.906 Average IOPS: 11319 Stddev IOPS: 1731.01 Max IOPS: 12533 Min IOPS: 6906 Average Latency(s): 0.000705734 Stddev Latency(s): 0.00224331 Max latency(s): 0.325178 Min latency(s): 0.000413413 This is test from fio with librbd it shows more or less vm performance. [test] ioengine=rbd clientname=admin pool=test rbdname=bench rw=randwrite bs=4k iodepth=256 direct=1 numjobs=1 fsync=0 size=10G runtime=300 time_based invalidate=0 test: (groupid=0, jobs=1): err= 0: pid=3495143: Tue Jun 11 11:56:04 2024 write: IOPS=83.6k, BW=326MiB/s (342MB/s)(95.6GiB/300002msec); 0 zone resets slat (nsec): min=975, max=2665.0k, avg=3943.68, stdev=2820.21 clat (usec): min=399, max=225434, avg=3058.67, stdev=1801.25 and for iodepth=1 test: (groupid=0, jobs=1): err= 0: pid=3503647: Tue Jun 11 11:57:48 2024 write: IOPS=1845, BW=7382KiB/s (7559kB/s)(159MiB/22033msec); 0 zone resets slat (nsec): min=2966, max=41133, avg=4381.81, stdev=1062.40 clat (usec): min=367, max=202364, avg=537.05, stdev=1009.49 and iodepth=256 and bs=16k test: (groupid=0, jobs=1): err= 0: pid=3505339: Tue Jun 11 12:03:27 2024 write: IOPS=79.6k, BW=1244MiB/s (1305MB/s)(365GiB/300002msec); 0 zone resets slat (nsec): min=1815, max=4497.4k, avg=5671.20, stdev=3540.33 clat (usec): min=446, max=267567, avg=3208.34, stdev=2038.58 lat (usec): min=451, max=267571, avg=3214.01, stdev=2038.60 BR, Sebastian
On 11 Jun 2024, at 02:23, Mark Lehrer <lehrer@gmail.com> wrote:
If they can do 1 TB/s with a single 16K write thread, that will be quite impressive :D Otherwise not really applicable. Ceph scaling has always been good.
More seriously, would you mind sending a link to this?
Thanks!
Mark
On Mon, Jun 10, 2024 at 12:01 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Eh? cf. Mark and Dan's 1TB/s presentation.
On Jun 10, 2024, at 13:58, Mark Lehrer <lehrer@gmail.com> wrote:
It seems like Ceph still hasn't adjusted to SSD performance.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
With good hardware and correct configuration, an all flash cluster should give: approx 1-2K write iops per thread (0.5-1 ms latency) approx 2-5K read iops per thread (0.2-0.5 ms latency) This is dependent on quality of drives and cpu/frequency but independent on number of drives or cores. Total iops should scale as you add threads, linearly at first but the rate will slowly saturate based on number of drives and cores. With 8 threads, you should easily get 8k write, 16k read iops or more. With only 8 threads/queue depth, you would not need more than 3 drives in total for performance, any extra drives will be for capacity as they will be idle most of the time. Some things to consider to get decent speeds: 1) Make sure you use enterprise ssd/nvme drives. Ceph syncs its writes, it is worth testing raw drive write speed using fio with direct,sync flags. 10K write iops and more is good. 2) tune your cpu: disable wait states, set min frequency: 100%, set governor to performance. Make sure you disable anything in BIOS relating to energy savings :) 3) disable volatile cache on your nvme 4) set the i/o scheduler on nvme drives to "none" 5) lower read_ahead_kb to 64KB or lower so not to affect random reads. 6) There are advanced tuning like numa pinning, but you should get decent speeds without doing fancy stuff. I would not recommend any drive caching. This could be good for bursty workloads but typically give worse results for consistent heavy loads. Ceph is best for consistent high load, if all you need is a single MySQL db with a few threads that may at times have bursty load but is mostly quiet, then probably Ceph is not the best solution. If however you have this MySQL db in one of your vms among thousands of other vms, then Ceph will be ideal. /Maged On 07/06/2024 18:32, Mark Lehrer wrote:
I've been using MySQL on Ceph forever, and have been down this road before but it's been a couple of years so I wanted to see if there is anything new here.
So the TL:DR version of this email - is there a good way to improve 16K write IOPs with a small number of threads? The OSDs themselves are idle so is this just a weakness in the algorithms or do ceph clients need some profiling? Or "other"?
Basically, this is one of the worst possible Ceph workloads so it is fun to try to push the limits. I also happen have a MySQL instance that is reaching the write IOPs limit so this is also a last-ditch effort to keep it on Ceph.
This cluster is as straightforward as it gets... 6 servers with 10 SSDs each, 100 Gb networking. I'm using size=3. During operations, the OSDs are more or less idle so I don't suspect any hardware limitations.
MySQL has no parallelism so the number of threads and effective queue depth stay pretty low. Therefore, as a proxy for MySQL I use rados bench with 16K writes and 8 threads. The RBD actually gets about 2x this level - still not so great.
I get about 2000 IOPs with this test:
# rados bench -p volumes 10 write -t 8 -b 16K hints = 1 Maintaining 8 concurrent writes of 16384 bytes to objects of size 16384 for up to 10 seconds or 0 objects Object prefix: benchmark_data_fstosinfra-5_3652583 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 8 2050 2042 31.9004 31.9062 0.00247633 0.00390848 2 8 4306 4298 33.5728 35.25 0.00278488 0.00371784 3 8 6607 6599 34.3645 35.9531 0.00277546 0.00363139 4 7 8951 8944 34.9323 36.6406 0.00414908 0.00357249 5 8 11292 11284 35.257 36.5625 0.00291434 0.00353997 6 8 13588 13580 35.3588 35.875 0.00306094 0.00353084 7 7 15933 15926 35.5432 36.6562 0.00308388 0.0035123 8 8 18361 18353 35.8399 37.9219 0.00314996 0.00348327 9 8 20629 20621 35.7947 35.4375 0.00352998 0.0034877 10 5 23010 23005 35.9397 37.25 0.00395566 0.00347376 Total time run: 10.003 Total writes made: 23010 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 35.9423 Stddev Bandwidth: 1.63433 Max bandwidth (MB/sec): 37.9219 Min bandwidth (MB/sec): 31.9062 Average IOPS: 2300 Stddev IOPS: 104.597 Max IOPS: 2427 Min IOPS: 2042 Average Latency(s): 0.0034737 Stddev Latency(s): 0.00163661 Max latency(s): 0.115932 Min latency(s): 0.00179735 Cleaning up (deleting benchmark objects) Removed 23010 objects Clean up completed and total clean up time :7.44664
Are there any good options to improve this? It seems like the client side is the bottleneck since the OSD servers are at like 15% utilization.
Thanks, Mark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
6) There are advanced tuning like numa pinning, but you should get decent speeds without doing fancy stuff.
This is why I’d asked the OP for the CPU in use. Mark and Dan’s recent and superlative presentation about 1TB/s with Ceph underscored how tunings can make a very real difference. On EPYS, for example: * Experiment with NPS = 4, 2, 1, 0 * Disabling IOMMU in the kernel may make a huge difference * Single-socket vs dual-socket systems make a difference. To be clear, that means like a higher-core -P CPU in a chassis designed for single-socket, not a dual socket unit left half empty with half the cores. CPU interconnects can matter, as when chasing critical latency things like how many chiplets comprise the CPU (compare Sapphire Rapids to Emerald Rapids) and how many cores per IO die are present.
participants (6)
-
Anthony D'Atri
-
Maged Mokhtar
-
Marc
-
Mark Lehrer
-
Sebastian
-
Tyler Stachecki