On 10/09/2020 19:37, Mark Nelson wrote:
On 9/10/20 11:03 AM, George Shuklin wrote:
... Are there any knobs to tweak to see higher performance for ceph-osd? I'm pretty sure it's not any kind of leveling, GC or other 'iops-related' issues (brd has performance of two order of magnitude higher).
So as you've seen, Ceph does a lot more than just write a chunk of data out to a block on disk. There's tons of encoding/decoding happening, crc checksums, crush calculations, onode lookups, write-ahead-logging, and other work involved that all adds latency. You can overcome some of that through parallelism, but 30K IOPs per OSD is probably pretty on-point for a nautilus era OSD. For octopus+ the cache refactor in bluestore should get you farther (40-50k+ for and OSD in isolation). The maximum performance we've seen in-house is around 70-80K IOPs on a single OSD using very fast NVMe and highly tuned settings.
A couple of things you can try:
- upgrade to octopus+ for the cache refactor
- Make sure you are using the equivalent of the latency-performance or latency-network tuned profile. The most important part is disabling CPU cstate transitions.
- increase osd_memory_target if you have a larger dataset (onode cache misses in bluestore add a lot of latency)
- enable turbo if it's disabled (higher clock speed generally helps)
On the write path you are correct that there is a limitation regarding a single kv sync thread. Over the years we've made this less of a bottleneck but it's possible you still could be hitting it. In our test lab we've managed to utilize up to around 12-14 cores on a single OSD in isolation with 16 tp_osd_tp worker threads and on a larger cluster about 6-7 cores per OSD. There's probably multiple factors at play, including context switching, cache thrashing, memory throughput, object creation/destruction, etc. If you decide to look into it further you may want to try wallclock profiling the OSD under load and seeing where it is spending its time.
Thank you for feedback. I forgot to mention this, it's Octopus, fresh installation. I've disabled CSTATE (governor=performance), it make no difference - same iops, same CPU use by ceph-osd I've just can't force Ceph to consume more than 330% of CPU. I can force read up to 150k IOPS (both network and local), hitting CPU limit, but write is somewhat restricted by ceph itself.