Hi Peter, I'm not quite sure if you're cluster is fully backed by NVMe drives based on your description, but you might be interested in the CPU scaling article we posted last fall. It's available here: https://ceph.io/en/news/blog/2022/ceph-osd-cpu-scaling/ That gives a good overview of what kind of performance you can get out of Ceph in a good environment with all NVMe drives. We also have a tuning article for using QEMU KVM which shows how big of a difference various tuning options in the whole IO pipeline can make: https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/ The gist of it is that there are a lot of things that can negatively affect performance, but if you can isolate and fix them it's possible to get reasonably high performance in the end. If you have HDD based OSDs with NVMe only for DB/WAL, you will ultimately be limited by the random IO performance of the HDDs. The WAL can help a little but not like a full tiering solution. We have some ideas regarding how to improve this in the future. If you have a test cluster or are simply experimenting, you could try deploying on top of Intel's OpenCAS or bcache. There have been reports of improvements for HDD backed clusters using these solutions, though they are not currently supported officially by the project afaik. Mark On 1/23/23 14:58, petersun(a)raksmart.com wrote: > I have my ceph IOPS very low with over 48 SSD backed on NVMs for DB/WAL on four physical servers. The whole cluster has only about 20K IO total. Looks the IOs are suppressed over bottleneck somewhere. Dstat shows a lots csw and interrupts over 150K, while I am using FIO bench 4K 128QD test. > I check SSD throughput only about 40M at 250 ios each. Network are total 20G and not full of traffic. CPU are around 50% idle on 2*E5 2950v2 each node. > Is it normal to get that high and how to reduce it? where else could be the bottleneck? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io