On 12/16/25 3:37 PM, Anthony D'Atri via ceph-users wrote:
I'm tryting to debug low performance on nvme-based cluster. Which specific drives?
INTEL SSDPF2KE032T1 Per specs it should give about 300k IOPS.
I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no scrubbing or replication atm, 1024 pg for 24 OSD/17TB.
I expect to see some performance (~250k IOPS total, 50% r/w, few hundred volumes with capped by IOPS, pre-warmed). I see about half of it.
I looked at drive utilization, it's about 70% (per atop) That's meaningless for SSDs.
Yes, I looked deeper, and I found that most of the time there is 0 or 1, with very rarely 2 inflight operations, and all of them are done (from nvme point of view) within 50-70µs. At the same time (under constant load from fio) I can see about 10-20 op_wip, so my current theory goes like this: Ceph accepts about 20 requests, proccess them slowly, and then write to devices (super fast), so device sits idle most of the time. OSD daemons are not CPU/memory constrained (~40 cores idle, plenty of memory), os it's just disparity between Ceph OSD speed and backend speed. I remember, few years ago (on lesser hardware) I done benchmark for ceph using brd (block ram disk) and ceph was able to churn up to 10k IOPS per daemon. Nowdays, it can up to 20k, I think. Actually, it's a good question: what is the maxium iops a single OSD daemon can deliver with perfectly fast underlaying storage and negligible network?
but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). Have you updated to the latest firmware with SST?
Yep. Certified and the most up to date firmware.