On Dec 22, 2025, at 6:47 AM, George Shuklin via ceph-users <ceph-users@ceph.io> wrote:
On 12/16/25 3:37 PM, Anthony D'Atri via ceph-users wrote:
I'm tryting to debug low performance on nvme-based cluster. Which specific drives?
INTEL SSDPF2KE032T1
P5520, excellent drive. We see people on the list using random client-class drives, so the drives in use are a first question to ask, because so many people don't understand the differences. I work with a community member who has his CephFS metadata pool on *Sabrent* client drives. Poster child for "flirting with disaster".
Ceph accepts about 20 requests, proccess them slowly, and then write to devices (super fast), so device sits idle most of the time. OSD daemons are not CPU/memory constrained (~40 cores idle, plenty of memory), os it's just disparity between Ceph OSD speed and backend speed.
For sure there are serializations in the PG and OSD code, which is one reason why a healthy PG ratio matters. Remember that NVMe SSDs can process lots of operations in parallel, too. Have you discussed this with Mark Nelson?
I remember, few years ago (on lesser hardware) I done benchmark for ceph using brd (block ram disk) and ceph was able to churn up to 10k IOPS per daemon. Nowdays, it can up to 20k, I think.
Mark would know for sure. When doing such tests, be sure that your pool is size=1 so that you aren't constrained by replication ops. In practice, network latency and replication are often significant factors in speed. Also, is the subject system free from swap? Do you have a static osd_memory_target, or do you have the autotuner enabled?
Actually, it's a good question: what is the maxium iops a single OSD daemon can deliver with perfectly fast underlaying storage and negligible network?
Mark territory.
but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). Have you updated to the latest firmware with SST?
Yep. Certified and the most up to date firmware.
Good, with the latest SST? Or if you got the drives from Dell, DSU? I worked for the Solidigm product team for a year, some of the firmware updates are very important.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io