Then I tested on the *Proxmox host*, and the results were significantly better.
My Proxmox prowess is limited, but from my experience with other virtualization platforms, I have to ask if there is any QoS throttling applied to VMs. With OpenStack or DO there is often IOPS and/or throughput throttling via libvirt to mitigate noisy neighbors.
fio --name=host-test --filename=/dev/rbd0 --ioengine=libaio --rw=randread --bs=4k --numjobs=4 --iodepth=32 --size=1G --runtime=60 --group_reporting
*IOPS*: *1.54M*
# *Bandwidth*: *6032MiB/s (6325MB/s)* # *Latency*:
* *Avg*: *39.8µs* * *99.9th percentile*: *71µs*
# *CPU Usage*: *usr=22.60%, sys=77.13%* #
Am 18.03.2025 um 15:27 schrieb Anthony D'Atri:
Which NVMe drive SKUs specifically?
# */dev/nvme6n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02QT5A8 # */dev/nvme5n1* – *KCD61LUL15T3* – 15.36 TB – SN: 42R0A036T5A8 # */dev/nvme4n1* – *KCD61LUL15T3* – 15.36 TB – SN: 6250A02UT5A8
Kioxia CD6. If you were using client-class drives all manner of performance issues would be expected. Is your server chassis at least PCIe Gen 4? If it’s Gen 3 that may hamper these drives. Also, how many of these are in your cluster? If it’s a small number you might still benefit from chopping each into at least 2 separate OSDs. And please send `ceph osd dump | grep pool`, having too few PGs wouldn’t do you any favors.
Are you running a recent kernel? penultimate: 6.8.12-8-pve (VM, yes)
Groovy. If you were running like a CentOS 6 or CentOS 7 kernel then NVMe issues might be expected as old kernels had rudimentary NVMe support.
Have you updated firmware on the NVMe devices?
No.
Kioxia appears to not release firmware updates publicly but your chassis brand (Dell, HP, SMCI, etc) might have an update. e.g. https://www.dell.com/support/home/en-vc/drivers/driversdetails?driverid=7ny5... If there is an available update I would strongly suggest applying.
Thanks again,
best regards, Gio
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io