With today’s
networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a less-dense probably is
fine with 25Gb/s. And of course PCI lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-…
I was able to reach 35 Gb/s network traffic on each server (5 servers,
with 6 NVMEs per server, one OSD per NVME) during a read benchmark
from cephfs, and I wouldn't treat that as a super-dense box. So 25Gb/s
may be a bit too tight.
Thanks for the data point — without real-world reports, it’s all just theoretical. In the
above presentation the point is made that latency is more important than throughput, but
this is very, very much a function of the use-case. For DBs on RBD volumes, there’s a lot
of truth to that especially for writes. For, say, object service or for things like
OpenStack Glance, it may often be the other way around.
The workload doesn’t demand NVMe performance, so SSD
seems to be the most cost effective way to handle this.
To be pedantic, NVMe devices *are* SSDs, but you most likely mean SATA SSDs.
The thing is, with recent drives, chassis, and CPU models, it can be very possible to
provision an NVMe server at a cost comparable to a conventional SATA SSD server, in which
case, why not?
— aad