With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a less-dense probably is fine with 25Gb/s. And of course PCI lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-c...
I was able to reach 35 Gb/s network traffic on each server (5 servers, with 6 NVMEs per server, one OSD per NVME) during a read benchmark from cephfs, and I wouldn't treat that as a super-dense box. So 25Gb/s may be a bit too tight.
Thanks for the data point — without real-world reports, it’s all just theoretical. In the above presentation the point is made that latency is more important than throughput, but this is very, very much a function of the use-case. For DBs on RBD volumes, there’s a lot of truth to that especially for writes. For, say, object service or for things like OpenStack Glance, it may often be the other way around.
The workload doesn’t demand NVMe performance, so SSD seems to be the most cost effective way to handle this.
To be pedantic, NVMe devices *are* SSDs, but you most likely mean SATA SSDs. The thing is, with recent drives, chassis, and CPU models, it can be very possible to provision an NVMe server at a cost comparable to a conventional SATA SSD server, in which case, why not? — aad