Thanks for your answer !
These things are always controversial, below are my thoughts. Your mileage may vary.
We plan to buy hardware for a new cluster ceph and would like some approbation on what we choose.
6 nodes DELL 6715 + 6 Powervault MD2412 enclosure
I would not recommend using external drive arrays:
* You're paying money for an HBA * Proprietary tool for managing the thing, granted this one looks like a JBOD vs one with an embedded RAID controller, which I've previously wrangled with * 24 Gb/s SAS is still SAS. SAS or SATA purchases today pretty much lock you into HDDs in the future.
This looks like a good storage efficency / price ratio for us today
For each node =>
1 CPU AMD EPYC 9475F 3,65 GHz, 48C/96T, 256M Cache (400 W)
Yikes. Higher-freq CPUs or those with the highest core counts tend to cost more in terms of oomph per euro than middle-of-the-road SKUs. Note the significant list price difference >compared to say the 9455P. For many purposes, solve for $ / ((cores) * (base)). Servethehome posts nice value analysis charts. Heck, if a single-socket system meets your needs for e.g. >RAM capacity and PCIe lanes, consider the 9555P.
Thanks we change the CPU based on your advice
RAM 16Gox16 = 256Go
For 12 OSDs you would likely be fine with 128GB, with slots empty for expansion.
Alrgith is it enough for MON/MGR/OSD ?
4x 100GB NVIDIA MELLANOX 100GB
Why 4x? With 12 HDD OSDs you'd be well-served by 2x 25 GE, skipping a replication network.
We thought that 2 x 100GB for public and 2 x 100GB for private would be a good setup with NVMe
HBA465e (externe, 22,5GB/s)
2 NVMe mixed 6,4To DBWAL 6 NVMe mixed 6,4To
mixed-use SSDs are usually overkill. They're usually the same hardware as read-intensive SSDs with the overprovisioning slider (and the price) bumped. "read-intensive" and "mixed-use" are marketing terms. I worked for the product / marketing team of an SSD manufacturer, trust me on this ;)
We saw that mixed-use SSDs can achieve twice the IOPS of read-intensive SSDs in write operations. This is why we plan to use this type of SSD for high-write workloads.
5 NVMe read 15,36To
for each enclosure =>
12 HDD 20To
HDD will be a replica 3 pool with the dbwal
With modern NVMe SSDs, I might suggest 1x 7.6 T for DB/WAL offload. You can go 12:1.
thanks
NVME mixed and read will be 2 differents pools (we will test replica 3 and EC to see which performance/storage efficency satisfy us the most)
Why two different, small pools? You'd be better off getting all read-intensive NVMe SSDs, and to keep things simple make all of them the same capacity. You would benefit from having >more OSDs in a single pool, especially since this is a small cluster.
If we take only read-intensive NVMe SSDs this is the plan
This cluster will be mostly use to store block (VM proxmox/kubernetes)
On the SSDs I presume? That workload on HDDs, especially at 20T, is unlikely to be adequate.
We thought that low-IOPS workloads would run pretty well on HDDs, as today our VMs are on a HDD pool with a cache tier SSD. Are HDDs/DBWAL unsuitable for this type of workload in Ceph? Full NVMEs is very expensive, I don't really know if we can do it in our cluster if EC is not suitable as we need around 400 To => In replica 3 the setup that I shown is around 230 To of NVMEs and 480 To of HDDs
and S3.
HDDs just for the bucket pool, right? Everything else on the SSDs?
At the begining we planned to use HDD for S3 and RBD
The crushmap will be like so : 3 rooms with 2 nodes per room.
So the point of failure can be for replica 3 => rooms and if we use EC 4+2 => host
With Tentacle fast EC, you *might* find that RBD on EC performance is acceptable, worth testing.
If EC is acceptable full NVMe could be an option
We saw that the thread needs for NVMe OSD are very expensive, does this CPU good enough to carry them ?
Be sure to read those references carefully. The one that describes gains up to 14 threads per is in a very specific context.
So let's say you have 8x 15T NVMe RI OSDs + 12x HDD OSDs. I might budget for 6 and 4 threads each, respectively, with some left over for mon, mgr, rgw, etc. The 96 thread CPU is a bit >tight given that math, but I think you'd probably be okay.
MON and MGR will be spread across the cluster and RGW on virtual machine
Oh, RGWs on VMs would alleviate the CPU appetite a bit. This is not an unknown strategy.
I would personally consider an all-NVMe chassis with a mix of TLC and QLC/QLC-class SSDs for the bucket pool. You'd use fewer RUs and your recovery times would be dramatically improved. >Consider how long it takes to write a 20T HDD.
We didn't have the difference beetwen TLC and QLC SSDs specs, is this a things in Ceph ? Our provider didn't display them
Thanks for your answer !
Vivien
________________________________ De : Anthony D'Atri <anthony.datri@gmail.com> Envoyé : mardi 18 novembre 2025 14:20:22 À : GLE, Vivien Cc : ceph-users@ceph.io Objet : Re: [ceph-users] New Cluster Ceph These things are always controversial, below are my thoughts. Your mileage may vary.
We plan to buy hardware for a new cluster ceph and would like some approbation on what we choose.
6 nodes DELL 6715 + 6 Powervault MD2412 enclosure
I would not recommend using external drive arrays: * You're paying money for an HBA * Proprietary tool for managing the thing, granted this one looks like a JBOD vs one with an embedded RAID controller, which I've previously wrangled with * 24 Gb/s SAS is still SAS. SAS or SATA purchases today pretty much lock you into HDDs in the future.
For each node =>
1 CPU AMD EPYC 9475F 3,65 GHz, 48C/96T, 256M Cache (400 W)
Yikes. Higher-freq CPUs or those with the highest core counts tend to cost more in terms of oomph per euro than middle-of-the-road SKUs. Note the significant list price difference compared to say the 9455P. For many purposes, solve for $ / ((cores) * (base)). Servethehome posts nice value analysis charts. Heck, if a single-socket system meets your needs for e.g. RAM capacity and PCIe lanes, consider the 9555P.
RAM 16Gox16 = 256Go
For 12 OSDs you would likely be fine with 128GB, with slots empty for expansion.
4x 100GB NVIDIA MELLANOX 100GB
Why 4x? With 12 HDD OSDs you'd be well-served by 2x 25 GE, skipping a replication network.
HBA465e (externe, 22,5GB/s)
2 NVMe mixed 6,4To DBWAL 6 NVMe mixed 6,4To
mixed-use SSDs are usually overkill. They're usually the same hardware as read-intensive SSDs with the overprovisioning slider (and the price) bumped. "read-intensive" and "mixed-use" are marketing terms. I worked for the product / marketing team of an SSD manufacturer, trust me on this ;)
5 NVMe read 15,36To
for each enclosure =>
12 HDD 20To
HDD will be a replica 3 pool with the dbwal
With modern NVMe SSDs, I might suggest 1x 7.6 T for DB/WAL offload. You can go 12:1.
NVME mixed and read will be 2 differents pools (we will test replica 3 and EC to see which performance/storage efficency satisfy us the most)
Why two different, small pools? You'd be better off getting all read-intensive NVMe SSDs, and to keep things simple make all of them the same capacity. You would benefit from having more OSDs in a single pool, especially since this is a small cluster.
This cluster will be mostly use to store block (VM proxmox/kubernetes)
On the SSDs I presume? That workload on HDDs, especially at 20T, is unlikely to be adequate.
and S3.
HDDs just for the bucket pool, right? Everything else on the SSDs?
The crushmap will be like so : 3 rooms with 2 nodes per room.
So the point of failure can be for replica 3 => rooms and if we use EC 4+2 => host
With Tentacle fast EC, you *might* find that RBD on EC performance is acceptable, worth testing.
We saw that the thread needs for NVMe OSD are very expensive, does this CPU good enough to carry them ?
Be sure to read those references carefully. The one that describes gains up to 14 threads per is in a very specific context. So let's say you have 8x 15T NVMe RI OSDs + 12x HDD OSDs. I might budget for 6 and 4 threads each, respectively, with some left over for mon, mgr, rgw, etc. The 96 thread CPU is a bit tight given that math, but I think you'd probably be okay.
MON and MGR will be spread across the cluster and RGW on virtual machine
Oh, RGWs on VMs would alleviate the CPU appetite a bit. This is not an unknown strategy. I would personally consider an all-NVMe chassis with a mix of TLC and QLC/QLC-class SSDs for the bucket pool. You'd use fewer RUs and your recovery times would be dramatically improved. Consider how long it takes to write a 20T HDD.
Thanks for your answer !
Vivien
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io