Thanks, Anthony! Yes, these 5 nodes will have 12 20TiB HDD, and the 2 4TiB NVME drives for (6 HDD) Wal+DB each. And I’m taking your 2+2 suggestion into account, so that k+m doesn’t exceed the failure domains. Do you think 25Gbps is enough for replication traffic? And what can you tell me about performance? Compared to “pure SSD,” is it significantly slower? Thanks in advance! Javier.- On 30/10/26 15:40, Anthony D'Atri wrote:
On Oct 30, 2026, at 1:42 PM, Javier Charne<ceph-users@ceph.io> wrote:
Hello!
Since NVME drives become quite expensive, we are thinking about adding a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS)
Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon 6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04
Total OSDs: 423 x 3.5 TB ~ (1.9 PB)
We want to add 5 servers, each with:
- 12 x 20 TB HDD,
- 4 NVME 2TB for WAL+DB, Why 4x NVMe SSDs for offload? For your workload you really only need one NVMe SSD for 20 HDDs' DB+WAL.
Remember though that you'll want SSD OSDs for the RBD replicated metadata pool. If you have 4x NVMe slots, you might provision
1x 2T split into 2x ssd device class OSDs for the meta pool(s) 2x 4T each taking 10x HDDs' WAL+DB
That leaves you with a slot in case your needs change later.
- 2 SSD for operative system,
- 256 GB RAM,
- 24 CPU
- 4 x 25 Gbps NIC interfaces. I would think that 2x would suffice.
One pool with replica 3 and another with EC 3+2 Why 3+2? Oh, are your existing OSD nodes all SSDs, so your HDD pools would be confined to these new 5x servers? If so, then I would suggest 2+2. For various reasons you want the number of failure domains to be > k+m.
Please feel free to share any comments or experiences you may have on this topic—they are more than welcome.
Thanks in advance!
Javier.-
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io