Hi, From my limited experience erasure coding with HDD is painful with only 2 parity shards. I would recommend to take at least 3 parity shards, because frequently when you take down 1 server you'll discover that some HDD drives on other servers are flaky and restart or goes down for no reason except the load increase due to rebalancing. Due to this we had a lot of read-only error as soon as one server need replacement. I have experienced on small clusters with EC2+2 where parity=3 was not possible (4 servers 96 HDD) and we choose to use replication instead that is less problematic with HDD in my experience. I was using enterprise grade hardware, but you know the doc says that you've to carefully choose disks and avoid using RAID controller... in real life most of us do not have the ability to do that. With 7+ servers I would choose EC3+3 for HDD, but below that I prefer to keep it safe and use replication x3. Regards Le mer. 4 nov. 2026 à 16:14, Javier Charne <ceph-users@ceph.io> a écrit :
Thanks, Anthony!
Yes, these 5 nodes will have 12 20TiB HDD, and the 2 4TiB NVME drives for (6 HDD) Wal+DB each.
And I’m taking your 2+2 suggestion into account, so that k+m doesn’t exceed the failure domains.
Do you think 25Gbps is enough for replication traffic?
And what can you tell me about performance? Compared to “pure SSD,” is it significantly slower?
Thanks in advance!
Javier.-
On 30/10/26 15:40, Anthony D'Atri wrote:
On Oct 30, 2026, at 1:42 PM, Javier Charne<ceph-users@ceph.io> wrote:
Hello!
Since NVME drives become quite expensive, we are thinking about adding
Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon
6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04
Total OSDs: 423 x 3.5 TB ~ (1.9 PB)
We want to add 5 servers, each with:
- 12 x 20 TB HDD,
- 4 NVME 2TB for WAL+DB,
Why 4x NVMe SSDs for offload? For your workload you really only need one NVMe SSD for 20 HDDs' DB+WAL.
Remember though that you'll want SSD OSDs for the RBD replicated
a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS) metadata pool. If you have 4x NVMe slots, you might provision
1x 2T split into 2x ssd device class OSDs for the meta pool(s) 2x 4T each taking 10x HDDs' WAL+DB
That leaves you with a slot in case your needs change later.
- 2 SSD for operative system,
- 256 GB RAM,
- 24 CPU
- 4 x 25 Gbps NIC interfaces. I would think that 2x would suffice.
One pool with replica 3 and another with EC 3+2 Why 3+2? Oh, are your existing OSD nodes all SSDs, so your HDD pools
would be confined to these new 5x servers? If so, then I would suggest 2+2. For various reasons you want the number of failure domains to be > k+m.
Please feel free to share any comments or experiences you may have on
this topic—they are more than welcome.
Thanks in advance!
Javier.-
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io