Hello! Since NVME drives become quite expensive, we are thinking about adding a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS) Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon 6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04 Total OSDs: 423 x 3.5 TB ~ (1.9 PB) We want to add 5 servers, each with: - 12 x 20 TB HDD, - 4 NVME 2TB for WAL+DB, - 2 SSD for operative system, - 256 GB RAM, - 24 CPU - 4 x 25 Gbps NIC interfaces. One pool with replica 3 and another with EC 3+2 Please feel free to share any comments or experiences you may have on this topic—they are more than welcome. Thanks in advance! Javier.-
On Oct 30, 2026, at 1:42 PM, Javier Charne <ceph-users@ceph.io> wrote:
Hello!
Since NVME drives become quite expensive, we are thinking about adding a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS)
Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon 6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04
Total OSDs: 423 x 3.5 TB ~ (1.9 PB)
We want to add 5 servers, each with:
- 12 x 20 TB HDD,
- 4 NVME 2TB for WAL+DB,
Why 4x NVMe SSDs for offload? For your workload you really only need one NVMe SSD for 20 HDDs' DB+WAL. Remember though that you'll want SSD OSDs for the RBD replicated metadata pool. If you have 4x NVMe slots, you might provision 1x 2T split into 2x ssd device class OSDs for the meta pool(s) 2x 4T each taking 10x HDDs' WAL+DB That leaves you with a slot in case your needs change later.
- 2 SSD for operative system,
- 256 GB RAM,
- 24 CPU
- 4 x 25 Gbps NIC interfaces.
I would think that 2x would suffice.
One pool with replica 3 and another with EC 3+2
Why 3+2? Oh, are your existing OSD nodes all SSDs, so your HDD pools would be confined to these new 5x servers? If so, then I would suggest 2+2. For various reasons you want the number of failure domains to be > k+m.
Please feel free to share any comments or experiences you may have on this topic—they are more than welcome.
Thanks in advance!
Javier.-
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, Anthony! Yes, these 5 nodes will have 12 20TiB HDD, and the 2 4TiB NVME drives for (6 HDD) Wal+DB each. And I’m taking your 2+2 suggestion into account, so that k+m doesn’t exceed the failure domains. Do you think 25Gbps is enough for replication traffic? And what can you tell me about performance? Compared to “pure SSD,” is it significantly slower? Thanks in advance! Javier.- On 30/10/26 15:40, Anthony D'Atri wrote:
On Oct 30, 2026, at 1:42 PM, Javier Charne<ceph-users@ceph.io> wrote:
Hello!
Since NVME drives become quite expensive, we are thinking about adding a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS)
Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon 6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04
Total OSDs: 423 x 3.5 TB ~ (1.9 PB)
We want to add 5 servers, each with:
- 12 x 20 TB HDD,
- 4 NVME 2TB for WAL+DB, Why 4x NVMe SSDs for offload? For your workload you really only need one NVMe SSD for 20 HDDs' DB+WAL.
Remember though that you'll want SSD OSDs for the RBD replicated metadata pool. If you have 4x NVMe slots, you might provision
1x 2T split into 2x ssd device class OSDs for the meta pool(s) 2x 4T each taking 10x HDDs' WAL+DB
That leaves you with a slot in case your needs change later.
- 2 SSD for operative system,
- 256 GB RAM,
- 24 CPU
- 4 x 25 Gbps NIC interfaces. I would think that 2x would suffice.
One pool with replica 3 and another with EC 3+2 Why 3+2? Oh, are your existing OSD nodes all SSDs, so your HDD pools would be confined to these new 5x servers? If so, then I would suggest 2+2. For various reasons you want the number of failure domains to be > k+m.
Please feel free to share any comments or experiences you may have on this topic—they are more than welcome.
Thanks in advance!
Javier.-
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hi, From my limited experience erasure coding with HDD is painful with only 2 parity shards. I would recommend to take at least 3 parity shards, because frequently when you take down 1 server you'll discover that some HDD drives on other servers are flaky and restart or goes down for no reason except the load increase due to rebalancing. Due to this we had a lot of read-only error as soon as one server need replacement. I have experienced on small clusters with EC2+2 where parity=3 was not possible (4 servers 96 HDD) and we choose to use replication instead that is less problematic with HDD in my experience. I was using enterprise grade hardware, but you know the doc says that you've to carefully choose disks and avoid using RAID controller... in real life most of us do not have the ability to do that. With 7+ servers I would choose EC3+3 for HDD, but below that I prefer to keep it safe and use replication x3. Regards Le mer. 4 nov. 2026 à 16:14, Javier Charne <ceph-users@ceph.io> a écrit :
Thanks, Anthony!
Yes, these 5 nodes will have 12 20TiB HDD, and the 2 4TiB NVME drives for (6 HDD) Wal+DB each.
And I’m taking your 2+2 suggestion into account, so that k+m doesn’t exceed the failure domains.
Do you think 25Gbps is enough for replication traffic?
And what can you tell me about performance? Compared to “pure SSD,” is it significantly slower?
Thanks in advance!
Javier.-
On 30/10/26 15:40, Anthony D'Atri wrote:
On Oct 30, 2026, at 1:42 PM, Javier Charne<ceph-users@ceph.io> wrote:
Hello!
Since NVME drives become quite expensive, we are thinking about adding
Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon
6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04
Total OSDs: 423 x 3.5 TB ~ (1.9 PB)
We want to add 5 servers, each with:
- 12 x 20 TB HDD,
- 4 NVME 2TB for WAL+DB,
Why 4x NVMe SSDs for offload? For your workload you really only need one NVMe SSD for 20 HDDs' DB+WAL.
Remember though that you'll want SSD OSDs for the RBD replicated
a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS) metadata pool. If you have 4x NVMe slots, you might provision
1x 2T split into 2x ssd device class OSDs for the meta pool(s) 2x 4T each taking 10x HDDs' WAL+DB
That leaves you with a slot in case your needs change later.
- 2 SSD for operative system,
- 256 GB RAM,
- 24 CPU
- 4 x 25 Gbps NIC interfaces. I would think that 2x would suffice.
One pool with replica 3 and another with EC 3+2 Why 3+2? Oh, are your existing OSD nodes all SSDs, so your HDD pools
would be confined to these new 5x servers? If so, then I would suggest 2+2. For various reasons you want the number of failure domains to be > k+m.
Please feel free to share any comments or experiences you may have on
this topic—they are more than welcome.
Thanks in advance!
Javier.-
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Javier, What ceph version are you planning to use?, If it is newer or equivalent to squid, you can easily use EC4+3 with the new CRUSH MSR introduced in squid. For the early version, you must manually modify the CRUSH rule for EC4+3. (Note, if using EC4+3 (pick 2 osd from 1 host), it's safe for 3 HDD failures, but not for 2 NVMe Metadata failures). Regards, On Wed, Nov 4, 2026 at 10:29 PM Julien Laurenceau <ceph-users@ceph.io> wrote:
Hi,
From my limited experience erasure coding with HDD is painful with only 2 parity shards. I would recommend to take at least 3 parity shards, because frequently when you take down 1 server you'll discover that some HDD drives on other servers are flaky and restart or goes down for no reason except the load increase due to rebalancing. Due to this we had a lot of read-only error as soon as one server need replacement.
I have experienced on small clusters with EC2+2 where parity=3 was not possible (4 servers 96 HDD) and we choose to use replication instead that is less problematic with HDD in my experience. I was using enterprise grade hardware, but you know the doc says that you've to carefully choose disks and avoid using RAID controller... in real life most of us do not have the ability to do that.
With 7+ servers I would choose EC3+3 for HDD, but below that I prefer to keep it safe and use replication x3.
Regards
Le mer. 4 nov. 2026 à 16:14, Javier Charne <ceph-users@ceph.io> a écrit :
Thanks, Anthony!
Yes, these 5 nodes will have 12 20TiB HDD, and the 2 4TiB NVME drives for (6 HDD) Wal+DB each.
And I’m taking your 2+2 suggestion into account, so that k+m doesn’t exceed the failure domains.
Do you think 25Gbps is enough for replication traffic?
And what can you tell me about performance? Compared to “pure SSD,” is it significantly slower?
Thanks in advance!
Javier.-
On 30/10/26 15:40, Anthony D'Atri wrote:
On Oct 30, 2026, at 1:42 PM, Javier Charne<ceph-users@ceph.io> wrote:
Hello!
Since NVME drives become quite expensive, we are thinking about adding
Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon
6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04
Total OSDs: 423 x 3.5 TB ~ (1.9 PB)
We want to add 5 servers, each with:
- 12 x 20 TB HDD,
- 4 NVME 2TB for WAL+DB,
Why 4x NVMe SSDs for offload? For your workload you really only need one NVMe SSD for 20 HDDs' DB+WAL.
Remember though that you'll want SSD OSDs for the RBD replicated
a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS) metadata pool. If you have 4x NVMe slots, you might provision
1x 2T split into 2x ssd device class OSDs for the meta pool(s) 2x 4T each taking 10x HDDs' WAL+DB
That leaves you with a slot in case your needs change later.
- 2 SSD for operative system,
- 256 GB RAM,
- 24 CPU
- 4 x 25 Gbps NIC interfaces. I would think that 2x would suffice.
One pool with replica 3 and another with EC 3+2 Why 3+2? Oh, are your existing OSD nodes all SSDs, so your HDD pools
would be confined to these new 5x servers? If so, then I would suggest 2+2. For various reasons you want the number of failure domains to be > k+m.
Please feel free to share any comments or experiences you may have on
this topic—they are more than welcome.
Thanks in advance!
Javier.-
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I agree with you that it's better to use Replica 3 with so few nodes. I was eager to try EC to make better use of the space, but—as you say—I don't want to open up another front. Thanks for your suggestions! Javier.- On 4/11/26 12:29, Julien Laurenceau wrote:
Hi,
From my limited experience erasure coding with HDD is painful with only 2 parity shards. I would recommend to take at least 3 parity shards, because frequently when you take down 1 server you'll discover that some HDD drives on other servers are flaky and restart or goes down for no reason except the load increase due to rebalancing. Due to this we had a lot of read-only error as soon as one server need replacement.
I have experienced on small clusters with EC2+2 where parity=3 was not possible (4 servers 96 HDD) and we choose to use replication instead that is less problematic with HDD in my experience. I was using enterprise grade hardware, but you know the doc says that you've to carefully choose disks and avoid using RAID controller... in real life most of us do not have the ability to do that.
With 7+ servers I would choose EC3+3 for HDD, but below that I prefer to keep it safe and use replication x3.
Regards
Le mer. 4 nov. 2026 à 16:14, Javier Charne <ceph-users@ceph.io> a écrit :
Thanks, Anthony!
Yes, these 5 nodes will have 12 20TiB HDD, and the 2 4TiB NVME drives for (6 HDD) Wal+DB each.
And I’m taking your 2+2 suggestion into account, so that k+m doesn’t exceed the failure domains.
Do you think 25Gbps is enough for replication traffic?
And what can you tell me about performance? Compared to “pure SSD,” is it significantly slower?
Thanks in advance!
Javier.-
On Oct 30, 2026, at 1:42 PM, Javier Charne<ceph-users@ceph.io> wrote:
Hello!
Since NVME drives become quite expensive, we are thinking about adding a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS) Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon 6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04 Total OSDs: 423 x 3.5 TB ~ (1.9 PB)
We want to add 5 servers, each with:
- 12 x 20 TB HDD,
- 4 NVME 2TB for WAL+DB, Why 4x NVMe SSDs for offload? For your workload you really only need one NVMe SSD for 20 HDDs' DB+WAL. Remember though that you'll want SSD OSDs for the RBD replicated
1x 2T split into 2x ssd device class OSDs for the meta pool(s) 2x 4T each taking 10x HDDs' WAL+DB
That leaves you with a slot in case your needs change later.
- 2 SSD for operative system,
- 256 GB RAM,
- 24 CPU
- 4 x 25 Gbps NIC interfaces. I would think that 2x would suffice.
One pool with replica 3 and another with EC 3+2 Why 3+2? Oh, are your existing OSD nodes all SSDs, so your HDD pools would be confined to these new 5x servers? If so, then I would suggest 2+2. For various reasons you want the number of failure domains to be > k+m. Please feel free to share any comments or experiences you may have on
On 30/10/26 15:40, Anthony D'Atri wrote: metadata pool. If you have 4x NVMe slots, you might provision this topic—they are more than welcome.
Thanks in advance!
Javier.-
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anh Phan Tuan
-
Anthony D'Atri
-
Javier Charne
-
Julien Laurenceau