Ceph cluster design advice
Hello Ceph community, We are provisioning a new Ceph cluster and would appreciate design advice. Below is our current hardware: - 13 data servers each with 18 x 24 TB HDD and 2 x 3.5 TB SSD (6 servers in rack A, 7 in rack B) - 3 control servers each with 2 x 960 GB SSD and 4 x 1.9 TB SSD (2 in rack A, 1 in rack B) The planned topology is: - A single CephFS backed by a data pool - The 3 control servers will run MON/MGR/MDS and other daemons. The 13 data servers will host OSDs and some extra MONs as needed. Current idea (constraints: we cannot add hardware) is on each data server we will create a RAID1 from the two 3.5 TB SSDs and carve it into a 500 GB virtual disk for the OS and a 3 TB virtual disk for DB/WAL (Bluestore). Each HDD will be an OSD, giving 18 x 24 TB = 432 TB raw HDD per data node. 1) Is it acceptable to host WAL/DB on a RAID1 virtual disk made from the two SSDs, or is that a bad idea for performance/reliability? 2) Is 3 TB for DB/WAL per data node (~167 GB per OSD) likely to be sufficient for 18 x 24 TB OSDs, or should we expect to need more DB space given CephFS metadata/object counts? (We've seen a 2% rule cited, that would be ~8.6 TB per node which is much larger) 3) Alternative: keep WAL/DB on the HDDs (i.e., do not separate), and use the 3 TB RAID1 SSD area plus the control servers' SSDs (4 x 1.9 TB each) as SSD OSDs for the CephFS metadata pool. Would that be a reasonable approach? 4) If we do SSD-backed metadata OSDs with 3x replication for the metadata pool, we were thinking of CRUSH domains: domain1 = 6 data servers in rack A (6 x 3 TB SSD OSDs), domain2 = 7 data servers in rack B (7 x 3 TB), domain3 = 3 control servers (12 x 1.9 TB SSD OSDs). Any pitfalls with that CRUSH layout? Any suggestions would be highly appreciated. Thank you, Gustavo
On Nov 4, 2025, at 1:32 PM, Gustavo Garcia Rondina <grondina@uchicago.edu> wrote:
Hello Ceph community,
We are provisioning a new Ceph cluster and would appreciate design advice. Below is our current hardware:
- 13 data servers each with 18 x 24 TB HDD and 2 x 3.5 TB SSD (6 servers in rack A, 7 in rack B)
I'd spread them over more racks if you can, and align the CRUSH topology. Are these SSDs SAS/SATA or NVMe?
- 3 control servers each with 2 x 960 GB SSD and 4 x 1.9 TB SSD (2 in rack A, 1 in rack B)
5 is better than 3, but you could place two additional mons on the OSD nodes. Are these SSDs SAS/SATA or NVMe?
The planned topology is:
- A single CephFS backed by a data pool - The 3 control servers will run MON/MGR/MDS and other daemons. The 13 data servers will host OSDs and some extra MONs as needed.
Ack.
Current idea (constraints: we cannot add hardware) is on each data server we will create a RAID1 from the two 3.5 TB SSDs and carve it into a 500 GB virtual disk for the OS and a 3 TB virtual disk for DB/WAL (Bluestore).
I recommend not mixing the OS and data. What SKU are these SSDs? Are they enterprise-class with PLP? If those are SAS/SATA SSDs, conventional wisdom is to not offload more than 4-5 HDD OSD WAL+DBs onto each.
Each HDD will be an OSD, giving 18 x 24 TB = 432 TB raw HDD per data node.
1) Is it acceptable to host WAL/DB on a RAID1 virtual disk made from the two SSDs, or is that a bad idea for performance/reliability?
It's better to map half the OSDs to each SSD. RADOS handles redundancy. Some people mirror offload SSDs, but IMHO that mostly just burns them twice as fast.
2) Is 3 TB for DB/WAL per data node (~167 GB per OSD) likely to be sufficient for 18 x 24 TB OSDs, or should we expect to need more DB space given CephFS metadata/object counts? (We've seen a 2% rule cited, that would be ~8.6 TB per node which is much larger)
Rightsizing depends in part on your workload. Will you host a modest number of big files, a zillion tiny ones, or a mix? I suspect with your numbers that the question is moot and you'll have to forego WAL+DB offload. This is one reason to disfavor dense OSD nodes. I have a client who bought similar systems and has added M.2 adapter PCIe AICs to provision M.2 NVMe SSDs for metadata OSDs and WAL+DB offload. Note that there are AFAICT only two *enterprise* M.2 SSDs available: the Micron 7450 PRO and a Samsung model, as this form factor has been more or less superseded by E1.S.
3) Alternative: keep WAL/DB on the HDDs (i.e., do not separate),
The value of offloading WAL+DB depends on your use-case. Ideally I would recommend using QLC SSDs for better performance and density, and in many cases cost, but this seems moot for your deployment.
and use the 3 TB RAID1 SSD area plus the control servers' SSDs (4 x 1.9 TB each) as SSD OSDs for the CephFS metadata pool. Would that be a reasonable approach?
You need the CephFS metadata pool, and ideally the first data pool, on SSDs. You would create a second data pool on the HDD OSDs and use the usual means to define your top level subdirectories to place data there. Skip the RAID. Make the 4x SSDs on the control servers into 4x3=12 OSDs. Which is fewer than I'd recommend for the metadata pool, but you have what you have. Be sure to bump the autoscaler target so you get more PGs on them. Since you have 13x OSD nodes, I'd consider not mirroring the boot SSDs and instead provisioning each as 2x OSDs, assuming that they are equivalent to the SSDs in the control servers, ie. all are NVMe, or all are SATA. I might hesitantly suggest partitioning 1TB of that boot SSD for the OS and use the rest as 2x OSDs as well, *iff* they are NVMe. If they are SAS/SATA, I would not mix OSD and OS. I have a customer who does that, and it has ... not worked out well.
4) If we do SSD-backed metadata OSDs with 3x replication for the metadata pool, we were thinking of CRUSH domains: domain1 = 6 data servers in rack A (6 x 3 TB SSD OSDs), domain2 = 7 data servers in rack B (7 x 3 TB), domain3 = 3 control servers (12 x 1.9 TB SSD OSDs). Any pitfalls with that CRUSH layout?
If you do that, your failure domains will have approximate CRUSH weights of 18, 21, and 22.8. With replication == the number of failure domains, only 18T of the second two will be usable.
Any suggestions would be highly appreciated.
Thank you, Gustavo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, Thank you for your detailed feedback.
I recommend not mixing the OS and data. What SKU are these SSDs? Are they enterprise-class with PLP? If those are SAS/SATA SSDs, conventional wisdom is to not offload more than 4-5 HDD OSD WAL+DBs onto each.
The SSDs are model SSDSC2KB038TZL. I haven't been able to find the datasheet to confirm if they have PLP.
Rightsizing depends in part on your workload. Will you host a modest number of big files, a zillion tiny ones, or a mix?
It's mostly a mix, it will be attached to an HPC cluster with a variety of users.
I suspect with your numbers that the question is moot and you'll have to forego WAL+DB offload.
I tend to agree.
Skip the RAID.
The reason why I have RAID in the data servers is because they have only the 2x SSDs and the 18x HDDs, i.e., there is *no* dedicated M.2 or alternative disk for the OS to be installed on -- overlook when the servers were spec'd, and now we have no possibility of changing anything in these servers (long story). The options that I considered: 1) OS in one of the 3.5 TB SSDs, use the other for DB or as OSD for metadata pool. (In this case if either fails, we lose 18x OSDs, that's why I was leaning for #2 below.) 2) RAID1 both SSDs, use 0.5 TB for the OS, then 3 TB for OSD for metadata pool. (This protects against single SSD failure.) 3) Diskless OS install and use all disks for Ceph. (Our Ceph expertise is not sufficient to attempt this.) So, we are thinking that #2 is the one that offers better reliability in terms of not having multiple single point of failure in the SSDs as RAID1 would protect that. As a side note, the RAID is hardware-managed, Linux sees the RAID volumes as normal disks, e.g., [root@data-02 ~]# lsblk -o WWN,SIZE,TYPE,VENDOR,MODEL | grep RAID 0x600062b101003b803043a322a6ee38d0 550G disk Lenovo RAID 940-32i 8GB 0x600062b101003b803043a3138fd13cab 3T disk Lenovo RAID 940-32i 8GB The controller has power loss protection: [root@data-02 ~]# /opt/MegaRAID/storcli/storcli64 /c0 show all | grep BBU BBU Status = 0 BBU = Yes BBU = Present Cache When BBU Bad = Off
If you do that, your failure domains will have approximate CRUSH weights of 18, 21, and 22.8. With replication == the number of failure domains, only 18T of the second two will be usable.
Thanks for pointing this out! Gustavo
On Nov 7, 2025, at 6:14 PM, Gustavo Garcia Rondina <grondina@uchicago.edu> wrote:
Hi Anthony,
Thank you for your detailed feedback.
It’s what Community Ambassadors do :D
I recommend not mixing the OS and data. What SKU are these SSDs? Are they enterprise-class with PLP? If those are SAS/SATA SSDs, conventional wisdom is to not offload more than 4-5 HDD OSD WAL+DBs onto each.
The SSDs are model SSDSC2KB038TZL.
P4520. Top tier SATA.
I haven't been able to find the datasheet to confirm if they have PLP.
They do for sure. Note that when using these for WAL+DB offload, conventional wisdom is to not exceed 5:1 for SATA SSDs. 36:1 as you propose would likely yield *worse* performance than leaving WAL+DB colocated with the main OSD data HDD. Especially if you have to share with OS and the metadata pool. Conventional wisdom for NVMe SSDs has been at most 10:1, though with PCIe gen 4+ TLC I suspect a higher ratio is ok.
Rightsizing depends in part on your workload. Will you host a modest number of big files, a zillion tiny ones, or a mix?
It's mostly a mix, it will be attached to an HPC cluster with a variety of users.
I might consider constructing your CephFS with the metadata and first data pools on 3x replicated pools constrained to the SSD OSDs and a second data pool added for the HDDs, replicated or EC as you see fit. Then use the layout mechanism to map top level directories to the HDD data pool. If you had more capacity you might create a directory on the SSD first pool for tiny objects but I think you probably can’t afford that. Incent your users to create fewer, larger files, even if they’re tarballs.
I suspect with your numbers that the question is moot and you'll have to forego WAL+DB offload.
I tend to agree.
Ack.
Skip the RAID.
The reason why I have RAID in the data servers is because they have only the 2x SSDs and the 18x HDDs, i.e., there is *no* dedicated M.2 or alternative disk for the OS to be installed on -- overlook when the servers were spec'd, and now we have no possibility of changing anything in these servers (long story).
Procurement can be thorny, so I understand. It’s unfortunate, though. Part of how I reply is to help future Cephers who may come across this thread. We mostly haven't seen SATA SSDs larger than 7.6 T in part due to the SATA interface bottleneck. A better strategy might have been a chassis with some NVMe slots, or AIC M.2 NVMe adapters, though enterprise grade M.2 NVMe SSDs will be difficult to find in the future, and even today are I think limited to 3.84T. Alternately populating some of the SATA slots with SSDs instead of HDDs so that there would be enough for OS, offload, and metadata. Orrrrr NVMe-only chassis with something like the Micron 6500+6550.
The options that I considered:
1) OS in one of the 3.5 TB SSDs, use the other for DB or as OSD for metadata pool. (In this case if either fails, we lose 18x OSDs, that's why I was leaning for #2 below.)
Remember that Ceph does replication for you. This strategy is effectively RAID on top of RAID. It would overload your SATA SSDs, resulting in slow RADOS and MDS ops. Your Prom /Grafana would show false alarms. My Cephalocon presentation from 2024 demonstrates this dynamic, and I’ve seen it on community systems. If mirrored, each of those SSDs would see the writes from 36 OSDs. Which would really be ugly. Put another way, what happens when one of your OSD nodes throws an error or catches fire? You cluster is designed to survive that. In the unlikely event of an SSD failure * you lose half that number of OSDs. If your cluster can’t handle that, you have a deeper problem. * Download SST from Solidigm and ensure the SSDs have up to date firmware. Or if Dell, DSU, etc.
2) RAID1 both SSDs, use 0.5 TB for the OS, then 3 TB for OSD for metadata pool. (This protects against single SSD failure.)
This is sub-optimal either way, but with your constraints, what I would do is mirror ~500 GiB for the OS, with minimal partioning and no swap, and use the balance on *each* SSD for an unmirrored OSD. You can do this with MD or HBA RAID, though I personally do not favor RAID HBAs. They are an expensive hassle, and the money better spent on faster media.
3) Diskless OS install and use all disks for Ceph. (Our Ceph expertise is not sufficient to attempt this.)
Impressive that you mention that. Croit does just this, and I think some VMware deployments. You need rock solid HA DHCP etc, and other aspects make it nontrivial to deploy.
So, we are thinking that #2 is the one that offers better reliability in terms of not having multiple single point of failure in the SSDs as RAID1 would protect that.
I think you’d not have a good experience, see my compromise suggestion above. Remind me how many chassis total? As a certain chassis count, the importance of mirrored boot (and network bonding) diminishes.
As a side note, the RAID is hardware-managed,
Doesn’t have to be though. I’ve experienced many issues with RAID HBAs, including that few deployments monitor them well. This also greatly confounds and complicates fleet management and drive observability.
Linux sees the RAID volumes as normal disks, e.g.,
[root@data-02 ~]# lsblk -o WWN,SIZE,TYPE,VENDOR,MODEL | grep RAID 0x600062b101003b803043a322a6ee38d0 550G disk Lenovo RAID 940-32i 8GB 0x600062b101003b803043a3138fd13cab 3T disk Lenovo RAID 940-32i 8GB
I’m not sure offhand if Lenovo uses LSI or Adaptec HBAs; most chassis vendors rebadge one or the other. It’s likely that you could set them to a passthrough aka HBA aka JBOD mode to expose the raw drives to the OS.
The controller has power loss protection:
[root@data-02 ~]# /opt/MegaRAID/storcli/storcli64 /c0 show all | grep BBU
Ok so that’s an LSI HBA. Aside from Dell’s H740P, most of those have a passthrough mode / personality.
BBU Status = 0 BBU = Yes BBU = Present Cache When BBU Bad = Off
Does it show that the BBU is a supercap vs a battery? I think they don’t use batteries any more, but a supercap is likely rated for all of 3 years. If it fails, you won’t know unless you monitor. The connectors can be finicky, and you don’t even want to know about gas gauge firmware. /c0 /cv show all Might give you more info. I’ve seen people turn this last setting on, which is a rather bad idea. That said, that cache usually is only in effect for VDs, so it isn’t even used for passthrough drives. One can define a single drive RAID 0 VD around each HDD, but split across 36 that doesn’t give you much per drive, and trust me, the operational hassle is substantial. When I see RAID HBAs I either just set passthrough and ignore the RoC, or consider reflashing with IT firmware. The latter is fraught and you’d be on your own.
If you do that, your failure domains will have approximate CRUSH weights of 18, 21, and 22.8. With replication == the number of failure domains, only 18T of the second two will be usable.
Thanks for pointing this out!
Gustavo
Hi Anthony,
They do for sure. Note that when using these for WAL+DB offload, conventional wisdom is to not exceed 5:1 for SATA SSDs. 36:1 as you propose would likely yield *worse* performance than leaving WAL+DB colocated with the main OSD data
Thanks for the info, that's helpful to know.
I might consider constructing your CephFS with the metadata and first data pools on 3x replicated pools constrained to the SSD OSDs and a second data pool added for the HDDs, replicated or EC as you see fit. Then use the layout
I will explore this path.
Remember that Ceph does replication for you. This strategy is effectively RAID on top of RAID. It would overload your SATA SSDs, resulting in slow RADOS and MDS ops. Your Prom /Grafana would show false alarms. My Cephalocon presentation from 2024 demonstrates this dynamic, and I’ve seen it on community systems.
If mirrored, each of those SSDs would see the writes from 36 OSDs. Which would really be ugly.
The main motivation for RAID1 is having more resilience for the OS, so in case one SSD fails, the system stays online.
* Download SST from Solidigm and ensure the SSDs have up to date firmware. Or if Dell, DSU, etc.
Noted, thank you!
what I would do is mirror ~500 GiB for the OS, with minimal partioning and no swap, and use the balance on *each* SSD for an unmirrored OSD.
I tried that with my HBA, but it didn't work, it seems that MD is a better idea anyway. I will go ahead and try that. One thing that popped up in my mind, if I do this, the reamining unused space, e.g., /dev/sdX2 in both disks, will be *partitions*, and not *disks*. Does Ceph have a problem with using partitions for OSDs?
I think you’d not have a good experience, see my compromise suggestion above. Remind me how many chassis total? As a certain chassis count, the importance of mirrored boot (and network bonding) diminishes.
We have 13x chassis. I am already convinced to not have OSDs on VDs :-)
Does it show that the BBU is a supercap vs a battery?
It looks like supercap: Cachevault_Info : =============== -------------------- Property Value -------------------- Type CVPM05 Temperature 24 C State Optimal -------------------- ... GasGaugeStatus : ============== ------------------------------ Property Value ------------------------------ Pack Energy 136 J Capacitance 100 % Remaining Reserve Space 0 ------------------------------
When I see RAID HBAs I either just set passthrough and ignore the RoC
I think I'll do just that and just use MD for the small RAID volume for the OS. Thank you, Gustavo
One thing that popped up in my mind, if I do this, the reamining unused space, e.g., /dev/sdX2 in both disks, will be *partitions*, and not *disks*. Does Ceph have a problem with using partitions for OSDs?
Nope, that should be fine. In fact back in the Filestore days you generally had two partitions.
I think you’d not have a good experience, see my compromise suggestion above. Remind me how many chassis total? As a certain chassis count, the importance of mirrored boot (and network bonding) diminishes.
We have 13x chassis. I am already convinced to not have OSDs on VDs :-)
Having been there, I like to help others avoid the torment ;)
Does it show that the BBU is a supercap vs a battery?
It looks like supercap:
Groovy. Those are usually rated for at least 3 years. There's a handy script here https://github.com/prometheus-community/node-exporter-textfile-collector-scr... for getting StorCLI info into Prometheus. If you have a fleet prom you might add that to your node_exporter deployment. It should be possible to give the bundled Ceph-managed node_exporter additional config to pick this up, but I haven't done that myself.
Cachevault_Info : ===============
-------------------- Property Value -------------------- Type CVPM05 Temperature 24 C State Optimal --------------------
...
GasGaugeStatus : ==============
------------------------------ Property Value ------------------------------ Pack Energy 136 J Capacitance 100 % Remaining Reserve Space 0 ------------------------------
When I see RAID HBAs I either just set passthrough and ignore the RoC
I think I'll do just that and just use MD for the small RAID volume for the OS.
Given your constraints, I think that's the best strategy. The only thing that MD for the OS won't give you is automatically handling the loss of the first boot drive across a power cycle, but I personally haven't found that to be critical.
Thank you, Gustavo
participants (2)
-
Anthony D'Atri
-
Gustavo Garcia Rondina