Ceph S3 Performance: HDDs for data + SSDs for metadata – Best practice?
Hi everyone, I'm currently planning a Ceph-based S3 storage backend, with a strong focus on balancing performance and cost (EUR per TB). Fully SSD-based setups seem too expensive for my use case, while HDD-only configurations might be too slow. My current idea is to use a hybrid approach: - HDDs for bluestore_block (bulk data) - Enterprise SSDs for bluestore_block.db and bluestore_block.wal (metadata and journal) The main client of this S3 backend will be Veeam, which, as far as I can tell, uses larger block sizes. Since this results in fewer random writes and mostly sequential I/O, I assume this kind of workload can work well with HDDs — as long as metadata is fast (hence SSDs for DB/WAL). Am I on the right track with this assumption? - Do you have any benchmark results (real-world or synthetic) comparing HDD-only vs Hybrid setups for object workloads like Veeam? That would help a lot to quantify the performance difference. - What SSD-to-HDD ratio would you recommend? (I’ve seen 1 SSD per 3–5 HDDs) Any advice or confirmation from those running similar setups would be much appreciated! So far i found: https://www.hyperscalers.com/image/catalog/00-Products/Storage%20Servers/Whi... => which mentions Ceph Verstion 0.94.5 and the performance boot does not seem too big here And here: https://www.ambedded.com.tw/en/use-case/use-case-08.html they dont mention such a Hybrid Setup Thanks a lot in advance! Best regards, Michael
Hi Michael, So challenges with Veeam backups are more around metadata. I do hope when you said enterprise SSD’s you meant NVME’s? By default Veeam will write everything into one bucket. The problem with this depending on the size of your environment is you will get into bucket sharding issues. You can also get issues with large OMAP objects. What happens is a bucket will get sharded when it gets to a certain number of objects. If you know how many objects you are likely to be storing then you can pre-shard a bucket for this larger number of objects to help prevent this issue. Newer versions of Veeam do allow for using multiple buckets at the backend which I would strongly suggest you use. The other large hit you get from Veeam is if you are using Object lock then you get lots of put object retention metadata changes which can generate a significant amount of metadata traffic. I would strongly suggest you put in some additional NVME’s to allow you to move the S3 metadata onto these devices to allow for better performance. Darren
On 4 Jun 2025, at 08:28, Ml Ml <mliebherr99@googlemail.com> wrote:
Hi everyone,
I'm currently planning a Ceph-based S3 storage backend, with a strong focus on balancing performance and cost (EUR per TB). Fully SSD-based setups seem too expensive for my use case, while HDD-only configurations might be too slow.
My current idea is to use a hybrid approach: - HDDs for bluestore_block (bulk data) - Enterprise SSDs for bluestore_block.db and bluestore_block.wal (metadata and journal)
The main client of this S3 backend will be Veeam, which, as far as I can tell, uses larger block sizes. Since this results in fewer random writes and mostly sequential I/O, I assume this kind of workload can work well with HDDs — as long as metadata is fast (hence SSDs for DB/WAL).
Am I on the right track with this assumption?
- Do you have any benchmark results (real-world or synthetic) comparing HDD-only vs Hybrid setups for object workloads like Veeam? That would help a lot to quantify the performance difference. - What SSD-to-HDD ratio would you recommend? (I’ve seen 1 SSD per 3–5 HDDs)
Any advice or confirmation from those running similar setups would be much appreciated!
So far i found: https://www.hyperscalers.com/image/catalog/00-Products/Storage%20Servers/Whi... => which mentions Ceph Verstion 0.94.5 and the performance boot does not seem too big here
And here: https://www.ambedded.com.tw/en/use-case/use-case-08.html they dont mention such a Hybrid Setup
Thanks a lot in advance!
Best regards, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So this is how we are doing it and it is working very well: - Put the index and metadata pools on fast NVMEs. The usage of our flash disks is at 1% disk space. - Only have the data pool on spinners - For every 5 HDDs we have 1 NVME that is serving the OSD metadata. - After the reef upgrade we enable autosharding, because we run a multisite setup - Having a bucket with 503 (prime number) shards is not a problem and we scaled one bucket from 11 shards to 401 without issue Am Mi., 4. Juni 2025 um 12:43 Uhr schrieb <darren@soothill.com>:
Hi Michael,
So challenges with Veeam backups are more around metadata.
I do hope when you said enterprise SSD’s you meant NVME’s?
By default Veeam will write everything into one bucket. The problem with this depending on the size of your environment is you will get into bucket sharding issues. You can also get issues with large OMAP objects.
What happens is a bucket will get sharded when it gets to a certain number of objects. If you know how many objects you are likely to be storing then you can pre-shard a bucket for this larger number of objects to help prevent this issue.
Newer versions of Veeam do allow for using multiple buckets at the backend which I would strongly suggest you use.
The other large hit you get from Veeam is if you are using Object lock then you get lots of put object retention metadata changes which can generate a significant amount of metadata traffic.
I would strongly suggest you put in some additional NVME’s to allow you to move the S3 metadata onto these devices to allow for better performance.
Darren
On 4 Jun 2025, at 08:28, Ml Ml <mliebherr99@googlemail.com> wrote:
Hi everyone,
I'm currently planning a Ceph-based S3 storage backend, with a strong focus on balancing performance and cost (EUR per TB). Fully SSD-based setups seem too expensive for my use case, while HDD-only configurations might be too slow.
My current idea is to use a hybrid approach: - HDDs for bluestore_block (bulk data) - Enterprise SSDs for bluestore_block.db and bluestore_block.wal (metadata and journal)
The main client of this S3 backend will be Veeam, which, as far as I can tell, uses larger block sizes. Since this results in fewer random writes and mostly sequential I/O, I assume this kind of workload can work well with HDDs — as long as metadata is fast (hence SSDs for DB/WAL).
Am I on the right track with this assumption?
- Do you have any benchmark results (real-world or synthetic) comparing HDD-only vs Hybrid setups for object workloads like Veeam? That would help a lot to quantify the performance difference. - What SSD-to-HDD ratio would you recommend? (I’ve seen 1 SSD per 3–5 HDDs)
Any advice or confirmation from those running similar setups would be much appreciated!
So far i found:
https://www.hyperscalers.com/image/catalog/00-Products/Storage%20Servers/Whi...
=> which mentions Ceph Verstion 0.94.5 and the performance boot does not seem too big here
And here: https://www.ambedded.com.tw/en/use-case/use-case-08.html they dont mention such a Hybrid Setup
Thanks a lot in advance!
Best regards, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi Michael,
So challenges with Veeam backups are more around metadata.
It’s conventional wisdom that Veeam uses tiny blocks (S3 objects) by default and that one should enable a “large block” setting.
I do hope when you said enterprise SSD’s you meant NVME’s?
SAS and SATA SSDs cost the same as NVMe SSDs with careful procurement, but they aren’t necessarily fatal. Remember that NVMe devices *are* SSDs. No CRTs or gamma ferric oxide in play.
By default Veeam will write everything into one bucket. The problem with this depending on the size of your environment is you will get into bucket sharding issues. You can also get issues with large OMAP objects.
In a metadata-heavy deployment with a relatively small number of large SSDs, it would seem advantageous to still split the drives into 2-3 OSDs just to get more RocksDB instances across which to spread omaps. And to avoid versioning when at all possible. And to use a recent Ceph release, Quincy or later IIRC, that shards RocksDB across column families, with legacy OSDs retrofitted via manual action. Recently implemented RocksDB compression should help too.
Newer versions of Veeam do allow for using multiple buckets at the backend which I would strongly suggest you use.
Agreed. Outside buckets can be difficult.
I would strongly suggest you put in some additional NVME’s to allow you to move the S3 metadata onto these devices to allow for better performance.
The OP described that already:
- Enterprise SSDs for bluestore_block.db and bluestore_block.wal (metadata and journal)
Remember that omaps are (currently) stored in RocksDB. This approach can actually be advantageous for RGW metadata since it spreads the bucket indexes across a much larger number of RocksDB instances.
I'm currently planning a Ceph-based S3 storage backend, with a strong focus on balancing performance and cost (EUR per TB).
Plug your numbers into the SNIA TCO calculator. https://www.snia.org/forums/cmsi/programs/TCOcalc
Fully SSD-based setups seem too expensive for my use case
The TCO might surprise you. Consider having to procure, manage, and house a greater number of chassis. RUs for most of us aren’t free, nor or TORs. A chassis built to handle LFF HDDs will usually be taller than an SFF/EDSFF SSD-only server, and you’ll likely be fussing with brittle adapters to stick a small SSD into that big, space-wasting slot. Toploaders can suffer from HBA and backplane saturation, not to mention a thundering herd when recovery happens. Plan for peak usage. Don’t just look at the unit economics of the drive, look at the holistic economics of the *drive bay*, i.e. everything else. Today’s 30TB spinners (be sure to avoid SMR!) use the same tired interface as 3TB drives a dozen years ago. Bottleneck city. If you spent more on SAS spinners, that’s money better spent on modern media, especially since you will need an AIC HBA, which most likely will be RAID-enabled, with FBWC and a BBU. These can easily have a list price of USD 2000. Universal drive bays cost more too. SAS is distinctly sunsetting in the enterprise SSD market, and SATA isn’t far beyond. Consider that for the — let’s be realistic — 10 year lifetime of servers purchased today. Be careful of the trap of deploying a small number of ultradense toploaders. And don’t make the mistake of an overly wide EC profile. 8+3 means that every write ties up 11 drives, and you want at lest 12 failure domains. This affects recovery and scrubs. A deployment that does not meet your needs is a false economy. A deployment with 30-122TB QLC SSDs will be rather more RU-dense, saving on DC spend, and much better able to handle small object hotspots and the increased demands of recovery. If a failed drive takes you two weeks to recover from, that’s two weeks of increased risk of data unavailability or loss.
, while HDD-only configurations might be too slow.
Generally, yes. Especially if you have lots of small S3 objects. And performance when you test with 1% full can be very different from down the road when you’re 60% full and fragmented.
My current idea is to use a hybrid approach: - HDDs for bluestore_block (bulk data) - Enterprise SSDs for bluestore_block.db and bluestore_block.wal (metadata and journal)
Not exactly a journal, which is why it isn’t called one, but that’s a nit.
The main client of this S3 backend will be Veeam, which, as far as I can tell, uses larger block sizes.
My understanding is that it needs to be configured to.
Since this results in fewer random writes and mostly sequential I/O,
Ceph OSDs usually experience random IO regardless, the IO blender effect. Unless maybe if you only have a single client at a time.
I assume this kind of workload can work well with HDDs — as long as metadata is fast (hence SSDs for DB/WAL).
Remember that hybrid OSDs may increase metadata performance and latency for small writes, but not necessarily throughput.
- What SSD-to-HDD ratio would you recommend? (I’ve seen 1 SSD per 3–5 HDDs)
Conventional wisdom is at most 5 per SAS/SATA SSD, 10 per NVMe SSD.
https://www.hyperscalers.com/image/catalog/00-Products/Storage%20Servers/Whi...
=> which mentions Ceph Verstion 0.94.5 and the performance boot does
Copyright 2015. I think I saw QCT present this at a Ceph Day back in the mists of time. Hammer does not in any way reflect the modern experience. Filestore OSDs, no Manager factored out of the mons yet, etc.
not seem too big here
And here: https://www.ambedded.com.tw/en/use-case/use-case-08.html they dont mention such a Hybrid Setup
It’s …novel.
Thanks a lot in advance!
Best regards, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Boris
-
darren＠soothill.com
-
Ml Ml