Hi, I confirm you can setup tiering between two differents S3 pools on the same ceph cluster with squid. You have to create 2 differents pools and placement groups relying on your two type of physical storage drives. You may use the same RGW client and use the S3 storage class attribute or you may also create a new RGW client dedicated to each pool. I use S3 bucket policy for this and you should prefer using standard AWS names of storage class ("GLACIER" or "STANDARD_IA"...) to do this easily using the aws cli. On 4/29/26 14:18, Anthony D'Atri via ceph-users wrote:
Is it possible to use RGW lifecycle policies to automatically move objects between different storage tiers for example, from a hot tier (NVMe) to a cold tier (HDD) based on inactivity over a period of time? Absolutely. This is one key application of S3 storage classes. Note that LC policies are run once a day by default, so with a heavy load you may wish to deploy a set of RGWs that are not user-facing, but which only handle LC tasks, and disable LC processing for the user-facing cohort of daemons.
Additionally, if those objects become active again (i.e., are accessed frequently after being moved to the cold tier), is there a mechanism to automatically move them back to the original hot tier (NVMe)? But of course! Transparent tiering is available from Squid, at least when offloading cold objects to a different endpoint. I'm not positive that this works natively within a single cluster, but it may well. You could though most likely still use LC policies, albeit in a less elegant fashion.
I may be misunderstanding how lifecycle management works in RGW, so please correct me if that’s the case. If this type of tiering is supported, could you point me toward the recommended approach or configuration? The following awesome blog posts from Daniel Parkes will guide you. When deploying a cold tier, it is ideal to leave the first STANDARD storage class as the default so that HEAD objects are always on faster storage. They don't take much space, but this accelerates various operations.
https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/
Depending on your object sizes and workload, consider replication vs erasure coding. When using EC for your slower -- or even faster -- storage class, be sure to select a value of m >= 2. Profiles wider than 8+3 may increase space amp and cost write, scrub, and recovery performance. Always choose (k+m < the number of CRUSH failure domains.
Writes to a replicated pool will generally be faster, but EC makes more efficient use of underlying storage.
This is nuanced, though for the smallest objects. Check out Mark Nelson's most excellent spreadsheet that shows this visually:
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP...
Through the Squid release, RADOS always writes full stripes, so if you are using EC, the very smallest objects will consume more underlying raw storage than one might think. The wider the EC profile, the more pronounced this effect. If you have a large proportion of tiny objects, say below 256 - 512 KiB, this space amplification may be significant. If the number of very small objects is dwarfed by the number * size of larger objects, this is amortized into the noise floor. If not, you can consider client headers or Lua scripting to assort objects to appropriate storage classes at ingest. Check out these presentations by Frederic Nass and Steve U:
https://ceph.io/en/news/blog/2024/auto-tiering-ceph-object-storage-part-1/ https://www.youtube.com/watch?v=qKd2Po2hTe4
Note that with the Tentacle release, bucket data pools with the EC optimizations flag significantly reduce this space amplification, lowering the size threshold for special handling and reducing the need to worry about it.
https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io