Den tors 9 apr. 2026 kl 15:58 skrev Alexis Bauvin via ceph-users <ceph-users@ceph.io>:
Since a few versions, Ceph has deprecated caching tiers with no alternative mentioned. Thus I've been looking and somewhat settled on using dm-cache for two reasons: it's LVM and ceph-volume mentions it's supported and "transparent":
For ceph-volume, the use of dm-cache is transparent, there is no difference for the tool, and it treats dm-cache like a plain logical volume.
However, what does this really mean?
It means you do it disk-for-disk instead of a fast pool in front of a slow pool. In the end, the same results should be there, faster accesses for writes and recently accessed data.
Is it that I can just slap an lvm cache on the LVs created by ceph-volume and call it a day? Is this the canonical replacement to Ceph's native cache tiers?
I guess it is one of the default alternatives now that tiering is off the map. We do a lot of "put WAL + DB on nvme partition, data on spin-disks" with something like 4 or 6 nvme partitions per OSD disk, so this mostly improves database access parts and less of the bulk data transfers, but we think that is ok.
Is the tuning of the cache settings somehow managed by Ceph or is it fully on the user? And, related, can the caching be declared in the orch OSD spec?
If you stack devices in LVM, then I guess that is where any tuning would have to happen, though I don't think there is a large need to tune those settings a lot. The defaults should probably work fine for most people.
As for the context: I have a cluster with *very* bad SSDs (consumer DRAM-less BX500) and I'm trying to get it somewhat OK until the market for flash improves. As the problems are due to GC pauses from the drive caused by the sustained IO, I'm hoping that putting a cache tier (with smaller faster enterprise SSDs) in front of the drives will change the IO behavior enough for the drives to perform their GC without impacting the cluster. I already have the good smaller drives that have WAL+DB for the slow drives, but they have enough free space to cache the block.
I don't have any idea on which method will be best in the end, but I think dm-cache is stable enough for people to recommend it. -- May the most significant bit of your life be positive.
Le 09/04/2026 16:44 CEST, Janne Johansson <icepic.dz@gmail.com> a écrit :
Den tors 9 apr. 2026 kl 15:58 skrev Alexis Bauvin via ceph-users <ceph-users@ceph.io>:
Since a few versions, Ceph has deprecated caching tiers with no alternative mentioned. Thus I've been looking and somewhat settled on using dm-cache for two reasons: it's LVM and ceph-volume mentions it's supported and "transparent":
For ceph-volume, the use of dm-cache is transparent, there is no difference for the tool, and it treats dm-cache like a plain logical volume.
However, what does this really mean?
It means you do it disk-for-disk instead of a fast pool in front of a slow pool. In the end, the same results should be there, faster accesses for writes and recently accessed data.
Oh, that makes sense. Thanks for the clarification, and it explains a lot. A subsequent question is: is it transparent to the point I can setup the cache on a live OSD, or do I need to recreate it and have ceph-volume have a go *with* the cache setup, so it knows about it? The way I understand it is "ceph does not care about dm-cache" so I would need, at worst, to restart the OSD.
Is it that I can just slap an lvm cache on the LVs created by ceph-volume and call it a day? Is this the canonical replacement to Ceph's native cache tiers?
I guess it is one of the default alternatives now that tiering is off the map. We do a lot of "put WAL + DB on nvme partition, data on spin-disks" with something like 4 or 6 nvme partitions per OSD disk, so this mostly improves database access parts and less of the bulk data transfers, but we think that is ok.
Yeah, HDD setups is what I tried to mimic, as I have the WAL+DB on actual write-intensive enterprise drives, but it can only do so much when the backend drive does GC pauses that last for 20 min and increase the IO latency to the 2s-20s range. I'm hoping caching can shield most workloads from it.
Is the tuning of the cache settings somehow managed by Ceph or is it fully on the user? And, related, can the caching be declared in the orch OSD spec?
If you stack devices in LVM, then I guess that is where any tuning would have to happen, though I don't think there is a large need to tune those settings a lot. The defaults should probably work fine for most people.
Noted. I don't expect to have to tune a lot, but given the (horrible) behavior of my drives, being able to tune flushes and invalidations is important.
As for the context: I have a cluster with *very* bad SSDs (consumer DRAM-less BX500) and I'm trying to get it somewhat OK until the market for flash improves. As the problems are due to GC pauses from the drive caused by the sustained IO, I'm hoping that putting a cache tier (with smaller faster enterprise SSDs) in front of the drives will change the IO behavior enough for the drives to perform their GC without impacting the cluster. I already have the good smaller drives that have WAL+DB for the slow drives, but they have enough free space to cache the block.
I don't have any idea on which method will be best in the end, but I think dm-cache is stable enough for people to recommend it.
That's what I gathered from poking around the internet. The fact that the ceph documentations mention it, while not an endorsement, is already meaningful for the viability of dm-cache. Thanks for the confirmation! Thank you for the reply and the pointers!
-- May the most significant bit of your life be positive.
participants (2)
-
Alexis Bauvin
-
Janne Johansson