On Thu, Apr 09, 2026 at 03:22:00PM +0000, Alexis Bauvin via ceph-users wrote:

...

Le 09/04/2026 16:44 CEST, Janne Johansson <icepic.dz@gmail.com> a écrit :

Den tors 9 apr. 2026 kl 15:58 skrev Alexis Bauvin via ceph-users <ceph-users@ceph.io>:

...

Since a few versions, Ceph has deprecated caching tiers with no alternative mentioned. Thus I've been looking and somewhat settled on using dm-cache for two reasons: it's LVM and ceph-volume mentions it's supported and "transparent":

...

For ceph-volume, the use of dm-cache is transparent, there is no difference for the tool, and it treats dm-cache like a plain logical volume.

However, what does this really mean?

It means you do it disk-for-disk instead of a fast pool in front of a slow pool. In the end, the same results should be there, faster accesses for writes and recently accessed data.

Oh, that makes sense. Thanks for the clarification, and it explains a lot.

A subsequent question is: is it transparent to the point I can setup the cache on a live OSD, or do I need to recreate it and have ceph-volume have a go *with* the cache setup, so it knows about it? The way I understand it is "ceph does not care about dm-cache" so I would need, at worst, to restart the OSD.

disclaimer: never used dm-cache in production, last experimented with it several years ago. I seem to remember that the transparency of dm-cache comes with a risk: LVM device name stays the same whether the cache device is attached or not. Nothing keeps you from writing new data to the device with cache device detached even if said caching device still contains some unwritten data. So you might lose that in-cache data using LVM volume when cache device is not attached (e.g. after reboot, perhaps ceph grabbing the device before cache device attach). And when re-attaching a cache device later it might just kill the fs by replaying outdated cached data. I have used bcache in production with ceph. Bcache requires you to format the block device as a bcache device, and the backing device knows if cached data is missing and refuses to be mounted until the corresponding cache device is attached and such cached data is accessible. This provides better reliabilty, but with some upfront cost and some additional parameters to set (cache mode, watermarks, ...) Still, this might just be outdated knowledge of the past, YMMV. Matthias

...

...

Is it that I can just slap an lvm cache on the LVs created by ceph-volume and call it a day? Is this the canonical replacement to Ceph's native cache tiers?

I guess it is one of the default alternatives now that tiering is off the map. We do a lot of "put WAL + DB on nvme partition, data on spin-disks" with something like 4 or 6 nvme partitions per OSD disk, so this mostly improves database access parts and less of the bulk data transfers, but we think that is ok.

Yeah, HDD setups is what I tried to mimic, as I have the WAL+DB on actual write-intensive enterprise drives, but it can only do so much when the backend drive does GC pauses that last for 20 min and increase the IO latency to the 2s-20s range. I'm hoping caching can shield most workloads from it.

...

...

Is the tuning of the cache settings somehow managed by Ceph or is it fully on the user? And, related, can the caching be declared in the orch OSD spec?

If you stack devices in LVM, then I guess that is where any tuning would have to happen, though I don't think there is a large need to tune those settings a lot. The defaults should probably work fine for most people.

Noted. I don't expect to have to tune a lot, but given the (horrible) behavior of my drives, being able to tune flushes and invalidations is important.

...

...

As for the context: I have a cluster with *very* bad SSDs (consumer DRAM-less BX500) and I'm trying to get it somewhat OK until the market for flash improves. As the problems are due to GC pauses from the drive caused by the sustained IO, I'm hoping that putting a cache tier (with smaller faster enterprise SSDs) in front of the drives will change the IO behavior enough for the drives to perform their GC without impacting the cluster. I already have the good smaller drives that have WAL+DB for the slow drives, but they have enough free space to cache the block.

I don't have any idea on which method will be best in the end, but I think dm-cache is stable enough for people to recommend it.

That's what I gathered from poking around the internet. The fact that the ceph documentations mention it, while not an endorsement, is already meaningful for the viability of dm-cache. Thanks for the confirmation!

Thank you for the reply and the pointers!

...

-- May the most significant bit of your life be positive.

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io