...

A subsequent question is: is it transparent to the point I can setup the cache on a live OSD, or do I need to recreate it and have ceph-volume have a go *with* the cache setup, so it knows about it? The way I understand it is "ceph does not care about dm-cache" so I would need, at worst, to restart the OSD.

disclaimer: never used dm-cache in production, last experimented with it several years ago.

I seem to remember that the transparency of dm-cache comes with a risk: LVM device name stays the same whether the cache device is attached or not. Nothing keeps you from writing new data to the device with cache device detached even if said caching device still contains some unwritten data. So you might lose that in-cache data using LVM volume when cache device is not attached (e.g. after reboot, perhaps ceph grabbing the device before cache device attach). And when re-attaching a cache device later it might just kill the fs by replaying outdated cached data.

I have used bcache in production with ceph. Bcache requires you to format the block device as a bcache device, and the backing device knows if cached data is missing and refuses to be mounted until the corresponding cache device is attached and such cached data is accessible. This provides better reliabilty, but with some upfront cost and some additional parameters to set (cache mode, watermarks, ...)

Still, this might just be outdated knowledge of the past, YMMV.

Ah, this somewhat changes things. It appears your knowledge is is not outdated, as it is explicitly documented in the lvm-cache manpage (https://man7.org/linux/man-pages/man7/lvmcache.7.html):

A second cache mode is "writeback". Writeback delays writing data blocks from the cache back to the origin LV. This mode will increase performance, but the loss of a cache device can result in lost data.

So, so much for that. Writethrough will not help my situation, and writeback can cause data loss. This leads to two questions: Since Ceph is aware of dm-cache (inferred by the fact the docs mention it), will the OSD stop if the cache device is not present? My setup has WAL+DB on dedicated drives, and I planned to use the free space left on those drives as a cache. Since the OSD cannot work if WAL+DB absent, I'm in a situation where the cache can only be absent if the WAL+DB is absent too. That is, WAL+DB is here iff cache is here too. Is the reasoning correct? Or can there still be a race where OSD starts writing data to the volume before the kernel has caching setup? Thanks for the warning! Alexis

Matthias

...

...

...

Is it that I can just slap an lvm cache on the LVs created by ceph-volume and call it a day? Is this the canonical replacement to Ceph's native cache tiers?

I guess it is one of the default alternatives now that tiering is off the map. We do a lot of "put WAL + DB on nvme partition, data on spin-disks" with something like 4 or 6 nvme partitions per OSD disk, so this mostly improves database access parts and less of the bulk data transfers, but we think that is ok.

Yeah, HDD setups is what I tried to mimic, as I have the WAL+DB on actual write-intensive enterprise drives, but it can only do so much when the backend drive does GC pauses that last for 20 min and increase the IO latency to the 2s-20s range. I'm hoping caching can shield most workloads from it.

...

...

Is the tuning of the cache settings somehow managed by Ceph or is it fully on the user? And, related, can the caching be declared in the orch OSD spec?

If you stack devices in LVM, then I guess that is where any tuning would have to happen, though I don't think there is a large need to tune those settings a lot. The defaults should probably work fine for most people.

Noted. I don't expect to have to tune a lot, but given the (horrible) behavior of my drives, being able to tune flushes and invalidations is important.

...

...

As for the context: I have a cluster with *very* bad SSDs (consumer DRAM-less BX500) and I'm trying to get it somewhat OK until the market for flash improves. As the problems are due to GC pauses from the drive caused by the sustained IO, I'm hoping that putting a cache tier (with smaller faster enterprise SSDs) in front of the drives will change the IO behavior enough for the drives to perform their GC without impacting the cluster. I already have the good smaller drives that have WAL+DB for the slow drives, but they have enough free space to cache the block.

I don't have any idea on which method will be best in the end, but I think dm-cache is stable enough for people to recommend it.

That's what I gathered from poking around the internet. The fact that the ceph documentations mention it, while not an endorsement, is already meaningful for the viability of dm-cache. Thanks for the confirmation!

Thank you for the reply and the pointers!

...

-- May the most significant bit of your life be positive.

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io