Re: Questions about dm-cache
On that we agree: WAL+DB dead is a dead OSD.
I've seen a speculative outline on recovering the OSD but have not seen someone try.
And it's indeed a fair tradeoff: I'd expect the bad backend drives to die long before
Spinners often have a lower TBW and AFR than even low-endurance SSDs.
the enterprise write-intensive drives
Do you find that you strictly need high endurance SSDs? 99% of enterprise SSDs never use more than 15% of their rated endurance. These days write-intensive usually means Optane-replacement SLC. More common here is mixed-use, which is typically the very same hardware as a "read intensive" SSD just with more overprovisioning. If you really need more endurance than comes out of the box with an RI device, you can often adjust the OP yourself at much lower cost.
I have for wal+db. (or, in the case of spinning rust, I'd expect a mechanical failure long before a flash failure).
Most SSD failures are firmware and can be fixed in-situ. I've had to replace 4000 HDDs with a design flaw - scattered around the world. Nightmare
Thus, in the end, it's no different from a dedicated wal+db iff wal+db stays dedicated: I've seen people suggest to keep wal+db colocated if there's a fast flash in front of the drive, as wal+db would essentially always be hot in the cache.
Indeed I've read studies that claim a superset of benefits from a transparent cache compared to just WAL+DB.
(note: I know that the OSD thinks it directly talks to the crap drives,
Reads are always served from the slow device, and writes over a certain size go there directly. I've seen use-cases where the hassle of WAL+DB offload doesn't actually make an appreciable improvement.
Le 10/04/2026 15:19 CEST, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
On that we agree: WAL+DB dead is a dead OSD.
I've seen a speculative outline on recovering the OSD but have not seen someone try.
Which means, for all intents and purposes, no block.db = stopped osd = no more write to block = no possibility of data loss.
And it's indeed a fair tradeoff: I'd expect the bad backend drives to die long before
Spinners often have a lower TBW and AFR than even low-endurance SSDs.
the enterprise write-intensive drives
Do you find that you strictly need high endurance SSDs? 99% of enterprise SSDs never use more than 15% of their rated endurance. These days write-intensive usually means Optane-replacement SLC. More common here is mixed-use, which is typically the very same hardware as a "read intensive" SSD just with more overprovisioning. If you really need more endurance than comes out of the box with an RI device, you can often adjust the OP yourself at much lower cost.
Oh I don't need the endurance. Even the IOPS I don't need. But given the shitshow the drives I have are, I wanted to make super duper sure the drives I got for wal+db could never ever be blamed. Even at the current market price, It ended up costing ~€700 for 8x400GB of enterprise write-intensive drives, which was marginally more expensive than the MU or RI drives (like less than €100 in total), so I'd say it's worth it. It's now the price brokers ask for a *single* enterprise 1.92TB drive. At least I know I cannot blame the hardware. And since I'm buying refurb, I reasoned WI drives would have more life left in them from the previous owner(s), as the overcommit is stupidly high on those (combined with SLC).
I have for wal+db. (or, in the case of spinning rust, I'd expect a mechanical failure long before a flash failure).
Most SSD failures are firmware and can be fixed in-situ. I've had to replace 4000 HDDs with a design flaw - scattered around the world. Nightmare
Thus, in the end, it's no different from a dedicated wal+db iff wal+db stays dedicated: I've seen people suggest to keep wal+db colocated if there's a fast flash in front of the drive, as wal+db would essentially always be hot in the cache.
Indeed I've read studies that claim a superset of benefits from a transparent cache compared to just WAL+DB.
I can definitely see why. The point however, was that having wal+db dedicated meant osd would stop using the drive if the fast one died. In terms of performance I'd expect no worse than with a cached colocated wal+db. However, we've now established that, for wal+db to be transparently cached, we'd need bcache to avoid data loss. dm-cache needs a way to block osd from using the drive if the cache is missing.
(note: I know that the OSD thinks it directly talks to the crap drives,
Reads are always served from the slow device, and writes over a certain size go there directly.
My understanding from dm-cache docs was that no, in plain dm-cache, reads hit the cache drive. For the behavior you describe (only writes use the cache), you'd need dm-writecache, which is another mode for dm-cache. As for the write size, I guess it's configurable, but I don't think it'll be an issue for my case: what I'm after is preventing the constant chatter or small random writes that hit the drive with a sustained workload. The occasional big write should not matter too much.
I've seen use-cases where the hassle of WAL+DB offload doesn't actually make an appreciable improvement.
I guess it really depends on the drive you have. For HDDs it's a no brainer as far as I understand it, but for SSDs you'd hit diminishing returns, even with NVMe WAL+DB. It's a case where transparent caching of block with an NVMe would make an improvement where WAL+DB does not. Thanks for all the info! Alexis
,
Reads are always served from the slow device, and writes over a certain size go there directly.
My understanding from dm-cache docs was that no, in plain dm-cache, reads hit the cache drive. For the behavior you describe (only writes use the cache), you'd need dm-writecache, which is another mode for dm-cache.
Clarify: I was talking about conventional offloaded WAL+DB, not dm-cache.
As for the write size, I guess it's configurable, but I don't think it'll be an issue for my case: what I'm after is preventing the constant chatter or small random writes that hit the drive with a sustained workload. The occasional big write should not matter too much.
That’s what I was after. One potential strategy is fronting coarse-IU QLC SSDs, in which case one would reeeealy want the writes to the backing device to be an aligned, even multiple of the IU.
I've seen use-cases where the hassle of WAL+DB offload doesn't actually make an appreciable improvement.
I guess it really depends on the drive you have. For HDDs it's a no brainer as far as I understand it,
I’ve worked with an HDD cluster that added colocated HDDs to existed offloaded HDDs. 40% of the OSDs were colocated and nobody noticed. There are no doubt workloads where it would make a bigger difference.
but for SSDs you'd hit diminishing returns, even with NVMe WAL+DB.
Absolutely. And new spend on SAS / SATA SSDs these days is a bad idea. They are disappearing from the market.
It's a case where transparent caching of block with an NVMe would make an improvement where WAL+DB does not Yep
Alexis
Le 11/04/2026 01:46 CEST, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
,
Reads are always served from the slow device, and writes over a certain size go there directly.
My understanding from dm-cache docs was that no, in plain dm-cache, reads hit the cache drive. For the behavior you describe (only writes use the cache), you'd need dm-writecache, which is another mode for dm-cache.
Clarify: I was talking about conventional offloaded WAL+DB, not dm-cache.
Ah, indeed, my bad. We have the same understanding! I even tried to improve the situation by increasing the deferred size, which helped a tiny bit. Tiny.
As for the write size, I guess it's configurable, but I don't think it'll be an issue for my case: what I'm after is preventing the constant chatter or small random writes that hit the drive with a sustained workload. The occasional big write should not matter too much.
That’s what I was after. One potential strategy is fronting coarse-IU QLC SSDs, in which case one would reeeealy want the writes to the backing device to be an aligned, even multiple of the IU.
Which is why I have high hopes for dm-writecache in the long run. Hit high watermark, flush data in a as-much sequential and coalesced manner possible, then let the drive rest. And actually, to come back to my original question: I finally made the jump today, and added ~110GB of dm-writecache in front of the backing block devices, keeping WAL+DB dedicated and on the same drive (so that the cache can never be absent for an alive OSD, to be safe, treating "transparent" as "I'm ignoring it"). I'm happy to report that, indeed, Ceph did not bat an eye and keeps chugging along. Except that now, practically no IO hits the backing crappy drive, and the cluster has the performance of those good drives (well, as long as the working data set does not get flushed, but my workloads will make it true virtually always). Thanks to everybody that took part in the discussion, and answered the questions of both a Ceph newbie and LVM newbie!
I've seen use-cases where the hassle of WAL+DB offload doesn't actually make an appreciable improvement.
I guess it really depends on the drive you have. For HDDs it's a no brainer as far as I understand it,
I’ve worked with an HDD cluster that added colocated HDDs to existed offloaded HDDs. 40% of the OSDs were colocated and nobody noticed. There are no doubt workloads where it would make a bigger difference.
Rough haha, so much for the money in the dedicated drives.
but for SSDs you'd hit diminishing returns, even with NVMe WAL+DB.
Absolutely. And new spend on SAS / SATA SSDs these days is a bad idea. They are disappearing from the market.
Good thing I'm buying refurb then :)
It's a case where transparent caching of block with an NVMe would make an improvement where WAL+DB does not Yep
Alexis
participants (2)
-
Alexis Bauvin
-
Anthony D'Atri