Le 10/04/2026 15:19 CEST, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
On that we agree: WAL+DB dead is a dead OSD.
I've seen a speculative outline on recovering the OSD but have not seen someone try.
Which means, for all intents and purposes, no block.db = stopped osd = no more write to block = no possibility of data loss.
And it's indeed a fair tradeoff: I'd expect the bad backend drives to die long before
Spinners often have a lower TBW and AFR than even low-endurance SSDs.
the enterprise write-intensive drives
Do you find that you strictly need high endurance SSDs? 99% of enterprise SSDs never use more than 15% of their rated endurance. These days write-intensive usually means Optane-replacement SLC. More common here is mixed-use, which is typically the very same hardware as a "read intensive" SSD just with more overprovisioning. If you really need more endurance than comes out of the box with an RI device, you can often adjust the OP yourself at much lower cost.
Oh I don't need the endurance. Even the IOPS I don't need. But given the shitshow the drives I have are, I wanted to make super duper sure the drives I got for wal+db could never ever be blamed. Even at the current market price, It ended up costing ~€700 for 8x400GB of enterprise write-intensive drives, which was marginally more expensive than the MU or RI drives (like less than €100 in total), so I'd say it's worth it. It's now the price brokers ask for a *single* enterprise 1.92TB drive. At least I know I cannot blame the hardware. And since I'm buying refurb, I reasoned WI drives would have more life left in them from the previous owner(s), as the overcommit is stupidly high on those (combined with SLC).
I have for wal+db. (or, in the case of spinning rust, I'd expect a mechanical failure long before a flash failure).
Most SSD failures are firmware and can be fixed in-situ. I've had to replace 4000 HDDs with a design flaw - scattered around the world. Nightmare
Thus, in the end, it's no different from a dedicated wal+db iff wal+db stays dedicated: I've seen people suggest to keep wal+db colocated if there's a fast flash in front of the drive, as wal+db would essentially always be hot in the cache.
Indeed I've read studies that claim a superset of benefits from a transparent cache compared to just WAL+DB.
I can definitely see why. The point however, was that having wal+db dedicated meant osd would stop using the drive if the fast one died. In terms of performance I'd expect no worse than with a cached colocated wal+db. However, we've now established that, for wal+db to be transparently cached, we'd need bcache to avoid data loss. dm-cache needs a way to block osd from using the drive if the cache is missing.
(note: I know that the OSD thinks it directly talks to the crap drives,
Reads are always served from the slow device, and writes over a certain size go there directly.
My understanding from dm-cache docs was that no, in plain dm-cache, reads hit the cache drive. For the behavior you describe (only writes use the cache), you'd need dm-writecache, which is another mode for dm-cache. As for the write size, I guess it's configurable, but I don't think it'll be an issue for my case: what I'm after is preventing the constant chatter or small random writes that hit the drive with a sustained workload. The occasional big write should not matter too much.
I've seen use-cases where the hassle of WAL+DB offload doesn't actually make an appreciable improvement.
I guess it really depends on the drive you have. For HDDs it's a no brainer as far as I understand it, but for SSDs you'd hit diminishing returns, even with NVMe WAL+DB. It's a case where transparent caching of block with an NVMe would make an improvement where WAL+DB does not. Thanks for all the info! Alexis