Hi Brian, El 22/10/20 a las 17:50, Brian Topping escribió:
On Oct 22, 2020, at 9:14 AM, Eneko Lacunza <elacunza@binovo.es <mailto:elacunza@binovo.es>> wrote:
Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size).
This way if a NVMe fails, you'll only lose 2 OSD.
Also, what size of WAL/DB partitions do you have now, and what spillover size?
Generally agreed against making a single giant striped bucket.
Note this may be a good use for RAID10 on WAL/DB if you are committed to multiple disks.
I generally put WAL/DB on RAID10 boot disks. It’s important to have reliable WAL/DB, but also important that the machine actually boots in the first place. With enough RAM and non-interactive use, most of the boot bits will be cached so there is no contention for the channel.
Happy for any critique on this as well!
Yeah, didn't think about a RAID10 really, although there wouldn't be enough space for 8x300GB = 2400GB WAL/DBs. I usually also use the boot disk for WAL/DBs, it happens our clusters are small and nodes not very dense. Also, using a RAID10 for WAL/DBs will: - make OSDs less movable between hosts (they'd have to be moved all together - with 2 OSD per NVMe you can move them around in pairs, although there would be data movement for sure) - Provide half the IOPS/bandwith for WAL/DB (I think there would be plenty for SAS magnetic drives though) + WAL/DBs will be safer (one disk failure won't lose any OSD) - You must really be sure your raid card is dependable. (sorry but I have seen so much management problems with top-tier RAID cards I avoid them like the plague). But it is an interesting idea nonetheless. Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun