Hi Dave, El 22/10/20 a las 16:48, Dave Hall escribió: Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2  SAS drives  and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size). This way if a NVMe fails, you'll only lose 2 OSD. Also, what size of WAL/DB partitions do you have now, and what spillover size? Not using such big disk here, sorry :) (no space needs) Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza(a)binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun

Last I checked HDD prices seemed linear from 10-16TB. Remember to include the cost of the drive bay, ie. the cost of the chassis, the RU(s) it takes up, power, switch ports etc. I’ll guess you’re talking LFF HDDs here and a 2U server? You also don’t tell us how many nodes total, which affects blast radius decisions. Think about what that would do to your total $/TB, including the chassis, CPU, switch ports, rack space, etc. Check if those bays are NVMe-only, or if they are tri-mode. If you do go with NVMe for WAL+DB, 4 drives is overkill. Performance-wise, assuming you use a quality NVMe drive and not some consumer-grade crap, you’re going to see sharply diminishing returns after just 1. Or you could mirror 2 as someone else describes. But really, consider the hassles of maintaining partitions and mapping as drives fail. When an HDD fails and you need to re-use its metadata partition for the replacement OSD, you have to be very careful when using a shared device that you re-use the original. Honestly, depending on your use-case, consider whether using 24xSFF SATA SSDs might not be cost-competitive, factoring in hassle, the time you’ll spend waiting for HDDs to do backfill, etc. With careful choices, and again depending on your undisclosed use-case, all-NVMe can with careful choices also be surprisingly cost-effective. If your data is cold, QLC is an option. With system vendors still pushing expensive RAID HBAs (they must have high margins), you could easly save $600 per chassis just by not having one. Not having to monitor and deal with the BBU/supercap, etc. Don’t cut anything less than an NVMe drive into more than one OSD. HDD seeks and IOPs are its bottlenecks and slicing/dicing isn’t going to work any magic. imho,ymmv

300 is overkill for many applications anyway. Why would you want to move them between hosts? This.

Eneko, On 10/22/2020 11:14 AM, Eneko Lacunza wrote: I was under the impression that everything that BlueStore puts on the SSD/NVMe could be reconstructed from information on the OSD. Am I mistaken about this?  If so, my single 1.6TB NVMe card is equally vulnerable. I recently posted another question to the list on this topic, since I now have spillover on 7 of 24 OSDs.  Since the data layout on the NVMe for BlueStore is not  traditional I've never quite figured out how to get this information.   The current partition size is 1.6TB /12 since we had the possibility to add for more drives to each node.  How that was divided between WAL, DB, etc. is something I'd like to be able to understand.  However, we're not going to add the extra 4 drives, so expanding the LVM partitions is now a possibility. > >> >> Also, any thoughts/recommendations on 12TB OSD drives?  For >> price/capacity this is a good size for us, but I'm wondering if my >> BlueFS spillovers are resulting from using drives that are too big.  >> I also thought I might have seen some comments about cutting large >> drives into multiple OSDs - could that be? > > Not using such big disk here, sorry :) (no space needs) > > Cheers >

Hi Brian, El 22/10/20 a las 18:41, Brian Topping escribió: Yes, I really like mdraid :) . Problem is BIOS/UEFI has to find a working bootable disk. I think some BIOS/UEFIs have settings for a secondary boot/UEFI bootfile, but that would have to be prepared and maintained manually, out of the mdraid10; and would only work with a total failure of the primary disk. Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza(a)binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun

Yes the UEFI problem with mirrored mdraid boot is well-documented. I’ve generally been working with BIOS partition maps which do not have the single point of failure UEFI has (/boot can be mounted as mirrored, any of them can be used as non-RAID by GRUB). But BIOS maps have problems as well with volume size. That said, the disks are portable at that point and really don’t have deep performance bottlenecks because mirroring and striping is cheap. Sent from my iPhone > On Oct 23, 2020, at 03:54, Eneko Lacunza <elacunza(a)binovo.es> wrote: > > Hi Brian, > >> El 22/10/20 a las 18:41, Brian Topping escribió: >> >>>> On Oct 22, 2020, at 10:34 AM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: >>> >>>> - You must really be sure your raid card is dependable. (sorry but I have seen so much management problems with top-tier RAID cards I avoid them like the plague). >>> This. >> I’d definitely avoid a RAID card. If I can do advanced encryption with an MMX instruction, I think I can certainly trust IOMMU to handle device multiplexing from software in an efficient manner, no? mdadm RAID is just fine for me and is reliably bootable from GRUB. >> >> I’m not an expert in driver mechanics, but mirroring should be very low overhead at the software level. >> >> Once it’s software RAID, moving disks between chassis is a simple process as well. >> >> Apologies I didn’t make that clear earlier... > Yes, I really like mdraid :) . Problem is BIOS/UEFI has to find a working bootable disk. I think some BIOS/UEFIs have settings for a secondary boot/UEFI bootfile, but that would have to be prepared and maintained manually, out of the mdraid10; and would only work with a total failure of the primary disk. > > Cheers > > -- > Eneko Lacunza | +34 943 569 206 > | elacunza(a)binovo.es > Zuzendari teknikoa | https://www.binovo.es > Director técnico | Astigarragako Bidea, 2 - 2º izda. > BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Eneko, # ceph health detail HEALTH_WARN BlueFS spillover detected on 7 OSD(s) BLUEFS_SPILLOVER BlueFS spillover detected on 7 OSD(s) osd.1 spilled over 648 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.3 spilled over 613 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.4 spilled over 485 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.10 spilled over 1008 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.17 spilled over 808 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.18 spilled over 2.5 GiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.20 spilled over 1.5 GiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device nvme0n1 259:1 0 1.5T 0 disk ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--6dcbb748--13f5--45cb--9d49--6c78d6589a71 │ 253:1 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--736a22a8--e4aa--4da9--b63b--295d8f5f2a3d │ 253:3 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--751c6623--9870--4123--b551--1fd7fc837341 │ 253:5 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--2a376e8d--abb1--42af--a4bd--4ae8734d703e │ 253:7 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--54fbe282--9b29--422b--bdb2--d7ed730bc589 │ 253:9 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--c1153cd2--2ec0--4e7f--a3d7--91dac92560ad │ 253:11 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--d613f4eb--6ddc--4dd5--a2b5--cb520b6ba922 │ 253:13 0 124G 0 lvm └─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--41f75c25--67db--46e8--a3fb--ddee9e7f7fc4 253:15 0 124G 0 lvm Dave Hall Binghamton Universitykdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On 10/23/2020 6:00 AM, Eneko Lacunza wrote: Hi Dave, El 22/10/20 a las 19:43, Dave Hall escribió: El 22/10/20 a las 16:48, Dave Hall escribió: (BTW, Nautilus 14.2.7 on Debian non-container.) We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc. We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers. The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes. Any thoughts on this approach? Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size). This way if a NVMe fails, you'll only lose 2 OSD. I was under the impression that everything that BlueStore puts on the SSD/NVMe could be reconstructed from information on the OSD. Am I mistaken about this? If so, my single 1.6TB NVMe card is equally vulnerable. I don't think so, that info only exists on that partition as was the case with filestore journal. Your single 1.6TB NVMe is vulnerable, yes. Also, what size of WAL/DB partitions do you have now, and what spillover size? I recently posted another question to the list on this topic, since I now have spillover on 7 of 24 OSDs. Since the data layout on the NVMe for BlueStore is not traditional I've never quite figured out how to get this information. The current partition size is 1.6TB /12 since we had the possibility to add for more drives to each node. How that was divided between WAL, DB, etc. is something I'd like to be able to understand. However, we're not going to add the extra 4 drives, so expanding the LVM partitions is now a possibility. Can you paste the warning message? If shows the spillover size. What size are the partitions on NVMe disk (lsblk) Cheers

Hi Dave, El 23/10/20 a las 22:28, Dave Hall escribió: So, this means that if you use 300GB WAL/DB partitions, your spillovers will be over (Bluestore is only ysing 28GiB as you can see). I don't know what is the performance penalty of current spillover, but at least you know those 300GB will be of use :) Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza(a)binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun