Hardware for new OSD nodes.
Hello, (BTW, Nautilus 14.2.7 on Debian non-container.) We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc. We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers. The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes. Any thoughts on this approach? Also, any thoughts/recommendations on 12TB OSD drives? For price/capacity this is a good size for us, but I'm wondering if my BlueFS spillovers are resulting from using drives that are too big. I also thought I might have seen some comments about cutting large drives into multiple OSDs - could that be? Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu
Hi Dave, El 22/10/20 a las 16:48, Dave Hall escribió:
Hello,
(BTW, Nautilus 14.2.7 on Debian non-container.)
We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc.
We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers.
The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes.
Any thoughts on this approach? Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size).
This way if a NVMe fails, you'll only lose 2 OSD. Also, what size of WAL/DB partitions do you have now, and what spillover size?
Also, any thoughts/recommendations on 12TB OSD drives? For price/capacity this is a good size for us, but I'm wondering if my BlueFS spillovers are resulting from using drives that are too big. I also thought I might have seen some comments about cutting large drives into multiple OSDs - could that be?
Not using such big disk here, sorry :) (no space needs) Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun
On Oct 22, 2020, at 9:14 AM, Eneko Lacunza <elacunza@binovo.es> wrote:
Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size).
This way if a NVMe fails, you'll only lose 2 OSD.
Also, what size of WAL/DB partitions do you have now, and what spillover size?
Generally agreed against making a single giant striped bucket. Note this may be a good use for RAID10 on WAL/DB if you are committed to multiple disks. I generally put WAL/DB on RAID10 boot disks. It’s important to have reliable WAL/DB, but also important that the machine actually boots in the first place. With enough RAM and non-interactive use, most of the boot bits will be cached so there is no contention for the channel. Happy for any critique on this as well! Brian
Hi Brian, El 22/10/20 a las 17:50, Brian Topping escribió:
On Oct 22, 2020, at 9:14 AM, Eneko Lacunza <elacunza@binovo.es <mailto:elacunza@binovo.es>> wrote:
Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size).
This way if a NVMe fails, you'll only lose 2 OSD.
Also, what size of WAL/DB partitions do you have now, and what spillover size?
Generally agreed against making a single giant striped bucket.
Note this may be a good use for RAID10 on WAL/DB if you are committed to multiple disks.
I generally put WAL/DB on RAID10 boot disks. It’s important to have reliable WAL/DB, but also important that the machine actually boots in the first place. With enough RAM and non-interactive use, most of the boot bits will be cached so there is no contention for the channel.
Happy for any critique on this as well!
Yeah, didn't think about a RAID10 really, although there wouldn't be enough space for 8x300GB = 2400GB WAL/DBs. I usually also use the boot disk for WAL/DBs, it happens our clusters are small and nodes not very dense. Also, using a RAID10 for WAL/DBs will: - make OSDs less movable between hosts (they'd have to be moved all together - with 2 OSD per NVMe you can move them around in pairs, although there would be data movement for sure) - Provide half the IOPS/bandwith for WAL/DB (I think there would be plenty for SAS magnetic drives though) + WAL/DBs will be safer (one disk failure won't lose any OSD) - You must really be sure your raid card is dependable. (sorry but I have seen so much management problems with top-tier RAID cards I avoid them like the plague). But it is an interesting idea nonetheless. Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun
Yeah, didn't think about a RAID10 really, although there wouldn't be enough space for 8x300GB = 2400GB WAL/DBs.
300 is overkill for many applications anyway.
Also, using a RAID10 for WAL/DBs will: - make OSDs less movable between hosts (they'd have to be moved all together - with 2 OSD per NVMe you can move them around in pairs
Why would you want to move them between hosts?
- You must really be sure your raid card is dependable. (sorry but I have seen so much management problems with top-tier RAID cards I avoid them like the plague).
This.
On Oct 22, 2020, at 10:34 AM, Anthony D'Atri <anthony.datri@gmail.com> wrote:
- You must really be sure your raid card is dependable. (sorry but I have seen so much management problems with top-tier RAID cards I avoid them like the plague).
This.
I’d definitely avoid a RAID card. If I can do advanced encryption with an MMX instruction, I think I can certainly trust IOMMU to handle device multiplexing from software in an efficient manner, no? mdadm RAID is just fine for me and is reliably bootable from GRUB. I’m not an expert in driver mechanics, but mirroring should be very low overhead at the software level. Once it’s software RAID, moving disks between chassis is a simple process as well. Apologies I didn’t make that clear earlier...
Hi Brian, El 22/10/20 a las 18:41, Brian Topping escribió:
On Oct 22, 2020, at 10:34 AM, Anthony D'Atri <anthony.datri@gmail.com> wrote:
- You must really be sure your raid card is dependable. (sorry but I have seen so much management problems with top-tier RAID cards I avoid them like the plague).
This. I’d definitely avoid a RAID card. If I can do advanced encryption with an MMX instruction, I think I can certainly trust IOMMU to handle device multiplexing from software in an efficient manner, no? mdadm RAID is just fine for me and is reliably bootable from GRUB.
I’m not an expert in driver mechanics, but mirroring should be very low overhead at the software level.
Once it’s software RAID, moving disks between chassis is a simple process as well.
Apologies I didn’t make that clear earlier...
Yes, I really like mdraid :) . Problem is BIOS/UEFI has to find a working bootable disk. I think some BIOS/UEFIs have settings for a secondary boot/UEFI bootfile, but that would have to be prepared and maintained manually, out of the mdraid10; and would only work with a total failure of the primary disk. Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun
Yes the UEFI problem with mirrored mdraid boot is well-documented. I’ve generally been working with BIOS partition maps which do not have the single point of failure UEFI has (/boot can be mounted as mirrored, any of them can be used as non-RAID by GRUB). But BIOS maps have problems as well with volume size. That said, the disks are portable at that point and really don’t have deep performance bottlenecks because mirroring and striping is cheap. Sent from my iPhone
On Oct 23, 2020, at 03:54, Eneko Lacunza <elacunza@binovo.es> wrote:
Hi Brian,
El 22/10/20 a las 18:41, Brian Topping escribió:
On Oct 22, 2020, at 10:34 AM, Anthony D'Atri <anthony.datri@gmail.com> wrote:
- You must really be sure your raid card is dependable. (sorry but I have seen so much management problems with top-tier RAID cards I avoid them like the plague). This. I’d definitely avoid a RAID card. If I can do advanced encryption with an MMX instruction, I think I can certainly trust IOMMU to handle device multiplexing from software in an efficient manner, no? mdadm RAID is just fine for me and is reliably bootable from GRUB.
I’m not an expert in driver mechanics, but mirroring should be very low overhead at the software level.
Once it’s software RAID, moving disks between chassis is a simple process as well.
Apologies I didn’t make that clear earlier... Yes, I really like mdraid :) . Problem is BIOS/UEFI has to find a working bootable disk. I think some BIOS/UEFIs have settings for a secondary boot/UEFI bootfile, but that would have to be prepared and maintained manually, out of the mdraid10; and would only work with a total failure of the primary disk.
Cheers
-- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Brian, Eneko, BTW, the Tyan LFF chassis we've been using has 12 x 3.5" bays in front and 2 x 2.5" SATA bays in back. We've been using 240GB SSDs in the rear bays for mirrored boot drives, so any NVMe we add is exclusively for OSD support. -Dave Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On 10/23/2020 11:55 AM, Brian Topping wrote:
Yes the UEFI problem with mirrored mdraid boot is well-documented. I’ve generally been working with BIOS partition maps which do not have the single point of failure UEFI has (/boot can be mounted as mirrored, any of them can be used as non-RAID by GRUB). But BIOS maps have problems as well with volume size.
That said, the disks are portable at that point and really don’t have deep performance bottlenecks because mirroring and striping is cheap.
Sent from my iPhone
On Oct 23, 2020, at 03:54, Eneko Lacunza <elacunza@binovo.es> wrote:
Hi Brian,
El 22/10/20 a las 18:41, Brian Topping escribió:
On Oct 22, 2020, at 10:34 AM, Anthony D'Atri <anthony.datri@gmail.com> wrote: - You must really be sure your raid card is dependable. (sorry but I have seen so much management problems with top-tier RAID cards I avoid them like the plague). This. I’d definitely avoid a RAID card. If I can do advanced encryption with an MMX instruction, I think I can certainly trust IOMMU to handle device multiplexing from software in an efficient manner, no? mdadm RAID is just fine for me and is reliably bootable from GRUB.
I’m not an expert in driver mechanics, but mirroring should be very low overhead at the software level.
Once it’s software RAID, moving disks between chassis is a simple process as well.
Apologies I didn’t make that clear earlier... Yes, I really like mdraid :) . Problem is BIOS/UEFI has to find a working bootable disk. I think some BIOS/UEFIs have settings for a secondary boot/UEFI bootfile, but that would have to be prepared and maintained manually, out of the mdraid10; and would only work with a total failure of the primary disk.
Cheers
-- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, El 22/10/20 a las 18:34, Anthony D'Atri escribió:
Yeah, didn't think about a RAID10 really, although there wouldn't be enough space for 8x300GB = 2400GB WAL/DBs. 300 is overkill for many applications anyway.
Yes, but he has spillover with 1600GB/12 WAL/DB. Seems he can make use of those 300GB.
Also, using a RAID10 for WAL/DBs will: - make OSDs less movable between hosts (they'd have to be moved all together - with 2 OSD per NVMe you can move them around in pairs Why would you want to move them between hosts?
I think the usual case is a server failure, so that won't be a problem. With small clusters (like ours) you may want to reorganize OSDs to a new server (let's say, move one OSD of earch server to the new server). But this is an uncommon corner-case, I agree :) Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun
Eneko, On 10/22/2020 11:14 AM, Eneko Lacunza wrote:
Hi Dave,
El 22/10/20 a las 16:48, Dave Hall escribió:
Hello,
(BTW, Nautilus 14.2.7 on Debian non-container.)
We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc.
We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers.
The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes.
Any thoughts on this approach? Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size).
This way if a NVMe fails, you'll only lose 2 OSD. I was under the impression that everything that BlueStore puts on the SSD/NVMe could be reconstructed from information on the OSD. Am I mistaken about this? If so, my single 1.6TB NVMe card is equally vulnerable.
Also, what size of WAL/DB partitions do you have now, and what spillover size?
I recently posted another question to the list on this topic, since I now have spillover on 7 of 24 OSDs. Since the data layout on the NVMe for BlueStore is not traditional I've never quite figured out how to get this information. The current partition size is 1.6TB /12 since we had the possibility to add for more drives to each node. How that was divided between WAL, DB, etc. is something I'd like to be able to understand. However, we're not going to add the extra 4 drives, so expanding the LVM partitions is now a possibility.
Also, any thoughts/recommendations on 12TB OSD drives? For price/capacity this is a good size for us, but I'm wondering if my BlueFS spillovers are resulting from using drives that are too big. I also thought I might have seen some comments about cutting large drives into multiple OSDs - could that be?
Not using such big disk here, sorry :) (no space needs)
Cheers
Hi Dave, El 22/10/20 a las 19:43, Dave Hall escribió:
El 22/10/20 a las 16:48, Dave Hall escribió:
(BTW, Nautilus 14.2.7 on Debian non-container.)
We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc.
We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers.
The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes.
Any thoughts on this approach?
Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size).
This way if a NVMe fails, you'll only lose 2 OSD. I was under the impression that everything that BlueStore puts on the SSD/NVMe could be reconstructed from information on the OSD. Am I mistaken about this? If so, my single 1.6TB NVMe card is equally vulnerable.
I don't think so, that info only exists on that partition as was the case with filestore journal. Your single 1.6TB NVMe is vulnerable, yes.
Also, what size of WAL/DB partitions do you have now, and what spillover size?
I recently posted another question to the list on this topic, since I now have spillover on 7 of 24 OSDs. Since the data layout on the NVMe for BlueStore is not traditional I've never quite figured out how to get this information. The current partition size is 1.6TB /12 since we had the possibility to add for more drives to each node. How that was divided between WAL, DB, etc. is something I'd like to be able to understand. However, we're not going to add the extra 4 drives, so expanding the LVM partitions is now a possibility. Can you paste the warning message? If shows the spillover size. What size are the partitions on NVMe disk (lsblk)
Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun
Eneko, # ceph health detail HEALTH_WARN BlueFS spillover detected on 7 OSD(s) BLUEFS_SPILLOVER BlueFS spillover detected on 7 OSD(s) osd.1 spilled over 648 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.3 spilled over 613 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.4 spilled over 485 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.10 spilled over 1008 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.17 spilled over 808 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.18 spilled over 2.5 GiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.20 spilled over 1.5 GiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device nvme0n1 259:1 0 1.5T 0 disk ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--6dcbb748--13f5--45cb--9d49--6c78d6589a71 │ 253:1 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--736a22a8--e4aa--4da9--b63b--295d8f5f2a3d │ 253:3 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--751c6623--9870--4123--b551--1fd7fc837341 │ 253:5 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--2a376e8d--abb1--42af--a4bd--4ae8734d703e │ 253:7 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--54fbe282--9b29--422b--bdb2--d7ed730bc589 │ 253:9 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--c1153cd2--2ec0--4e7f--a3d7--91dac92560ad │ 253:11 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--d613f4eb--6ddc--4dd5--a2b5--cb520b6ba922 │ 253:13 0 124G 0 lvm └─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--41f75c25--67db--46e8--a3fb--ddee9e7f7fc4 253:15 0 124G 0 lvm Dave Hall Binghamton Universitykdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On 10/23/2020 6:00 AM, Eneko Lacunza wrote: Hi Dave, El 22/10/20 a las 19:43, Dave Hall escribió: El 22/10/20 a las 16:48, Dave Hall escribió: (BTW, Nautilus 14.2.7 on Debian non-container.) We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc. We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers. The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes. Any thoughts on this approach? Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size). This way if a NVMe fails, you'll only lose 2 OSD. I was under the impression that everything that BlueStore puts on the SSD/NVMe could be reconstructed from information on the OSD. Am I mistaken about this? If so, my single 1.6TB NVMe card is equally vulnerable. I don't think so, that info only exists on that partition as was the case with filestore journal. Your single 1.6TB NVMe is vulnerable, yes. Also, what size of WAL/DB partitions do you have now, and what spillover size? I recently posted another question to the list on this topic, since I now have spillover on 7 of 24 OSDs. Since the data layout on the NVMe for BlueStore is not traditional I've never quite figured out how to get this information. The current partition size is 1.6TB /12 since we had the possibility to add for more drives to each node. How that was divided between WAL, DB, etc. is something I'd like to be able to understand. However, we're not going to add the extra 4 drives, so expanding the LVM partitions is now a possibility. Can you paste the warning message? If shows the spillover size. What size are the partitions on NVMe disk (lsblk) Cheers
Hi Dave, El 23/10/20 a las 22:28, Dave Hall escribió:
Eneko,
# ceph health detail HEALTH_WARN BlueFS spillover detected on 7 OSD(s) BLUEFS_SPILLOVER BlueFS spillover detected on 7 OSD(s) osd.1 spilled over 648 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.3 spilled over 613 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.4 spilled over 485 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.10 spilled over 1008 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.17 spilled over 808 MiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.18 spilled over 2.5 GiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device osd.20 spilled over 1.5 GiB metadata from 'db' device (28 GiB used of 124 GiB) to slow device
nvme0n1 259:1 0 1.5T 0 disk ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--6dcbb748--13f5--45cb--9d49--6c78d6589a71 │ 253:1 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--736a22a8--e4aa--4da9--b63b--295d8f5f2a3d │ 253:3 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--751c6623--9870--4123--b551--1fd7fc837341 │ 253:5 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--2a376e8d--abb1--42af--a4bd--4ae8734d703e │ 253:7 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--54fbe282--9b29--422b--bdb2--d7ed730bc589 │ 253:9 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--c1153cd2--2ec0--4e7f--a3d7--91dac92560ad │ 253:11 0 124G 0 lvm ├─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--d613f4eb--6ddc--4dd5--a2b5--cb520b6ba922 │ 253:13 0 124G 0 lvm └─ceph--block--dbs--a2b7a161--d4da--4b86--a191--37564008adca-osd--block--db--41f75c25--67db--46e8--a3fb--ddee9e7f7fc4 253:15 0 124G 0 lvm
So, this means that if you use 300GB WAL/DB partitions, your spillovers will be over (Bluestore is only ysing 28GiB as you can see). I don't know what is the performance penalty of current spillover, but at least you know those 300GB will be of use :) Cheers
Dave Hall Binghamton University kdhall@binghamton.edu <mailto:kdhall@binghamton.edu> 607-760-2328 (Cell) 607-777-4641 (Office) On 10/23/2020 6:00 AM, Eneko Lacunza wrote:
Hi Dave,
El 22/10/20 a las 19:43, Dave Hall escribió:
El 22/10/20 a las 16:48, Dave Hall escribió:
(BTW, Nautilus 14.2.7 on Debian non-container.)
We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc.
We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers.
The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes.
Any thoughts on this approach?
Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size).
This way if a NVMe fails, you'll only lose 2 OSD. I was under the impression that everything that BlueStore puts on the SSD/NVMe could be reconstructed from information on the OSD. Am I mistaken about this? If so, my single 1.6TB NVMe card is equally vulnerable.
I don't think so, that info only exists on that partition as was the case with filestore journal. Your single 1.6TB NVMe is vulnerable, yes.
Also, what size of WAL/DB partitions do you have now, and what spillover size?
I recently posted another question to the list on this topic, since I now have spillover on 7 of 24 OSDs. Since the data layout on the NVMe for BlueStore is not traditional I've never quite figured out how to get this information. The current partition size is 1.6TB /12 since we had the possibility to add for more drives to each node. How that was divided between WAL, DB, etc. is something I'd like to be able to understand. However, we're not going to add the extra 4 drives, so expanding the LVM partitions is now a possibility. Can you paste the warning message? If shows the spillover size. What size are the partitions on NVMe disk (lsblk)
Cheers
-- Eneko Lacunza | +34 943 569 206 | elacunza@binovo.es Zuzendari teknikoa | https://www.binovo.es Director técnico | Astigarragako Bidea, 2 - 2º izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun
If there is only a small amount of fast storage available (e.g., less than a gigabyte), we recommend using it as a WAL device. If there is more, provisioning a DB device makes more sense. The BlueStore journal will always be placed on the fastest device available, so using a DB device will provide the same benefit that the WAL device would while /also/ allowing additional metadata to be stored there (if it will fit). This makes me wonder if I shouldn't just move my DBs onto the HDDs and switch to WAL on the NVMe partitions. Does anybody have any thoughts on
Eneko and all, Regarding my current BlueFS Spillover issues, I've just noticed in https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/ that it says: this? BTW, I don't think I have WAL set up, but I'd really like to check on both WAL and Journal settings to see if I can make any improvements. I also have 150GB left on my mirrored boot drive. I could un-mirror part of this and get 300GB of SATA SSD. Thoughts? -Dave Dave Hall Binghamton University kdhall@binghamton.edu On 10/23/2020 6:00 AM, Eneko Lacunza wrote:
Hi Dave,
El 22/10/20 a las 19:43, Dave Hall escribió:
El 22/10/20 a las 16:48, Dave Hall escribió:
(BTW, Nautilus 14.2.7 on Debian non-container.)
We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc.
We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers.
The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes.
Any thoughts on this approach?
Don't stripe them, if one NVMe fails you'll lose all OSDs. Just use 1 NVMe drive for 2 SAS drives and provision 300GB for WAL/DB for each OSD (see related threads on this mailing list about why that exact size).
This way if a NVMe fails, you'll only lose 2 OSD. I was under the impression that everything that BlueStore puts on the SSD/NVMe could be reconstructed from information on the OSD. Am I mistaken about this? If so, my single 1.6TB NVMe card is equally vulnerable.
I don't think so, that info only exists on that partition as was the case with filestore journal. Your single 1.6TB NVMe is vulnerable, yes.
Also, what size of WAL/DB partitions do you have now, and what spillover size?
I recently posted another question to the list on this topic, since I now have spillover on 7 of 24 OSDs. Since the data layout on the NVMe for BlueStore is not traditional I've never quite figured out how to get this information. The current partition size is 1.6TB /12 since we had the possibility to add for more drives to each node. How that was divided between WAL, DB, etc. is something I'd like to be able to understand. However, we're not going to add the extra 4 drives, so expanding the LVM partitions is now a possibility. Can you paste the warning message? If shows the spillover size. What size are the partitions on NVMe disk (lsblk)
Cheers
Also, any thoughts/recommendations on 12TB OSD drives? For price/capacity this is a good size for us
Last I checked HDD prices seemed linear from 10-16TB. Remember to include the cost of the drive bay, ie. the cost of the chassis, the RU(s) it takes up, power, switch ports etc. I’ll guess you’re talking LFF HDDs here and a 2U server? You also don’t tell us how many nodes total, which affects blast radius decisions.
has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays)
Think about what that would do to your total $/TB, including the chassis, CPU, switch ports, rack space, etc. Check if those bays are NVMe-only, or if they are tri-mode. If you do go with NVMe for WAL+DB, 4 drives is overkill. Performance-wise, assuming you use a quality NVMe drive and not some consumer-grade crap, you’re going to see sharply diminishing returns after just 1. Or you could mirror 2 as someone else describes. But really, consider the hassles of maintaining partitions and mapping as drives fail. When an HDD fails and you need to re-use its metadata partition for the replacement OSD, you have to be very careful when using a shared device that you re-use the original. Honestly, depending on your use-case, consider whether using 24xSFF SATA SSDs might not be cost-competitive, factoring in hassle, the time you’ll spend waiting for HDDs to do backfill, etc. With careful choices, and again depending on your undisclosed use-case, all-NVMe can with careful choices also be surprisingly cost-effective. If your data is cold, QLC is an option. With system vendors still pushing expensive RAID HBAs (they must have high margins), you could easly save $600 per chassis just by not having one. Not having to monitor and deal with the BBU/supercap, etc.
but I'm wondering if my BlueFS spillovers are resulting from using drives that are too big. I also thought I might have seen some comments about cutting large drives into multiple OSDs - could that be?
Don’t cut anything less than an NVMe drive into more than one OSD. HDD seeks and IOPs are its bottlenecks and slicing/dicing isn’t going to work any magic. imho,ymmv
participants (4)
-
Anthony D'Atri
-
Brian Topping
-
Dave Hall
-
Eneko Lacunza