Hi We have a bunch of HDD OSD hosts with DB/WAL on PCI NVMe, either 2 x 3.2TB or 1 x 6.4TB. We used to have 4 SSDs pr node for journals before bluestore and those have been repurposed for an SSD pool (wear level is fine). We've been using the following service specs to avoid the PCI NVMe devices for bluestore being provisioned as OSDs: --- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G <-- only use devices smaller than 1TB = not PCI NVMe filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:' <-- only use devices larger than 1TB for DB/WAL filter_logic: AND objectstore: bluestore --- We just bought a few 7.68 TB SATA SSDs to add to the SSD pool which aren't being picked up by the osd.fast spec because they are too large and they could also be picked up as DB/WAL with the current specs. As far as I can determine there is no way to achieve what I want with the existing specs, as I can't filter on PCI vs SATA, only rotational or not, I can't use size, as it only can define an in between range, not an outside range, and I can't use filter_logic OR for the sizes because I need the rotational qualifier to be AND. I can do a osd.fast2 spec with size: 7000G: and change the db_devices size for osd.slow to something like 1000G:7000G but curious to see if anyone would have a different suggestion? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
On 26/06/2024 08:48, Torkil Svensgaard wrote:
Hi
We have a bunch of HDD OSD hosts with DB/WAL on PCI NVMe, either 2 x 3.2TB or 1 x 6.4TB. We used to have 4 SSDs pr node for journals before bluestore and those have been repurposed for an SSD pool (wear level is fine).
We've been using the following service specs to avoid the PCI NVMe devices for bluestore being provisioned as OSDs:
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G <-- only use devices smaller than 1TB = not PCI NVMe filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:' <-- only use devices larger than 1TB for DB/WAL filter_logic: AND objectstore: bluestore ---
We just bought a few 7.68 TB SATA SSDs to add to the SSD pool which aren't being picked up by the osd.fast spec because they are too large and they could also be picked up as DB/WAL with the current specs.
As far as I can determine there is no way to achieve what I want with the existing specs, as I can't filter on PCI vs SATA, only rotational or not, I can't use size, as it only can define an in between range, not an outside range, and I can't use filter_logic OR for the sizes because I need the rotational qualifier to be AND.
I can do a osd.fast2 spec with size: 7000G: and change the db_devices size for osd.slow to something like 1000G:7000G but curious to see if anyone would have a different suggestion?
Regarding this last part, this is the new SSD as ceph orch device ls sees it: ssd ATA_SAMSUNG_MZ7L37T6HBLA-00A07_S6EPNN0X504375 7153G But this in a spec doesn't match it: size: '7000G:' This does: size: '6950G:' I can't get that to make sense. The value from ceph orch device ls looks like GiB. The documentation[1] states that the spec file uses GB and 7000GB should be less than 7153GiB (and so should 7000GiB for that matter)? Some sort of internal rounding off? Mvh. Torkil [1] https://docs.ceph.com/en/latest/cephadm/services/osd/
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
----- Le 26 Juin 24, à 10:50, Torkil Svensgaard torkil@drcmr.dk a écrit :
On 26/06/2024 08:48, Torkil Svensgaard wrote:
Hi
We have a bunch of HDD OSD hosts with DB/WAL on PCI NVMe, either 2 x 3.2TB or 1 x 6.4TB. We used to have 4 SSDs pr node for journals before bluestore and those have been repurposed for an SSD pool (wear level is fine).
We've been using the following service specs to avoid the PCI NVMe devices for bluestore being provisioned as OSDs:
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G <-- only use devices smaller than 1TB = not PCI NVMe filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:' <-- only use devices larger than 1TB for DB/WAL filter_logic: AND objectstore: bluestore ---
We just bought a few 7.68 TB SATA SSDs to add to the SSD pool which aren't being picked up by the osd.fast spec because they are too large and they could also be picked up as DB/WAL with the current specs.
As far as I can determine there is no way to achieve what I want with the existing specs, as I can't filter on PCI vs SATA, only rotational or not, I can't use size, as it only can define an in between range, not an outside range, and I can't use filter_logic OR for the sizes because I need the rotational qualifier to be AND.
I can do a osd.fast2 spec with size: 7000G: and change the db_devices size for osd.slow to something like 1000G:7000G but curious to see if anyone would have a different suggestion?
Regarding this last part, this is the new SSD as ceph orch device ls sees it:
ssd ATA_SAMSUNG_MZ7L37T6HBLA-00A07_S6EPNN0X504375 7153G
But this in a spec doesn't match it:
size: '7000G:'
This does:
size: '6950G:'
I can't get that to make sense. The value from ceph orch device ls looks like GiB. The documentation[1] states that the spec file uses GB and 7000GB should be less than 7153GiB (and so should 7000GiB for that matter)? Some sort of internal rounding off?
Mvh.
Torkil
I've examined the code, and here's what I found (I'd appreciate if someone could confirm my understanding): The orchestrator gets the disk size from ceph-volume inventory command (human_readable_size) and compares it to whatever size value(s) you set for the OSD service. $ cephadm shell ceph-volume inventory /dev/sdc --format json | jq .sys_api.human_readable_size "3.64 TB" The 'size:' spec you set is in GB (only GB and MB are supported). However, ceph-volume inventory output can use other units (TB in this example). Therefore, the orchestrator first converts both values to bytes. Since the ceph-volume inventory produces a figure with only 2 decimals and the conversion uses powers of 10 (1e+9 for GB, 1e+12 for TB), the matching size here would be "size: 3640GB". This was confirmed by my testing a few months ago. If my understanding is correct, it may be worth adding to the doc [2] that the device size is human_readable_size in TB from ceph-volume inventory x 10 GB. Regards, Frédéric. [1] https://github.com/ceph/ceph/blob/main/src/python-common/ceph/deployment/dri... [2] https://docs.ceph.com/en/latest/cephadm/services/osd/
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
But this in a spec doesn't match it:
size: '7000G:'
This does:
size: '6950G:'
There definitely is some rounding within Ceph, and base 2 vs base 10 shenanigans.
$ cephadm shell ceph-volume inventory /dev/sdc --format json | jq .sys_api.human_readable_size "3.64 TB"
Ceph like humans thinks in terms of base 2 units, e.g. GiB and TiB. Storage manufacturers are, well, mustelids and almost always express in terms of base 10 units, GB and TB, because they read as slightly higher.
The 'size:' spec you set is in GB (only GB and MB are supported). However, ceph-volume inventory output can use other units (TB in this example). Therefore, the orchestrator first converts both values to bytes. Since the ceph-volume inventory produces a figure with only 2 decimals and the conversion uses powers of 10 (1e+9 for GB, 1e+12 for TB), the matching size here would be "size: 3640GB". This was confirmed by my testing a few months ago.
If my understanding is correct, it may be worth adding to the doc [2] that the device size is human_readable_size in TB from ceph-volume inventory x 10 GB.
Please enter a tracker ticket for this with details and tag me. Do you mean x1000 not x10?
Regards, Frédéric.
[1] https://github.com/ceph/ceph/blob/main/src/python-common/ceph/deployment/dri... [2] https://docs.ceph.com/en/latest/cephadm/services/osd/
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
----- Le 28 Juin 24, à 15:27, Anthony D'Atri anthony.datri@gmail.com a écrit :
But this in a spec doesn't match it:
size: '7000G:'
This does:
size: '6950G:'
There definitely is some rounding within Ceph, and base 2 vs base 10 shenanigans.
$ cephadm shell ceph-volume inventory /dev/sdc --format json | jq .sys_api.human_readable_size "3.64 TB"
Ceph like humans thinks in terms of base 2 units, e.g. GiB and TiB. Storage manufacturers are, well, mustelids and almost always express in terms of base 10 units, GB and TB, because they read as slightly higher.
The 'size:' spec you set is in GB (only GB and MB are supported). However, ceph-volume inventory output can use other units (TB in this example). Therefore, the orchestrator first converts both values to bytes. Since the ceph-volume inventory produces a figure with only 2 decimals and the conversion uses powers of 10 (1e+9 for GB, 1e+12 for TB), the matching size here would be "size: 3640GB". This was confirmed by my testing a few months ago.
If my understanding is correct, it may be worth adding to the doc [2] that the device size is human_readable_size in TB from ceph-volume inventory x 10 GB.
Please enter a tracker ticket for this with details and tag me. Do you mean x1000 not x10?
Of course x1000. ;-) Here's the tracker: https://tracker.ceph.com/issues/66754 Frédéric.
Regards, Frédéric.
[1] https://github.com/ceph/ceph/blob/main/src/python-common/ceph/deployment/dri... [2] https://docs.ceph.com/en/latest/cephadm/services/osd/
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Torkil, Ruben, I see two theoretical ways to do this without additional OSD service. One that probably doesn't work :-) and another one that could work depending on how the orchestrator prioritize its actions based on services criteria. The one that probably doesn't work is by specifying multiple exact size with an OR filter in osd.fast service (with no modifications to the osd.slow service): --- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: size: 400G size: 7680G filter_logic: OR objectstore: bluestore I doubt the orchestrator can handle multiple exact size criteria. The one that may work is: --- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 <-- remove size criteria filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:6900G' model: NVME-QQQQ-987 <-- specify NVMe's model filter_logic: AND objectstore: bluestore --- This may work if the orchestrator gives priority to services with more specific criteria. If not then you may want to add SSDs vendor's criteria (if 3.2TB and 6.4TB SSDs drives are from the same vendor AND NVMEs are from another) to the osd.fast service. Regards, Frédéric. ----- Le 26 Juin 24, à 8:48, Torkil Svensgaard torkil@drcmr.dk a écrit :
Hi
We have a bunch of HDD OSD hosts with DB/WAL on PCI NVMe, either 2 x 3.2TB or 1 x 6.4TB. We used to have 4 SSDs pr node for journals before bluestore and those have been repurposed for an SSD pool (wear level is fine).
We've been using the following service specs to avoid the PCI NVMe devices for bluestore being provisioned as OSDs:
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G <-- only use devices smaller than 1TB = not PCI NVMe filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:' <-- only use devices larger than 1TB for DB/WAL filter_logic: AND objectstore: bluestore ---
We just bought a few 7.68 TB SATA SSDs to add to the SSD pool which aren't being picked up by the osd.fast spec because they are too large and they could also be picked up as DB/WAL with the current specs.
As far as I can determine there is no way to achieve what I want with the existing specs, as I can't filter on PCI vs SATA, only rotational or not, I can't use size, as it only can define an in between range, not an outside range, and I can't use filter_logic OR for the sizes because I need the rotational qualifier to be AND.
I can do a osd.fast2 spec with size: 7000G: and change the db_devices size for osd.slow to something like 1000G:7000G but curious to see if anyone would have a different suggestion?
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 27-06-2024 10:56, Frédéric Nass wrote:
Hi Torkil, Ruben,
Hi Frédéric
I see two theoretical ways to do this without additional OSD service. One that probably doesn't work :-) and another one that could work depending on how the orchestrator prioritize its actions based on services criteria.
The one that probably doesn't work is by specifying multiple exact size with an OR filter in osd.fast service (with no modifications to the osd.slow service):
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: size: 400G size: 7680G filter_logic: OR objectstore: bluestore
I doubt the orchestrator can handle multiple exact size criteria.
I am pretty sure this wouldn't work because it would also match HDDs.
The one that may work is:
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 <-- remove size criteria filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:6900G' model: NVME-QQQQ-987 <-- specify NVMe's model filter_logic: AND objectstore: bluestore ---
This may work if the orchestrator gives priority to services with more specific criteria. If not then you may want to add SSDs vendor's criteria (if 3.2TB and 6.4TB SSDs drives are from the same vendor AND NVMEs are from another) to the osd.fast service.
We considered trying to test if some ordering or priority could sort this but we resigned to just do multiple specs which wouldn't depend on undocumented behavior that might change. service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G filter_logic: AND objectstore: bluestore --- service_type: osd service_id: fast2 service_name: osd.fast2 placement: host_pattern: '*' spec: data_devices: rotational: 0 size: '6990G:' filter_logic: AND objectstore: bluestore It might be better to go with even more specs and use models instead of sizes for everything not HDD but we have a lot of different models so as long as it's not broken this will do. Thanks for the suggestions! Mvh. Torkil
Regards, Frédéric.
----- Le 26 Juin 24, à 8:48, Torkil Svensgaard torkil@drcmr.dk a écrit :
Hi
We have a bunch of HDD OSD hosts with DB/WAL on PCI NVMe, either 2 x 3.2TB or 1 x 6.4TB. We used to have 4 SSDs pr node for journals before bluestore and those have been repurposed for an SSD pool (wear level is fine).
We've been using the following service specs to avoid the PCI NVMe devices for bluestore being provisioned as OSDs:
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G <-- only use devices smaller than 1TB = not PCI NVMe filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:' <-- only use devices larger than 1TB for DB/WAL filter_logic: AND objectstore: bluestore ---
We just bought a few 7.68 TB SATA SSDs to add to the SSD pool which aren't being picked up by the osd.fast spec because they are too large and they could also be picked up as DB/WAL with the current specs.
As far as I can determine there is no way to achieve what I want with the existing specs, as I can't filter on PCI vs SATA, only rotational or not, I can't use size, as it only can define an in between range, not an outside range, and I can't use filter_logic OR for the sizes because I need the rotational qualifier to be AND.
I can do a osd.fast2 spec with size: 7000G: and change the db_devices size for osd.slow to something like 1000G:7000G but curious to see if anyone would have a different suggestion?
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Torkil, I didn't want to suggest using multiple OSD services from the start as you were trying to avoid adding more. Here, we've been using per hosts (listing hosts and not using wildcard pattern) OSDs specs, as buying new hardware over time, our cluster became more heterogeneous than before in terms of hardware. We chose to do this on per host basis to be more deterministic and less prone to unexpected orchestrator behaviors over time. That's what counts the most in the end I think. By the way, we've reached RHCS support a year ago to ask for a regex host pattern, which led to this [1] and that [2], thanks to Adam. It's been merged to Reef but not Quincy yet. This will help reduce the number of OSDs services as regular expressions will allow catching more hosts than Python's fnmatch did. Bests, Frédéric. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2219373 [2] https://github.com/ceph/ceph/pull/53803 ----- Le 28 Juin 24, à 10:34, Torkil Svensgaard torkil@drcmr.dk a écrit :
On 27-06-2024 10:56, Frédéric Nass wrote:
Hi Torkil, Ruben,
Hi Frédéric
I see two theoretical ways to do this without additional OSD service. One that probably doesn't work :-) and another one that could work depending on how the orchestrator prioritize its actions based on services criteria.
The one that probably doesn't work is by specifying multiple exact size with an OR filter in osd.fast service (with no modifications to the osd.slow service):
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: size: 400G size: 7680G filter_logic: OR objectstore: bluestore
I doubt the orchestrator can handle multiple exact size criteria.
I am pretty sure this wouldn't work because it would also match HDDs.
The one that may work is:
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 <-- remove size criteria filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:6900G' model: NVME-QQQQ-987 <-- specify NVMe's model filter_logic: AND objectstore: bluestore ---
This may work if the orchestrator gives priority to services with more specific criteria. If not then you may want to add SSDs vendor's criteria (if 3.2TB and 6.4TB SSDs drives are from the same vendor AND NVMEs are from another) to the osd.fast service.
We considered trying to test if some ordering or priority could sort this but we resigned to just do multiple specs which wouldn't depend on undocumented behavior that might change.
service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G filter_logic: AND objectstore: bluestore --- service_type: osd service_id: fast2 service_name: osd.fast2 placement: host_pattern: '*' spec: data_devices: rotational: 0 size: '6990G:' filter_logic: AND objectstore: bluestore
It might be better to go with even more specs and use models instead of sizes for everything not HDD but we have a lot of different models so as long as it's not broken this will do.
Thanks for the suggestions!
Mvh.
Torkil
Regards, Frédéric.
----- Le 26 Juin 24, à 8:48, Torkil Svensgaard torkil@drcmr.dk a écrit :
Hi
We have a bunch of HDD OSD hosts with DB/WAL on PCI NVMe, either 2 x 3.2TB or 1 x 6.4TB. We used to have 4 SSDs pr node for journals before bluestore and those have been repurposed for an SSD pool (wear level is fine).
We've been using the following service specs to avoid the PCI NVMe devices for bluestore being provisioned as OSDs:
--- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G <-- only use devices smaller than 1TB = not PCI NVMe filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:' <-- only use devices larger than 1TB for DB/WAL filter_logic: AND objectstore: bluestore ---
We just bought a few 7.68 TB SATA SSDs to add to the SSD pool which aren't being picked up by the osd.fast spec because they are too large and they could also be picked up as DB/WAL with the current specs.
As far as I can determine there is no way to achieve what I want with the existing specs, as I can't filter on PCI vs SATA, only rotational or not, I can't use size, as it only can define an in between range, not an outside range, and I can't use filter_logic OR for the sizes because I need the rotational qualifier to be AND.
I can do a osd.fast2 spec with size: 7000G: and change the db_devices size for osd.slow to something like 1000G:7000G but curious to see if anyone would have a different suggestion?
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Anthony D'Atri
-
Frédéric Nass
-
Torkil Svensgaard