cephadm crush_device_class not applied
Hi, I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec: cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore I see that cephadm has stored it correctly: ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}} And it has the OSDSPEC_AFFINITY set: cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd'] But the OSDs still are created with hdd device class: ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000 I have tried it with two different indentations: spec: crush_device_class: hdd-ec and as seen above: crush_device_class: hdd-ec placement: label: osd spec: According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation: 2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2. Trying to create it manually also doesn't work as expected: soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph' soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000 This is in the mgr debug output from the manual creation: 2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')) So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here? Thanks! Eugen [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
Reading the docs again, I noticed that apparently the keyword "paths" is required to use with crush_device_class (why?), but that doesn't work either. I tried it by specifying the class both globally in the spec file as well as per device, still no change, the OSDs come up as "hdd". Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec:
cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore
I see that cephadm has stored it correctly:
ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}}
And it has the OSDSPEC_AFFINITY set:
cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd']
But the OSDs still are created with hdd device class:
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000
I have tried it with two different indentations:
spec: crush_device_class: hdd-ec
and as seen above:
crush_device_class: hdd-ec placement: label: osd spec:
According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation:
2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command
Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2.
Trying to create it manually also doesn't work as expected:
soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph'
soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000
This is in the mgr debug output from the manual creation:
2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore '''))
So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here?
Thanks! Eugen
[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
Apparently, I can only use "well known" device classes in the specs, like nvme, ssd or hdd. Every other string (even without hyphens etc.) doesn't work. Zitat von Eugen Block <eblock@nde.ag>:
Reading the docs again, I noticed that apparently the keyword "paths" is required to use with crush_device_class (why?), but that doesn't work either. I tried it by specifying the class both globally in the spec file as well as per device, still no change, the OSDs come up as "hdd".
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec:
cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore
I see that cephadm has stored it correctly:
ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}}
And it has the OSDSPEC_AFFINITY set:
cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd']
But the OSDs still are created with hdd device class:
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000
I have tried it with two different indentations:
spec: crush_device_class: hdd-ec
and as seen above:
crush_device_class: hdd-ec placement: label: osd spec:
According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation:
2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command
Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2.
Trying to create it manually also doesn't work as expected:
soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph'
soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000
This is in the mgr debug output from the manual creation:
2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore '''))
So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here?
Thanks! Eugen
[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
It works as expected in Pacific 16.2.15 (at least how I expect it to work). I applied the same spec file and now have my custom device classes (the test class was the result of a manual daemon add command): soc9-ceph:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 1 hdd-ec 0.00980 osd.1 up 1.00000 1.00000 2 hdd-ec 0.00980 osd.2 up 1.00000 1.00000 3 hdd-ec 0.00980 osd.3 up 1.00000 1.00000 4 hdd-ec 0.00980 osd.4 up 1.00000 1.00000 5 hdd-ec 0.00980 osd.5 up 1.00000 1.00000 0 test 0.00980 osd.0 up 1.00000 1.00000 So apparently, there was a change since Quincy. For me it's a regression, or is this even a bug? I'd appreciate any comments. Zitat von Eugen Block <eblock@nde.ag>:
Apparently, I can only use "well known" device classes in the specs, like nvme, ssd or hdd. Every other string (even without hyphens etc.) doesn't work.
Zitat von Eugen Block <eblock@nde.ag>:
Reading the docs again, I noticed that apparently the keyword "paths" is required to use with crush_device_class (why?), but that doesn't work either. I tried it by specifying the class both globally in the spec file as well as per device, still no change, the OSDs come up as "hdd".
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec:
cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore
I see that cephadm has stored it correctly:
ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}}
And it has the OSDSPEC_AFFINITY set:
cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd']
But the OSDs still are created with hdd device class:
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000
I have tried it with two different indentations:
spec: crush_device_class: hdd-ec
and as seen above:
crush_device_class: hdd-ec placement: label: osd spec:
According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation:
2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command
Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2.
Trying to create it manually also doesn't work as expected:
soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph'
soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000
This is in the mgr debug output from the manual creation:
2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore '''))
So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here?
Thanks! Eugen
[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
I think this PR [1] is responsible. And here are the three supported classes [2]: class to_ceph_volume(object): _supported_device_classes = [ "hdd", "ssd", "nvme" ] Why this limitation? [1] https://github.com/ceph/ceph/pull/49555 [2] https://github.com/ceph/ceph/blob/v18.2.2/src/python-common/ceph/deployment/... Zitat von Eugen Block <eblock@nde.ag>:
It works as expected in Pacific 16.2.15 (at least how I expect it to work). I applied the same spec file and now have my custom device classes (the test class was the result of a manual daemon add command):
soc9-ceph:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 1 hdd-ec 0.00980 osd.1 up 1.00000 1.00000 2 hdd-ec 0.00980 osd.2 up 1.00000 1.00000 3 hdd-ec 0.00980 osd.3 up 1.00000 1.00000 4 hdd-ec 0.00980 osd.4 up 1.00000 1.00000 5 hdd-ec 0.00980 osd.5 up 1.00000 1.00000 0 test 0.00980 osd.0 up 1.00000 1.00000
So apparently, there was a change since Quincy. For me it's a regression, or is this even a bug? I'd appreciate any comments.
Zitat von Eugen Block <eblock@nde.ag>:
Apparently, I can only use "well known" device classes in the specs, like nvme, ssd or hdd. Every other string (even without hyphens etc.) doesn't work.
Zitat von Eugen Block <eblock@nde.ag>:
Reading the docs again, I noticed that apparently the keyword "paths" is required to use with crush_device_class (why?), but that doesn't work either. I tried it by specifying the class both globally in the spec file as well as per device, still no change, the OSDs come up as "hdd".
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec:
cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore
I see that cephadm has stored it correctly:
ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}}
And it has the OSDSPEC_AFFINITY set:
cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd']
But the OSDs still are created with hdd device class:
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000
I have tried it with two different indentations:
spec: crush_device_class: hdd-ec
and as seen above:
crush_device_class: hdd-ec placement: label: osd spec:
According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation:
2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command
Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2.
Trying to create it manually also doesn't work as expected:
soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph'
soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000
This is in the mgr debug output from the manual creation:
2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore '''))
So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here?
Thanks! Eugen
[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
Interesting - my prior research had supported the idea that device classes are named arbitrarily. For sure these three don’t cover everything — one might have, say, nvme-qlc-coarse, nvme-qlc-4k, nvme-slc, nvme-tlc-value, nvme-tlc-performance, etc.
On Oct 3, 2024, at 5:20 PM, Eugen Block <eblock@nde.ag> wrote:
I think this PR [1] is responsible. And here are the three supported classes [2]:
class to_ceph_volume(object):
_supported_device_classes = [ "hdd", "ssd", "nvme" ]
Why this limitation?
I'm not sure, but that's going to break with a lot of people's Pacific specifications when they upgrade. We heavily utilize this functionality, and use different device class names for a lot of good reasons. This seems like a regression to me. David On Thu, Oct 3, 2024, at 16:20, Eugen Block wrote:
I think this PR [1] is responsible. And here are the three supported classes [2]:
class to_ceph_volume(object):
_supported_device_classes = [ "hdd", "ssd", "nvme" ]
Why this limitation?
[1] https://github.com/ceph/ceph/pull/49555 [2] https://github.com/ceph/ceph/blob/v18.2.2/src/python-common/ceph/deployment/...
Zitat von Eugen Block <eblock@nde.ag>:
It works as expected in Pacific 16.2.15 (at least how I expect it to work). I applied the same spec file and now have my custom device classes (the test class was the result of a manual daemon add command):
soc9-ceph:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 1 hdd-ec 0.00980 osd.1 up 1.00000 1.00000 2 hdd-ec 0.00980 osd.2 up 1.00000 1.00000 3 hdd-ec 0.00980 osd.3 up 1.00000 1.00000 4 hdd-ec 0.00980 osd.4 up 1.00000 1.00000 5 hdd-ec 0.00980 osd.5 up 1.00000 1.00000 0 test 0.00980 osd.0 up 1.00000 1.00000
So apparently, there was a change since Quincy. For me it's a regression, or is this even a bug? I'd appreciate any comments.
Zitat von Eugen Block <eblock@nde.ag>:
Apparently, I can only use "well known" device classes in the specs, like nvme, ssd or hdd. Every other string (even without hyphens etc.) doesn't work.
Zitat von Eugen Block <eblock@nde.ag>:
Reading the docs again, I noticed that apparently the keyword "paths" is required to use with crush_device_class (why?), but that doesn't work either. I tried it by specifying the class both globally in the spec file as well as per device, still no change, the OSDs come up as "hdd".
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec:
cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore
I see that cephadm has stored it correctly:
ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}}
And it has the OSDSPEC_AFFINITY set:
cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd']
But the OSDs still are created with hdd device class:
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000
I have tried it with two different indentations:
spec: crush_device_class: hdd-ec
and as seen above:
crush_device_class: hdd-ec placement: label: osd spec:
According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation:
2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command
Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2.
Trying to create it manually also doesn't work as expected:
soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph'
soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000
This is in the mgr debug output from the manual creation:
2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore '''))
So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here?
Thanks! Eugen
[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Eugen, Check this one here: https://github.com/ceph/ceph/pull/55534 It's fixed in 18.2.4 and should be in upcoming 17.2.8. Cheers, Frédéric. ________________________________ De : Eugen Block <eblock@nde.ag> Envoyé : jeudi 3 octobre 2024 23:21 À : ceph-users@ceph.io Objet : [ceph-users] Re: cephadm crush_device_class not applied I think this PR [1] is responsible. And here are the three supported classes [2]: class to_ceph_volume(object): _supported_device_classes = [ "hdd", "ssd", "nvme" ] Why this limitation? [1] https://github.com/ceph/ceph/pull/49555 [2] https://github.com/ceph/ceph/blob/v18.2.2/src/python-common/ceph/deployment/... Zitat von Eugen Block <eblock@nde.ag>:
It works as expected in Pacific 16.2.15 (at least how I expect it to work). I applied the same spec file and now have my custom device classes (the test class was the result of a manual daemon add command):
soc9-ceph:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 1 hdd-ec 0.00980 osd.1 up 1.00000 1.00000 2 hdd-ec 0.00980 osd.2 up 1.00000 1.00000 3 hdd-ec 0.00980 osd.3 up 1.00000 1.00000 4 hdd-ec 0.00980 osd.4 up 1.00000 1.00000 5 hdd-ec 0.00980 osd.5 up 1.00000 1.00000 0 test 0.00980 osd.0 up 1.00000 1.00000
So apparently, there was a change since Quincy. For me it's a regression, or is this even a bug? I'd appreciate any comments.
Zitat von Eugen Block <eblock@nde.ag>:
Apparently, I can only use "well known" device classes in the specs, like nvme, ssd or hdd. Every other string (even without hyphens etc.) doesn't work.
Zitat von Eugen Block <eblock@nde.ag>:
Reading the docs again, I noticed that apparently the keyword "paths" is required to use with crush_device_class (why?), but that doesn't work either. I tried it by specifying the class both globally in the spec file as well as per device, still no change, the OSDs come up as "hdd".
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec:
cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore
I see that cephadm has stored it correctly:
ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}}
And it has the OSDSPEC_AFFINITY set:
cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd']
But the OSDs still are created with hdd device class:
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000
I have tried it with two different indentations:
spec: crush_device_class: hdd-ec
and as seen above:
crush_device_class: hdd-ec placement: label: osd spec:
According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation:
2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command
Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2.
Trying to create it manually also doesn't work as expected:
soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph'
soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000
This is in the mgr debug output from the manual creation:
2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore '''))
So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here?
Thanks! Eugen
[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Oh that’s good news! I haven’t come that far yet, given we had a national holiday yesterday and I had a day off today. :-) I’m glad it’s gonna be fixed in Quincy as well. Thanks! Zitat von Frédéric Nass <frederic.nass@univ-lorraine.fr>:
Hey Eugen,
Check this one here: https://github.com/ceph/ceph/pull/55534
It's fixed in 18.2.4 and should be in upcoming 17.2.8.
Cheers, Frédéric.
________________________________ De : Eugen Block <eblock@nde.ag> Envoyé : jeudi 3 octobre 2024 23:21 À : ceph-users@ceph.io Objet : [ceph-users] Re: cephadm crush_device_class not applied
I think this PR [1] is responsible. And here are the three supported classes [2]:
class to_ceph_volume(object):
_supported_device_classes = [ "hdd", "ssd", "nvme" ]
Why this limitation?
[1] https://github.com/ceph/ceph/pull/49555 [2] https://github.com/ceph/ceph/blob/v18.2.2/src/python-common/ceph/deployment/...
Zitat von Eugen Block <eblock@nde.ag>:
It works as expected in Pacific 16.2.15 (at least how I expect it to work). I applied the same spec file and now have my custom device classes (the test class was the result of a manual daemon add command):
soc9-ceph:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 1 hdd-ec 0.00980 osd.1 up 1.00000 1.00000 2 hdd-ec 0.00980 osd.2 up 1.00000 1.00000 3 hdd-ec 0.00980 osd.3 up 1.00000 1.00000 4 hdd-ec 0.00980 osd.4 up 1.00000 1.00000 5 hdd-ec 0.00980 osd.5 up 1.00000 1.00000 0 test 0.00980 osd.0 up 1.00000 1.00000
So apparently, there was a change since Quincy. For me it's a regression, or is this even a bug? I'd appreciate any comments.
Zitat von Eugen Block <eblock@nde.ag>:
Apparently, I can only use "well known" device classes in the specs, like nvme, ssd or hdd. Every other string (even without hyphens etc.) doesn't work.
Zitat von Eugen Block <eblock@nde.ag>:
Reading the docs again, I noticed that apparently the keyword "paths" is required to use with crush_device_class (why?), but that doesn't work either. I tried it by specifying the class both globally in the spec file as well as per device, still no change, the OSDs come up as "hdd".
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec:
cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore
I see that cephadm has stored it correctly:
ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}}
And it has the OSDSPEC_AFFINITY set:
cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd']
But the OSDs still are created with hdd device class:
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000
I have tried it with two different indentations:
spec: crush_device_class: hdd-ec
and as seen above:
crush_device_class: hdd-ec placement: label: osd spec:
According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation:
2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command
Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2.
Trying to create it manually also doesn't work as expected:
soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph'
soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000
This is in the mgr debug output from the manual creation:
2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore '''))
So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here?
Thanks! Eugen
[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
David Orman
-
Eugen Block
-
Frédéric Nass