it looks like you've hit https://tracker.ceph.com/issues/58946 which has a candidate fix open, but nothing merged. The description on the PR with the candidate fix says "When osdspec_affinity is not set, the drive selection code will fail. This can happen when a device has multiple LVs where some of are used by Ceph and at least one LV isn't used by Ceph." so maybe you can start there in terms of finding a potential workaround for now. On Wed, Aug 16, 2023 at 12:05 PM Adam Huffman <adam.huffman.lists@gmail.com> wrote:
I've been having fun today trying to invite a new disk that replaced a failing one into a cluster.
One of my attempts to apply an OSD spec was clearly wrong, because I now have this error:
Module 'cephadm' has failed: 'osdspec_affinity'
and this was the traceback in the mgr logs:
Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 224, in refresh r = self._refresh_host_devices(host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 396, in _refresh_host_devices self.update_osdspec_previews(host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in update_osdspec_previews previews.extend(self.mgr.osd_service.get_previews(search_host)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 258, in get_previews return self.generate_previews(osdspecs, host) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 291, in generate_previews for host, ds in self.prepare_drivegroup(osdspec): File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 225, in prepare_drivegroup existing_daemons=len(dd_for_spec_and_host)) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 35, in __init__ self._data = self.assign_devices('data_devices', self.spec.data_devices) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 19, in wrapper return f(self, ds) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 134, in assign_devices if lv['osdspec_affinity'] != self.spec.service_id: KeyError: 'osdspec_affinity'
This cluster is running 16.2.13.
The exported service spec is:
service_type: osd service_id: osd_spec-0.3 service_name: osd.osd_spec-0.3 placement: host_pattern: cepho-* spec: data_devices: rotational: true db_devices: model: SSDPE2KE032T8L encrypted: true filter_logic: AND objectstore: bluestore
Best Wishes, Adam _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io