16 Aug
2023
16 Aug
'23
7:04 p.m.
I've been having fun today trying to invite a new disk that replaced a
failing one into a cluster.
One of my attempts to apply an OSD spec was clearly wrong, because I now
have this error:
Module 'cephadm' has failed: 'osdspec_affinity'
and this was the traceback in the mgr logs:
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work
return f(*arg)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 224, in refresh
r = self._refresh_host_devices(host)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 396, in
_refresh_host_devices
self.update_osdspec_previews(host)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in
update_osdspec_previews
previews.extend(self.mgr.osd_service.get_previews(search_host))
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 258, in
get_previews
return self.generate_previews(osdspecs, host)
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 291, in
generate_previews
for host, ds in self.prepare_drivegroup(osdspec):
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 225, in
prepare_drivegroup
existing_daemons=len(dd_for_spec_and_host))
File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
line 35, in __init__
self._data = self.assign_devices('data_devices',
self.spec.data_devices)
File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
line 19, in wrapper
return f(self, ds)
File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
line 134, in assign_devices
if lv['osdspec_affinity'] != self.spec.service_id:
KeyError: 'osdspec_affinity'
This cluster is running 16.2.13.
The exported service spec is:
service_type: osd
service_id: osd_spec-0.3
service_name: osd.osd_spec-0.3
placement:
host_pattern: cepho-*
spec:
data_devices:
rotational: true
db_devices:
model: SSDPE2KE032T8L
encrypted: true
filter_logic: AND
objectstore: bluestore
Best Wishes,
Adam
16 Aug
16 Aug
8:02 p.m.
it looks like you've hit https://tracker.ceph.com/issues/58946 which has a
candidate fix open, but nothing merged. The description on the PR with the
candidate fix says "When osdspec_affinity is not set, the drive selection
code will fail. This can happen when a device has multiple LVs where some
of are used by Ceph and at least one LV isn't used by Ceph." so maybe you
can start there in terms of finding a potential workaround for now.
On Wed, Aug 16, 2023 at 12:05 PM Adam Huffman <adam.huffman.lists(a)gmail.com>
wrote:
I've been having fun today trying to invite a new
disk that replaced a
failing one into a cluster.
One of my attempts to apply an OSD spec was clearly wrong, because I now
have this error:
Module 'cephadm' has failed: 'osdspec_affinity'
and this was the traceback in the mgr logs:
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work
return f(*arg)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 224, in refresh
r = self._refresh_host_devices(host)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 396, in
_refresh_host_devices
self.update_osdspec_previews(host)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in
update_osdspec_previews
previews.extend(self.mgr.osd_service.get_previews(search_host))
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 258, in
get_previews
return self.generate_previews(osdspecs, host)
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 291, in
generate_previews
for host, ds in self.prepare_drivegroup(osdspec):
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 225, in
prepare_drivegroup
existing_daemons=len(dd_for_spec_and_host))
File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
line 35, in __init__
self._data = self.assign_devices('data_devices',
self.spec.data_devices)
File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
line 19, in wrapper
return f(self, ds)
File
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
line 134, in assign_devices
if lv['osdspec_affinity'] != self.spec.service_id:
KeyError: 'osdspec_affinity'
This cluster is running 16.2.13.
The exported service spec is:
service_type: osd
service_id: osd_spec-0.3
service_name: osd.osd_spec-0.3
placement:
host_pattern: cepho-*
spec:
data_devices:
rotational: true
db_devices:
model: SSDPE2KE032T8L
encrypted: true
filter_logic: AND
objectstore: bluestore
Best Wishes,
Adam
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:59 p.m.
New subject: Messenger v2 Connection mode config options
After reviewing:
https://docs.ceph.com/en/reef/rados/configuration/msgr2/#connection-mode-co…
I am still confused about the difference between:
ms_service_mode•<https://docs.ceph.com/en/reef/rados/configuration/msgr2…
a list of permitted modes for clients to use when connecting to the cluster.
and
ms_client_mode•<https://docs.ceph.com/en/reef/rados/configuration/msgr2/…
a list of connection modes, in order of preference, for clients to use (or allow) when
talking to a Ceph cluster.
Is there a difference between clients “connecting to the cluster” and “use when talking to
a Ceph cluster”? Or are there different categories of “clients” here?
Thank you,
Josh Beaman
31 Aug
31 Aug
6:34 p.m.
In the end, forcing a restart of cephadm at least cleared the "error"
status, and nodes were not wrongly shown as offline (another symptom of the
cephadm state).
One pending OSD creation did proceed.
It still keeps trying to apply the rule and logs it as a warning now rather
than crashing.
On Wed, 16 Aug 2023 at 18:03, Adam King <adking(a)redhat.com> wrote:
it looks like you've hit
https://tracker.ceph.com/issues/58946 which has
a candidate fix open, but nothing merged. The description on the PR with
the candidate fix says "When osdspec_affinity is not set, the drive
selection code will fail. This can happen when a device has multiple LVs
where some of are used by Ceph and at least one LV isn't used by Ceph." so
maybe you can start there in terms of finding a potential workaround for
now.
On Wed, Aug 16, 2023 at 12:05 PM Adam Huffman <
adam.huffman.lists(a)gmail.com> wrote:
> I've been having fun today trying to invite a new disk that replaced a
> failing one into a cluster.
>
> One of my attempts to apply an OSD spec was clearly wrong, because I now
> have this error:
>
> Module 'cephadm' has failed: 'osdspec_affinity'
>
> and this was the traceback in the mgr logs:
>
> Traceback (most recent call last):
> File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work
> return f(*arg)
> File "/usr/share/ceph/mgr/cephadm/serve.py", line 224, in refresh
> r = self._refresh_host_devices(host)
> File "/usr/share/ceph/mgr/cephadm/serve.py", line 396, in
> _refresh_host_devices
> self.update_osdspec_previews(host)
> File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in
> update_osdspec_previews
> previews.extend(self.mgr.osd_service.get_previews(search_host))
> File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 258, in
> get_previews
> return self.generate_previews(osdspecs, host)
> File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 291, in
> generate_previews
> for host, ds in self.prepare_drivegroup(osdspec):
> File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 225, in
> prepare_drivegroup
> existing_daemons=len(dd_for_spec_and_host))
> File
>
>
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
> line 35, in __init__
> self._data = self.assign_devices('data_devices',
> self.spec.data_devices)
> File
>
>
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
> line 19, in wrapper
> return f(self, ds)
> File
>
>
"/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py",
> line 134, in assign_devices
> if lv['osdspec_affinity'] != self.spec.service_id:
> KeyError: 'osdspec_affinity'
>
> This cluster is running 16.2.13.
>
> The exported service spec is:
>
> service_type: osd
> service_id: osd_spec-0.3
> service_name: osd.osd_spec-0.3
> placement:
> host_pattern: cepho-*
> spec:
> data_devices:
> rotational: true
> db_devices:
> model: SSDPE2KE032T8L
> encrypted: true
> filter_logic: AND
> objectstore: bluestore
>
> Best Wishes,
> Adam
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
262
days inactive
277
days old
3 comments
3 participants
participants (3)
-
Adam Huffman -
Adam King -
Beaman, Joshua