22 Mar
2021
22 Mar
'21
7:22 p.m.
Hello,
follow up from my mail from 2020 [0], it seems that OSDs sometimes have
"multiple classes" assigned:
[15:47:15] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush rm-device-class osd.4
done removing class of osd(s): 4
[15:47:17] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush rm-device-class osd.4
osd.4 belongs to no class,
[15:47:20] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush set-device-class xruk
osd.4
set osd(s) 4 to class 'xruk'
[15:47:45] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush set-device-class xruk
osd.4
osd.4 already set to class xruk. set-device-class item id 4 name 'osd.4'
device_class 'xruk': no change.
[15:47:47] server6.place6:/var/lib/ceph/osd/ceph-4# /usr/bin/ceph-osd -i 4 --pid-file
/var/run/ceph/osd.4.pid -c /etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup
ceph
2021-03-22 15:48:02.773 7fe2f81e4d80 -1 osd.4 94608 log_to_monitors {default=true}
2021-03-22 15:48:02.777 7fe2f81e4d80 -1 osd.4 94608 mon_cmd_maybe_osd_create fail:
'osd.4 has already bound to class 'xruk', can not reset class to
'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class
first': (16) Device or resource busy
[15:48:02] server6.place6:/var/lib/ceph/osd/ceph-4#
[15:48:02] server6.place6:/var/lib/ceph/osd/ceph-4#
We are running ceph 14.2.9.
As written before, it also seems that the affected OSD is peering with
OSDs from the wrong class (hdd). Does anyone have a hint on how to fix
this?
Best regards,
Nico
[0]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SFJVJI5XUD7…
--
Sustainable and modern Infrastructures by ungleich.ch
23 Mar
23 Mar
1:21 p.m.
On 3/22/21 3:52 PM, Nico Schottelius wrote:
Hello,
follow up from my mail from 2020 [0], it seems that OSDs sometimes have
"multiple classes" assigned:
[15:47:15] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush rm-device-class osd.4
done removing class of osd(s): 4
[15:47:17] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush rm-device-class osd.4
osd.4 belongs to no class,
[15:47:20] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush set-device-class xruk
osd.4
set osd(s) 4 to class 'xruk'
[15:47:45] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush set-device-class xruk
osd.4
osd.4 already set to class xruk. set-device-class item id 4 name 'osd.4'
device_class 'xruk': no change.
[15:47:47] server6.place6:/var/lib/ceph/osd/ceph-4# /usr/bin/ceph-osd -i 4 --pid-file
/var/run/ceph/osd.4.pid -c /etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup
ceph
2021-03-22 15:48:02.773 7fe2f81e4d80 -1 osd.4 94608 log_to_monitors {default=true}
2021-03-22 15:48:02.777 7fe2f81e4d80 -1 osd.4 94608 mon_cmd_maybe_osd_create fail:
'osd.4 has already bound to class 'xruk', can not reset class to
'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class
first': (16) Device or resource busy
[15:48:02] server6.place6:/var/lib/ceph/osd/ceph-4#
[15:48:02] server6.place6:/var/lib/ceph/osd/ceph-4#
We are running ceph 14.2.9.
As written before, it also seems that the affected OSD is peering with
OSDs from the wrong class (hdd). Does anyone have a hint on how to fix
this?
Do you have: osd_class_update_on_start enabled?
On our cluster NVMe OSDs would try to wrongly add themselves to "SSD"
class (which didn't succeed). But maybe sometimes your OSDs do manage to
put themselve in a wrong class? Just guessing. But I would turn that
off. The same for this parameter:
osd_crush_update_on_start
Gr. Stefan
2:30 p.m.
Stefan Kooman <stefan(a)bit.nl> writes:
So this one is a bit funky. It seems to be off, but the behaviour would
indicate it isn't. Checking the typical configurations:
[10:38:53] black2.place6:~# ceph config-key get config/global/osd_class_update_on_start;
echo ""
obtained 'config/global/osd_class_update_on_start'
false
[10:39:59] black2.place6:~# ceph-conf -D | grep osd_class_update_on_start
osd_class_update_on_start = true
[10:47:24] black2.place6:~# grep osd_class_update_on_start /etc/ceph/ceph.conf
[10:52:59] black2.place6:~# ceph config dump | grep osd_class_update_on_start
global advanced osd_class_update_on_start false
[10:53:38] black2.place6:~#
So it looks like it's already disabled.
I am not sure where ceph-conf reads the value true from, but I assume
it's a builtin.
I was also searching for osd_class_update_on_start in the Internet and
it seems there is no reference to it in the ceph documentation. Do you
have any pointers to it?
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
OSDs from the
wrong class (hdd). Does anyone have a hint on how to fix
this?
Do you have: osd_class_update_on_start enabled?
3:04 p.m.
On 3/23/21 11:00 AM, Nico Schottelius wrote:
Stefan Kooman <stefan(a)bit.nl> writes:
So this one is a bit funky. It seems to be off, but the behaviour would
indicate it isn't. Checking the typical configurations:
[10:38:53] black2.place6:~# ceph config-key get config/global/osd_class_update_on_start;
echo ""
obtained 'config/global/osd_class_update_on_start'
false
[10:39:59] black2.place6:~# ceph-conf -D | grep osd_class_update_on_start
osd_class_update_on_start = true
[10:47:24] black2.place6:~# grep osd_class_update_on_start /etc/ceph/ceph.conf
[10:52:59] black2.place6:~# ceph config dump | grep osd_class_update_on_start
global advanced osd_class_update_on_start false
[10:53:38] black2.place6:~#
So it looks like it's already disabled.
What does a "ceph daemon osd.$id config get osd_class_update_on_start"
give on that host for an OSD that is running there?
It depends on settings on logging of the OSD daemons, but in our case it
was logged to the daemon log I believe (or syslog, dunno anymore).
OSDs from
the wrong class (hdd). Does anyone have a hint on how to fix
this?
Do you have: osd_class_update_on_start enabled?
I am not sure where ceph-conf reads the value true from, but I assume
it's a builtin.
I was also searching for osd_class_update_on_start in the Internet and
it seems there is no reference to it in the ceph documentation. Do you
have any pointers to it?
Not anymore with new Ceph documentation. But the parameter is self
explaining, it will try to put itself into the proper class at startup.
Source code: src/common/options.cc
Option("osd_class_update_on_start", Option::TYPE_BOOL,
Option::LEVEL_ADVANCED)
.set_default(true)
.set_description("set OSD device class on startup"),
Gr. Stefan
25 Mar
25 Mar
4:35 p.m.
Stefan Kooman <stefan(a)bit.nl> writes:
Out of curiosity, do you have any clue why it's not in there anymore?
On 3/23/21 11:00 AM, Nico Schottelius wrote:
That returns
[12:52:24] server6.place6:~# ceph daemon osd.4 config get osd_class_update_on_start
{
"osd_class_update_on_start": "false"
}
for all involved OSDs.
Stefan Kooman <stefan(a)bit.nl> writes:
So this one is a bit funky. It
seems to be off, but the behaviour
would
indicate it isn't. Checking the typical configurations:
[10:38:53] black2.place6:~# ceph config-key get
config/global/osd_class_update_on_start; echo ""
obtained 'config/global/osd_class_update_on_start'
false
[10:39:59] black2.place6:~# ceph-conf -D | grep
osd_class_update_on_start
osd_class_update_on_start = true
[10:47:24] black2.place6:~# grep osd_class_update_on_start
/etc/ceph/ceph.conf
[10:52:59] black2.place6:~# ceph config dump | grep
osd_class_update_on_start
global advanced osd_class_update_on_start false
[10:53:38] black2.place6:~#
So it looks like it's already disabled.
What does a "ceph daemon osd.$id config get osd_class_update_on_start"
give on that host for an OSD that is running there? OSDs from
the wrong class (hdd). Does anyone have a hint on how to fix
this?
Do you have: osd_class_update_on_start enabled? It depends on settings on logging of the OSD daemons,
but in our case
it was logged to the daemon log I believe (or syslog, dunno anymore).
It's so strange, because none of the configurations indicate to use a
"hdd" class. Which, btw, we also don't use in other cases (i.e. none of
our used classes is hdd), so I suspect some builtin to try to setup the
class.
I am not sure
where ceph-conf reads the value true from, but I
assume
it's a builtin.
I was also searching for osd_class_update_on_start in the Internet
and
it seems there is no reference to it in the ceph documentation. Do you
have any pointers to it?
Not anymore with new Ceph documentation. But the parameter is self
explaining, it will try to put itself into the proper class at
startup. Source code: src/common/options.cc
Option("osd_class_update_on_start", Option::TYPE_BOOL,
Option::LEVEL_ADVANCED)
.set_default(true)
.set_description("set OSD device class on startup"),
The description I am somewhat missing is "set based on which criteria?"
In any case, it seems that the running OSD has the correct class
assigned. However I can see that that OSD has connections open to
unrelated osds:
tcp6 0 0 2a0a:e5c0:2:1:21b:21ff:febc:bf30:6805
2a0a:e5c0:2:1:21b:21ff:febb:68dc:57280 ESTABLISHED 17034/ceph-osd
So something is "not good" or "not correct" with this osd. This
particular one is in a special class that serves 1 pool with only 3 osds
in it. However this osd has around 200 connections established to what I
can see most (all?) other osds in the cluster.
To my understanding, it seems wrong that ceph osds form a complete mesh,
especially if they will never exchange data with the osds they are
connected to.
Can somebody confirm that osds should only connect to osds they share
data with?
And if my assumption is correct: is there any way to tell this osd to
behave correctly and only establish connections to osds of the same
class? (i.e. correctly assigning the class)
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
30 Mar
30 Mar
6:12 p.m.
On 3/25/21 1:05 PM, Nico Schottelius wrote:
Out of curiosity, do you have any clue why it's not in there anymore?
It might still be, but I cannot find it anymore ...
Gr. Stefan
it seems there is no reference to it in the ceph
documentation. Do you
have any pointers to it?
Not anymore with new Ceph documentation.
1141
days inactive
1149
days old
6 comments
3 participants
participants (3)
-
Konstantin Shalygin -
Nico Schottelius -
Stefan Kooman