22 Sep
2020
22 Sep
'20
3:59 p.m.
Hello,
recently we wanted to re-adjust rebalancing speed in one cluster with
ceph tell osd.* injectargs '--osd-max-backfills 4'
ceph tell osd.* injectargs '--osd-recovery-max-active 4'
The first osds responded and after about 6-7 osds ceph tell stopped
progressing, just after it encountered a dead osd (osd.10). We have
since then removed osd.10 and all osds in the cluster are up.
However as soon as we issue either of the above tell commands, it just
hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in
"Activating" and "Peering" states.
It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few
minutes later the pgs are peered/active.
We can reproduce this problem also with very busy osds, which have been
moved to another host - they also do not react to the ceph tell commands.
We are mostly on 14.2.9, besides the rgw:
[16:44:47] black2.place6:~# ceph versions
{
"mon": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 3
},
"mgr": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 3
},
"osd": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 85
},
"mds": {},
"rgw": {
"ceph version 20200428-923-g4004f081ec
(4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1
},
"overall": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 91,
"ceph version 20200428-923-g4004f081ec
(4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1
}
}
Did anyone see this before and/or do you have a hint on how to debug
ceph tell as it is not a daemon on its own?
Best regards,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
22 Sep
22 Sep
4:53 p.m.
Hi,
However as soon as we issue either of the above tell
commands, it just
hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in
"Activating" and "Peering" states.
It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few
minutes later the pgs are peered/active.
We can reproduce this problem also with very busy osds, which have been
moved to another host - they also do not react to the ceph tell commands.
Does this also happen when you issue a osd specific "tell", i.e. ceph
tell osd.13 injectargs '--osd-max-backfills 4'
Does this also happen when you loop over it one by one?
Did anyone see this before and/or do you have a hint
on how to debug
ceph tell as it is not a daemon on its own?
IIRC I have seen this, but not in combination with PGs peering /
activating. Has the config change become effective on alls OSDs: verify
with ceph daemon osd.13 config get osd_max_backfills (for all OSDs)
Gr. Stefan
5:07 p.m.
Hello Stefan,
Stefan Kooman <stefan(a)bit.nl> writes:
Just checked - most OSDs did not apply the new setting, setting it
explicitly on them works however.
Best regards,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Hi,
It does hang for some of them, but if I "ping" / select specific OSDs,
this does not happen.
However as soon as we issue either of the above
tell commands, it just
hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in
"Activating" and "Peering" states.
It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few
minutes later the pgs are peered/active.
We can reproduce this problem also with very busy osds, which have been
moved to another host - they also do not react to the ceph tell commands.
Does this also happen when you issue a osd specific "tell", i.e. ceph
tell osd.13 injectargs '--osd-max-backfills 4'
Does this also happen when you loop over it one by one? Did anyone see
this before and/or do you have a hint on how to debug
ceph tell as it is not a daemon on its own?
IIRC I have seen this, but not in combination with PGs peering /
activating. Has the config change become effective on alls OSDs: verify
with ceph daemon osd.13 config get osd_max_backfills (for all OSDs)
8:09 p.m.
I started now to iterate over all osds in the tree and some of the osds
are completely unresponsive:
[18:27:18] black1.place6:~# for osd in $(ceph osd tree | grep osd. | awk '{ print $4
}'); do echo $osd; ceph tell $osd injectargs '--osd-max-backfills 1'; done
osd.20
osd.56
osd.62
osd.63
^CTraceback (most recent call last):
File "/usr/bin/ceph", line 1266, in <module>
retval = main()
File "/usr/bin/ceph", line 1182, in main
prefix='get_command_descriptions')
File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1459, in
json_command
inbuf, timeout, verbose)
File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1329, in
send_command_retry
return send_command(*args, **kwargs)
File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1361, in
send_command
cluster.osd_command, osdid, cmd, inbuf, timeout=timeout)
File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1311, in
run_in_thread
t.join(timeout=timeout)
File "/usr/lib/python3.7/threading.py", line 1036, in join
self._wait_for_tstate_lock(timeout=max(timeout, 0))
File "/usr/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
osd.64
osd.65
What's the best way to figure out why osd.63 does not react to the tell
command?
Best regards,
Nico
Nico Schottelius <nico.schottelius(a)ungleich.ch> writes:
Hello Stefan,
Stefan Kooman <stefan(a)bit.nl> writes:
Just checked - most OSDs did not apply the new setting, setting it
explicitly on them works however.
Best regards,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Hi,
It does hang for some of them, but if I "ping" / select specific OSDs,
this does not happen.
However as soon as we issue either of the above
tell commands, it just
hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in
"Activating" and "Peering" states.
It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few
minutes later the pgs are peered/active.
We can reproduce this problem also with very busy osds, which have been
moved to another host - they also do not react to the ceph tell commands.
Does this also happen when you issue a osd specific "tell", i.e. ceph
tell osd.13 injectargs '--osd-max-backfills 4'
Does this also happen when you loop over it one by one? Did
anyone see this before and/or do you have a hint on how to debug
ceph tell as it is not a daemon on its own?
IIRC I have seen this, but not in combination with PGs peering /
activating. Has the config change become effective on alls OSDs: verify
with ceph daemon osd.13 config get osd_max_backfills (for all OSDs)
8:32 p.m.
Update to the update: currently debugging why pgs are stuck in the
peering state:
[18:57:49] black2.place6:~# ceph pg dump all | grep 2.7d1
dumped all
2.7d1 16666 0 0 0 0 69698617344 0
0 3002 3002 peering
2020-09-22 18:49:28.587859 80407'8126117 80915:35142541 [22,84] 22
[22,84] 22 80407'8126117 2020-09-22 17:23:11.860334 79594'8122364
2020-09-21 13:27:16.376009 0
The problem is that
ceph pg 2.7d1 query
hangs and does not output information. Does anyone know what could be
the cause for this?
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
9:05 p.m.
So the same problem happens with pgs which are in "unknown" state,
[19:31:08] black2.place6:~# ceph pg 2.5b2 query | tee query_2.5b2
hangs until the pg actually because active again. I assume that this
should not be the case, should it?
Nico Schottelius <nico.schottelius(a)ungleich.ch> writes:
Update to the update: currently debugging why pgs are
stuck in the
peering state:
[18:57:49] black2.place6:~# ceph pg dump all | grep 2.7d1
dumped all
2.7d1 16666 0 0 0 0 69698617344 0
0 3002 3002 peering
2020-09-22 18:49:28.587859 80407'8126117 80915:35142541 [22,84] 22
[22,84] 22 80407'8126117 2020-09-22 17:23:11.860334 79594'8122364
2020-09-21 13:27:16.376009 0
The problem is that
ceph pg 2.7d1 query
hangs and does not output information. Does anyone know what could be
the cause for this?
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
23 Sep
23 Sep
2:02 a.m.
Follow up on the tell hanging: iterating over all osds and trying to
raise the max-backfills gives hanging ceph tell processes like this:
root 1007846 15.3 1.2 918388 50972 pts/5 Sl 00:03 0:48 /usr/bin/python3
/usr/bin/ceph tell osd.4 injectargs --osd-max-backfill
root 1007890 0.4 0.9 850664 37596 pts/5 Sl 00:03 0:01 /usr/bin/python3
/usr/bin/ceph tell osd.7 injectargs --osd-max-backfill
root 1007930 0.3 0.9 842472 37484 pts/5 Sl 00:03 0:01 /usr/bin/python3
/usr/bin/ceph tell osd.11 injectargs --osd-max-backfil
root 1007987 0.3 0.9 850668 37540 pts/5 Sl 00:03 0:01 /usr/bin/python3
/usr/bin/ceph tell osd.18 injectargs --osd-max-backfil
root 1008054 0.4 0.9 850664 37600 pts/5 Sl 00:03 0:01 /usr/bin/python3
/usr/bin/ceph tell osd.29 injectargs --osd-max-backfil
root 1008147 14.7 1.2 910192 50648 pts/5 Sl 00:03 0:42 /usr/bin/python3
/usr/bin/ceph tell osd.33 injectargs --osd-max-backfil
root 1008205 0.3 0.9 842468 37524 pts/5 Sl 00:03 0:01 /usr/bin/python3
/usr/bin/ceph tell osd.45 injectargs --osd-max-backfil
root 1008246 0.3 0.9 850664 37828 pts/5 Sl 00:04 0:01 /usr/bin/python3
/usr/bin/ceph tell osd.48 injectargs --osd-max-backfil
...
Additionally many of the tell processes get into an infinite loop and
print this error over and over again:
2020-09-23 00:09:48.766 7f07e5f99700 0 --1-
[2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >>
v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c8055680 0x7f07c8053740 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got
BADAUTHORIZER
2020-09-23 00:09:48.774 7f07e5f99700 0 --1-
[2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >>
v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c804f590 0x7f07c80505c0 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got
BADAUTHORIZER
2020-09-23 00:09:48.786 7f07e5f99700 0 --1-
[2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >>
v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c8055680 0x7f07c8053740 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got
BADAUTHORIZER
2020-09-23 00:09:48.790 7f07e5f99700 0 --1-
[2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >>
v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c804f590 0x7f07c80505c0 :-1
s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got
BADAUTHORIZER
2020-09-23 00:09:48.798 7f07e5f99700 0 --1-
[2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >>
v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c8055680
0x7f07c8053740 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0
l=1).handle_connect_reply_2 connect got BADAUTHORIZ
Nico Schottelius <nico.schottelius(a)ungleich.ch> writes:
So the same problem happens with pgs which are in
"unknown" state,
[19:31:08] black2.place6:~# ceph pg 2.5b2 query | tee query_2.5b2
hangs until the pg actually because active again. I assume that this
should not be the case, should it?
Nico Schottelius <nico.schottelius(a)ungleich.ch> writes:
> Update to the update: currently debugging why pgs are stuck in the
> peering state:
>
> [18:57:49] black2.place6:~# ceph pg dump all | grep 2.7d1
> dumped all
> 2.7d1 16666 0 0 0 0 69698617344 0
0 3002 3002
peering 2020-09-22 18:49:28.587859 80407'8126117 80915:35142541 [22,84]
22 [22,84] 22 80407'8126117 2020-09-22 17:23:11.860334
79594'8122364 2020-09-21 13:27:16.376009 0
>
> The problem is that
>
> ceph pg 2.7d1 query
>
> hangs and does not output information. Does anyone know what could be
> the cause for this?
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
1331
days inactive
1331
days old
6 comments
2 participants
participants (2)
-
Nico Schottelius -
Stefan Kooman