Hello, recently we wanted to re-adjust rebalancing speed in one cluster with ceph tell osd.* injectargs '--osd-max-backfills 4' ceph tell osd.* injectargs '--osd-recovery-max-active 4' The first osds responded and after about 6-7 osds ceph tell stopped progressing, just after it encountered a dead osd (osd.10). We have since then removed osd.10 and all osds in the cluster are up. However as soon as we issue either of the above tell commands, it just hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in "Activating" and "Peering" states. It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few minutes later the pgs are peered/active. We can reproduce this problem also with very busy osds, which have been moved to another host - they also do not react to the ceph tell commands. We are mostly on 14.2.9, besides the rgw: [16:44:47] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } } Did anyone see this before and/or do you have a hint on how to debug ceph tell as it is not a daemon on its own? Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Hi,
However as soon as we issue either of the above tell commands, it just hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in "Activating" and "Peering" states.
It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few minutes later the pgs are peered/active.
We can reproduce this problem also with very busy osds, which have been moved to another host - they also do not react to the ceph tell commands.
Does this also happen when you issue a osd specific "tell", i.e. ceph tell osd.13 injectargs '--osd-max-backfills 4' Does this also happen when you loop over it one by one?
Did anyone see this before and/or do you have a hint on how to debug ceph tell as it is not a daemon on its own?
IIRC I have seen this, but not in combination with PGs peering / activating. Has the config change become effective on alls OSDs: verify with ceph daemon osd.13 config get osd_max_backfills (for all OSDs) Gr. Stefan
Hello Stefan, Stefan Kooman <stefan@bit.nl> writes:
Hi,
However as soon as we issue either of the above tell commands, it just hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in "Activating" and "Peering" states.
It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few minutes later the pgs are peered/active.
We can reproduce this problem also with very busy osds, which have been moved to another host - they also do not react to the ceph tell commands.
Does this also happen when you issue a osd specific "tell", i.e. ceph tell osd.13 injectargs '--osd-max-backfills 4'
Does this also happen when you loop over it one by one?
It does hang for some of them, but if I "ping" / select specific OSDs, this does not happen.
Did anyone see this before and/or do you have a hint on how to debug ceph tell as it is not a daemon on its own?
IIRC I have seen this, but not in combination with PGs peering / activating. Has the config change become effective on alls OSDs: verify with ceph daemon osd.13 config get osd_max_backfills (for all OSDs)
Just checked - most OSDs did not apply the new setting, setting it explicitly on them works however. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
I started now to iterate over all osds in the tree and some of the osds are completely unresponsive: [18:27:18] black1.place6:~# for osd in $(ceph osd tree | grep osd. | awk '{ print $4 }'); do echo $osd; ceph tell $osd injectargs '--osd-max-backfills 1'; done osd.20 osd.56 osd.62 osd.63 ^CTraceback (most recent call last): File "/usr/bin/ceph", line 1266, in <module> retval = main() File "/usr/bin/ceph", line 1182, in main prefix='get_command_descriptions') File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1459, in json_command inbuf, timeout, verbose) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1329, in send_command_retry return send_command(*args, **kwargs) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1361, in send_command cluster.osd_command, osdid, cmd, inbuf, timeout=timeout) File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1311, in run_in_thread t.join(timeout=timeout) File "/usr/lib/python3.7/threading.py", line 1036, in join self._wait_for_tstate_lock(timeout=max(timeout, 0)) File "/usr/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock elif lock.acquire(block, timeout): KeyboardInterrupt osd.64 osd.65 What's the best way to figure out why osd.63 does not react to the tell command? Best regards, Nico Nico Schottelius <nico.schottelius@ungleich.ch> writes:
Hello Stefan,
Stefan Kooman <stefan@bit.nl> writes:
Hi,
However as soon as we issue either of the above tell commands, it just hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in "Activating" and "Peering" states.
It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few minutes later the pgs are peered/active.
We can reproduce this problem also with very busy osds, which have been moved to another host - they also do not react to the ceph tell commands.
Does this also happen when you issue a osd specific "tell", i.e. ceph tell osd.13 injectargs '--osd-max-backfills 4'
Does this also happen when you loop over it one by one?
It does hang for some of them, but if I "ping" / select specific OSDs, this does not happen.
Did anyone see this before and/or do you have a hint on how to debug ceph tell as it is not a daemon on its own?
IIRC I have seen this, but not in combination with PGs peering / activating. Has the config change become effective on alls OSDs: verify with ceph daemon osd.13 config get osd_max_backfills (for all OSDs)
Just checked - most OSDs did not apply the new setting, setting it explicitly on them works however.
Best regards,
Nico
-- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Update to the update: currently debugging why pgs are stuck in the peering state: [18:57:49] black2.place6:~# ceph pg dump all | grep 2.7d1 dumped all 2.7d1 16666 0 0 0 0 69698617344 0 0 3002 3002 peering 2020-09-22 18:49:28.587859 80407'8126117 80915:35142541 [22,84] 22 [22,84] 22 80407'8126117 2020-09-22 17:23:11.860334 79594'8122364 2020-09-21 13:27:16.376009 0 The problem is that ceph pg 2.7d1 query hangs and does not output information. Does anyone know what could be the cause for this? -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
So the same problem happens with pgs which are in "unknown" state, [19:31:08] black2.place6:~# ceph pg 2.5b2 query | tee query_2.5b2 hangs until the pg actually because active again. I assume that this should not be the case, should it? Nico Schottelius <nico.schottelius@ungleich.ch> writes:
Update to the update: currently debugging why pgs are stuck in the peering state:
[18:57:49] black2.place6:~# ceph pg dump all | grep 2.7d1 dumped all 2.7d1 16666 0 0 0 0 69698617344 0 0 3002 3002 peering 2020-09-22 18:49:28.587859 80407'8126117 80915:35142541 [22,84] 22 [22,84] 22 80407'8126117 2020-09-22 17:23:11.860334 79594'8122364 2020-09-21 13:27:16.376009 0
The problem is that
ceph pg 2.7d1 query
hangs and does not output information. Does anyone know what could be the cause for this?
-- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Follow up on the tell hanging: iterating over all osds and trying to raise the max-backfills gives hanging ceph tell processes like this: root 1007846 15.3 1.2 918388 50972 pts/5 Sl 00:03 0:48 /usr/bin/python3 /usr/bin/ceph tell osd.4 injectargs --osd-max-backfill root 1007890 0.4 0.9 850664 37596 pts/5 Sl 00:03 0:01 /usr/bin/python3 /usr/bin/ceph tell osd.7 injectargs --osd-max-backfill root 1007930 0.3 0.9 842472 37484 pts/5 Sl 00:03 0:01 /usr/bin/python3 /usr/bin/ceph tell osd.11 injectargs --osd-max-backfil root 1007987 0.3 0.9 850668 37540 pts/5 Sl 00:03 0:01 /usr/bin/python3 /usr/bin/ceph tell osd.18 injectargs --osd-max-backfil root 1008054 0.4 0.9 850664 37600 pts/5 Sl 00:03 0:01 /usr/bin/python3 /usr/bin/ceph tell osd.29 injectargs --osd-max-backfil root 1008147 14.7 1.2 910192 50648 pts/5 Sl 00:03 0:42 /usr/bin/python3 /usr/bin/ceph tell osd.33 injectargs --osd-max-backfil root 1008205 0.3 0.9 842468 37524 pts/5 Sl 00:03 0:01 /usr/bin/python3 /usr/bin/ceph tell osd.45 injectargs --osd-max-backfil root 1008246 0.3 0.9 850664 37828 pts/5 Sl 00:04 0:01 /usr/bin/python3 /usr/bin/ceph tell osd.48 injectargs --osd-max-backfil ... Additionally many of the tell processes get into an infinite loop and print this error over and over again: 2020-09-23 00:09:48.766 7f07e5f99700 0 --1- [2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c8055680 0x7f07c8053740 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2020-09-23 00:09:48.774 7f07e5f99700 0 --1- [2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c804f590 0x7f07c80505c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2020-09-23 00:09:48.786 7f07e5f99700 0 --1- [2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c8055680 0x7f07c8053740 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2020-09-23 00:09:48.790 7f07e5f99700 0 --1- [2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c804f590 0x7f07c80505c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2020-09-23 00:09:48.798 7f07e5f99700 0 --1- [2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/2338294673 >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6858/12824 conn(0x7f07c8055680 0x7f07c8053740 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZ Nico Schottelius <nico.schottelius@ungleich.ch> writes:
So the same problem happens with pgs which are in "unknown" state,
[19:31:08] black2.place6:~# ceph pg 2.5b2 query | tee query_2.5b2
hangs until the pg actually because active again. I assume that this should not be the case, should it?
Nico Schottelius <nico.schottelius@ungleich.ch> writes:
Update to the update: currently debugging why pgs are stuck in the peering state:
[18:57:49] black2.place6:~# ceph pg dump all | grep 2.7d1 dumped all 2.7d1 16666 0 0 0 0 69698617344 0 0 3002 3002 peering 2020-09-22 18:49:28.587859 80407'8126117 80915:35142541 [22,84] 22 [22,84] 22 80407'8126117 2020-09-22 17:23:11.860334 79594'8122364 2020-09-21 13:27:16.376009 0
The problem is that
ceph pg 2.7d1 query
hangs and does not output information. Does anyone know what could be the cause for this?
-- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
participants (2)
-
Nico Schottelius
-
Stefan Kooman