ceph orch upgrade stuck between 16.2.7 and 16.2.13
Hi, A healthy 16.2.7 cluster should get an upgrade to 16.2.13. ceph orch upgrade start --ceph-version 16.2.13 did upgrade MONs, MGRs and 25% of the OSDs and is now stuck. We tried several "ceph orch upgrade stop" and starts again. We "failed" the active MGR but no progress. We set the debug logging with "ceph config set mgr mgr/cephadm/log_to_cluster_level debug" but it only tells that it starts: 2023-08-15T09:05:58.548896+0200 mgr.cephmon01 [INF] Upgrade: Started with target quay.io/ceph/ceph:v16.2.13 How can we check what is happening (or not happening) here? How do we get cephadm to complete the task? Current status is: # ceph orch upgrade status { "target_image": "quay.io/ceph/ceph:v16.2.13", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false } # ceph -s cluster: id: 3098199a-c7f5-4baf-901c-f178131be6f4 health: HEALTH_WARN There are daemons running an older version of ceph services: mon: 5 daemons, quorum cephmon02,cephmon01,cephmon03,cephmon04,cephmon05 (age 4d) mgr: cephmon03(active, since 8d), standbys: cephmon01, cephmon02 mds: 2/2 daemons up, 1 standby, 2 hot standby osd: 202 osds: 202 up (since 11d), 202 in (since 13d) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 2/2 healthy pools: 11 pools, 4961 pgs objects: 98.84M objects, 347 TiB usage: 988 TiB used, 1.3 PiB / 2.3 PiB avail pgs: 4942 active+clean 19 active+clean+scrubbing+deep io: client: 89 MiB/s rd, 598 MiB/s wr, 25 op/s rd, 157 op/s wr progress: Upgrade to quay.io/ceph/ceph:v16.2.13 (0s) [............................] # ceph versions { "mon": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3 }, "osd": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 48, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 154 }, "mds": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 5 }, "rgw": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "overall": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 56, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 161 } } Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, literally minutes before your email popped up in my inbox I had announced that I would upgrade our cluster from 16.2.10 to 16.2.13 tomorrow. Now I'm hesitating. ;-) I guess I would start looking on the nodes where it failed to upgrade OSDs and check out the cephadm.log as well as syslog. Did you see progress messages in the mgr log for the successfully updated OSDs (or MON/MGR)? Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
A healthy 16.2.7 cluster should get an upgrade to 16.2.13.
ceph orch upgrade start --ceph-version 16.2.13
did upgrade MONs, MGRs and 25% of the OSDs and is now stuck.
We tried several "ceph orch upgrade stop" and starts again. We "failed" the active MGR but no progress. We set the debug logging with "ceph config set mgr mgr/cephadm/log_to_cluster_level debug" but it only tells that it starts:
2023-08-15T09:05:58.548896+0200 mgr.cephmon01 [INF] Upgrade: Started with target quay.io/ceph/ceph:v16.2.13
How can we check what is happening (or not happening) here? How do we get cephadm to complete the task?
Current status is:
# ceph orch upgrade status { "target_image": "quay.io/ceph/ceph:v16.2.13", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false }
# ceph -s cluster: id: 3098199a-c7f5-4baf-901c-f178131be6f4 health: HEALTH_WARN There are daemons running an older version of ceph services: mon: 5 daemons, quorum cephmon02,cephmon01,cephmon03,cephmon04,cephmon05 (age 4d) mgr: cephmon03(active, since 8d), standbys: cephmon01, cephmon02 mds: 2/2 daemons up, 1 standby, 2 hot standby osd: 202 osds: 202 up (since 11d), 202 in (since 13d) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 2/2 healthy pools: 11 pools, 4961 pgs objects: 98.84M objects, 347 TiB usage: 988 TiB used, 1.3 PiB / 2.3 PiB avail pgs: 4942 active+clean 19 active+clean+scrubbing+deep io: client: 89 MiB/s rd, 598 MiB/s wr, 25 op/s rd, 157 op/s wr progress: Upgrade to quay.io/ceph/ceph:v16.2.13 (0s) [............................]
# ceph versions { "mon": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3 }, "osd": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 48, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 154 }, "mds": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 5 }, "rgw": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "overall": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 56, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 161 } }
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 8/15/23 11:02, Eugen Block wrote:
I guess I would start looking on the nodes where it failed to upgrade OSDs and check out the cephadm.log as well as syslog. Did you see progress messages in the mgr log for the successfully updated OSDs (or MON/MGR)?
The issue is that there is no information on which OSD cephadm tries to upgrade next. There is no failure reported. It seems to just sit there and wait for something. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
with the log to cluster level already on debug, if you do a "ceph mgr fail" what does cephadm log to the cluster before it reports sleeping? It should at least be doing something if it's responsive at all. Also, in "ceph orch ps" and "ceph orch device ls" are the REFRESHED columns reporting that they've refreshed the info recently (last 10 minutes for daemons, last 30 minutes for devices)? On Tue, Aug 15, 2023 at 3:46 AM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
A healthy 16.2.7 cluster should get an upgrade to 16.2.13.
ceph orch upgrade start --ceph-version 16.2.13
did upgrade MONs, MGRs and 25% of the OSDs and is now stuck.
We tried several "ceph orch upgrade stop" and starts again. We "failed" the active MGR but no progress. We set the debug logging with "ceph config set mgr mgr/cephadm/log_to_cluster_level debug" but it only tells that it starts:
2023-08-15T09:05:58.548896+0200 mgr.cephmon01 [INF] Upgrade: Started with target quay.io/ceph/ceph:v16.2.13
How can we check what is happening (or not happening) here? How do we get cephadm to complete the task?
Current status is:
# ceph orch upgrade status { "target_image": "quay.io/ceph/ceph:v16.2.13", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false }
# ceph -s cluster: id: 3098199a-c7f5-4baf-901c-f178131be6f4 health: HEALTH_WARN There are daemons running an older version of ceph
services: mon: 5 daemons, quorum cephmon02,cephmon01,cephmon03,cephmon04,cephmon05 (age 4d) mgr: cephmon03(active, since 8d), standbys: cephmon01, cephmon02 mds: 2/2 daemons up, 1 standby, 2 hot standby osd: 202 osds: 202 up (since 11d), 202 in (since 13d) rgw: 2 daemons active (2 hosts, 1 zones)
data: volumes: 2/2 healthy pools: 11 pools, 4961 pgs objects: 98.84M objects, 347 TiB usage: 988 TiB used, 1.3 PiB / 2.3 PiB avail pgs: 4942 active+clean 19 active+clean+scrubbing+deep
io: client: 89 MiB/s rd, 598 MiB/s wr, 25 op/s rd, 157 op/s wr
progress: Upgrade to quay.io/ceph/ceph:v16.2.13 (0s) [............................]
# ceph versions { "mon": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3 }, "osd": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 48, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 154 }, "mds": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 5 }, "rgw": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "overall": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 56, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 161 } }
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 8/15/23 16:36, Adam King wrote:
with the log to cluster level already on debug, if you do a "ceph mgr fail" what does cephadm log to the cluster before it reports sleeping? It should at least be doing something if it's responsive at all. Also, in "ceph orch ps" and "ceph orch device ls" are the REFRESHED columns reporting that they've refreshed the info recently (last 10 minutes for daemons, last 30 minutes for devices)?
They have been refreshed very recently. The issue seems to be a bit larger than just the not working upgrade. We are now not even able to restart a daemon. When I issue the command # ceph orch daemon restart crash.cephmon01 these two lines show up in the cephadm log but nothing else happens: 2023-08-16T10:35:41.640027+0200 mgr.cephmon01 [INF] Schedule restart daemon crash.cephmon01 2023-08-16T10:35:41.640497+0200 mgr.cephmon01 [DBG] _kick_serve_loop The container for crash.cephmon01 does not get restarted. It looks like the service loop does not get executed. Can we see what jobs are in this queue and why they do not get executed? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
I don't really have a good idea right now, but there was a thread [1] about ssh sessions that are not removed, maybe that could have such an impact? And if you crank up the debug level to 30, do you see anything else? ceph config set mgr debug_mgr 30 [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/I452F3PWBAU4... Zitat von Robert Sander <r.sander@heinlein-support.de>:
On 8/15/23 16:36, Adam King wrote:
with the log to cluster level already on debug, if you do a "ceph mgr fail" what does cephadm log to the cluster before it reports sleeping? It should at least be doing something if it's responsive at all. Also, in "ceph orch ps" and "ceph orch device ls" are the REFRESHED columns reporting that they've refreshed the info recently (last 10 minutes for daemons, last 30 minutes for devices)?
They have been refreshed very recently.
The issue seems to be a bit larger than just the not working upgrade.
We are now not even able to restart a daemon.
When I issue the command
# ceph orch daemon restart crash.cephmon01
these two lines show up in the cephadm log but nothing else happens:
2023-08-16T10:35:41.640027+0200 mgr.cephmon01 [INF] Schedule restart daemon crash.cephmon01 2023-08-16T10:35:41.640497+0200 mgr.cephmon01 [DBG] _kick_serve_loop
The container for crash.cephmon01 does not get restarted.
It looks like the service loop does not get executed.
Can we see what jobs are in this queue and why they do not get executed?
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 8/16/23 12:10, Eugen Block wrote:
I don't really have a good idea right now, but there was a thread [1] about ssh sessions that are not removed, maybe that could have such an impact? And if you crank up the debug level to 30, do you see anything else?
It was something similar. There were leftover ceph-volume processes running on some of the OSD nodes. After killing them the cephadm orchestrator is now able to resume the upgrade. As we also restarted the MGR processes (with systemctl restart CONTAINER) there were no leftover SSH sessions. But the still running ceph-volume processes must have used a lock that blocked new cephadm commands. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Great, thanks for the update! Just yesterday I wanted to cleanup a couple of test clusters and remove some old container images which seemed to still be in use although several upgrades had been processed. Those were quite old ceph-volume inventory processes, dating back to the initial cluster bootstrap. But obviously, they didn't have such an impact as you describe. Anyway, good to know that it's not a major issue so I can upgrade our cluster as well. Although I'm waiting for a PR that still didn't make it into latest pacific, so maybe I'll wait for a bit longer. Thanks! Eugen Zitat von Robert Sander <r.sander@heinlein-support.de>:
On 8/16/23 12:10, Eugen Block wrote:
I don't really have a good idea right now, but there was a thread [1] about ssh sessions that are not removed, maybe that could have such an impact? And if you crank up the debug level to 30, do you see anything else?
It was something similar. There were leftover ceph-volume processes running on some of the OSD nodes. After killing them the cephadm orchestrator is now able to resume the upgrade.
As we also restarted the MGR processes (with systemctl restart CONTAINER) there were no leftover SSH sessions.
But the still running ceph-volume processes must have used a lock that blocked new cephadm commands.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I've seen this before where the ceph-volume process hanging causes the whole serve loop to get stuck (we have a patch to get it to timeout properly in reef and are backporting to quincy but nothing for pacific unfortunately). That's why I was asking about the REFRESHED column in the orch ps/ orch device ls output. Typically when this happens it presents as the REFRESHED column reporting not having refreshed anything since the ceph-volume process started hanging. Either way, if you killed those ceph-volume processes and any new ones aren't hanging and the serve loop is running okay I'd expect the issues to clear up. This could (and most likely did) cause both the daemon restarts to not happen and the upgrade to not progress. On Wed, Aug 16, 2023 at 8:50 AM Robert Sander <r.sander@heinlein-support.de> wrote:
On 8/16/23 12:10, Eugen Block wrote:
I don't really have a good idea right now, but there was a thread [1] about ssh sessions that are not removed, maybe that could have such an impact? And if you crank up the debug level to 30, do you see anything else?
It was something similar. There were leftover ceph-volume processes running on some of the OSD nodes. After killing them the cephadm orchestrator is now able to resume the upgrade.
As we also restarted the MGR processes (with systemctl restart CONTAINER) there were no leftover SSH sessions.
But the still running ceph-volume processes must have used a lock that blocked new cephadm commands.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Adam King
-
Eugen Block
-
Robert Sander