orch upgrade mgr starts too slow and is terminated?
Hi all, upon updating to 16.2.2 via cephadm the upgrade is being stuck on the first mgr Looking into this via docker logs I see that it is still loading modules when it is apparently terminated and restarted in a loop When pausing the update, the mgr succeeds to start with the new version, however when resuming the update, it seems to try to update it again even tho it already has the new version, leading to the exact same loop. Is there some setting or workaround to increase the time before it is attempted to be redeployed, or can this behavior be caused by something else? Greetings, Kai
Hi! I hit the same issue. This was a bug in 16.2.0 that wasn't completely fixed, but I think we have it this time. Kicking of a 16.2.3 build now to resolve the problem. (Basically, sometimes docker calls the image docker.io/ceph/ceph:foo and somethings it's ceph/ceph:foo, and our attempt to normalize missed one case.) sage On Thu, May 6, 2021 at 9:59 AM Kai Börnert <kai.boernert@posteo.de> wrote:
Hi all,
upon updating to 16.2.2 via cephadm the upgrade is being stuck on the first mgr
Looking into this via docker logs I see that it is still loading modules when it is apparently terminated and restarted in a loop
When pausing the update, the mgr succeeds to start with the new version, however when resuming the update, it seems to try to update it again even tho it already has the new version, leading to the exact same loop.
Is there some setting or workaround to increase the time before it is attempted to be redeployed, or can this behavior be caused by something else?
Greetings,
Kai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 06.05.21 um 17:18 schrieb Sage Weil:
I hit the same issue. This was a bug in 16.2.0 that wasn't completely fixed, but I think we have it this time. Kicking of a 16.2.3 build now to resolve the problem.
Great. I also hit that today. Thanks for fixing it quickly. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Hi, thanks for your explanation. I see the 16.2.3 build is published, but (of course) I cannot update to it, as the version I currently use does still have this bug. Is there some workaround/hack I can use to upgrade? So far I tried to directly specify the image names and go via --ceph-version, those attempts all end in the mgr restart loop. Could changing the installed docker version to the bundled one work? (ubuntu 18lts with docker-ce currently installed as described in https://docs.docker.com/engine/install/ubuntu/) Greetings, Kai On 5/6/21 5:18 PM, Sage Weil wrote:
Hi!
I hit the same issue. This was a bug in 16.2.0 that wasn't completely fixed, but I think we have it this time. Kicking of a 16.2.3 build now to resolve the problem.
(Basically, sometimes docker calls the image docker.io/ceph/ceph:foo and somethings it's ceph/ceph:foo, and our attempt to normalize missed one case.)
sage
On Thu, May 6, 2021 at 9:59 AM Kai Börnert <kai.boernert@posteo.de> wrote:
Hi all,
upon updating to 16.2.2 via cephadm the upgrade is being stuck on the first mgr
Looking into this via docker logs I see that it is still loading modules when it is apparently terminated and restarted in a loop
When pausing the update, the mgr succeeds to start with the new version, however when resuming the update, it seems to try to update it again even tho it already has the new version, leading to the exact same loop.
Is there some setting or workaround to increase the time before it is attempted to be redeployed, or can this behavior be caused by something else?
Greetings,
Kai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 07.05.21 um 10:42 schrieb Kai Börnert:
Hi, thanks for your explanation.
I see the 16.2.3 build is published, but (of course) I cannot update to it, as the version I currently use does still have this bug.
Is there some workaround/hack I can use to upgrade?
I had success with stopping the "looping" mgr container via "systemctl stop" on the node. Cephadm then switches to another MGR to continue the upgrade. After that I just started the stopped mgr container and the upgrade continued. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Thanks, this did work for me, (additionally I had to stop and restart the upgrade after stopping and restarting the first mgr manually, as it was in a failed no stdby mgr state), the cluster is now proceeding to updating the osds :) On 5/7/21 11:28 AM, Robert Sander wrote:
Am 07.05.21 um 10:42 schrieb Kai Börnert:
Hi, thanks for your explanation.
I see the 16.2.3 build is published, but (of course) I cannot update to it, as the version I currently use does still have this bug.
Is there some workaround/hack I can use to upgrade? I had success with stopping the "looping" mgr container via "systemctl stop" on the node. Cephadm then switches to another MGR to continue the upgrade. After that I just started the stopped mgr container and the upgrade continued.
Regards
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Kai Börnert
-
Robert Sander
-
Sage Weil