Upgrade 16.2.11 -> 17.2.0 failed
Dear List, Today i was sucessfully upgrading with cephadm from 16.2.8 -> 16.2.9 -> 16.2.10 -> 16.2.11 Now i wanted to upgrade to 17.2.0 but after starting the upgrade with ``` # ceph orch upgrade start --ceph-version 17.2.0 ``` The orch manager module seems to be gone now and the upgrade don't seem to run. ``` # ceph orch upgrade status Error ENOENT: No orchestrator configured (try `ceph orch set backend`) # ceph orch set backend cephadm Error ENOENT: Module not found ``` During the failed upgrade all nodes had the 16.2.11 cephadm installed. Fortunately the cluster is still running... somehow. I installed the latest 17.2.X cephadm on all nodes and rebooted them nodes, but this didn't help. Does someone have a hint? Yours, bbk
That's very odd, I haven't seen this before. What container image is the upgraded mgr running on (to know for sure, can check the podman/docker run command at the end of the /var/lib/ceph/<fsid>/mgr.<mgr-id>/unit.run file on the mgr's host)? Also, could maybe try "ceph mgr module enable cephadm" to see if it does anything? On Tue, Mar 14, 2023 at 9:23 AM bbk <bbk@nocloud.ch> wrote:
Dear List,
Today i was sucessfully upgrading with cephadm from 16.2.8 -> 16.2.9 -> 16.2.10 -> 16.2.11
Now i wanted to upgrade to 17.2.0 but after starting the upgrade with
``` # ceph orch upgrade start --ceph-version 17.2.0 ```
The orch manager module seems to be gone now and the upgrade don't seem to run.
``` # ceph orch upgrade status Error ENOENT: No orchestrator configured (try `ceph orch set backend`)
# ceph orch set backend cephadm Error ENOENT: Module not found ```
During the failed upgrade all nodes had the 16.2.11 cephadm installed.
Fortunately the cluster is still running... somehow. I installed the latest 17.2.X cephadm on all nodes and rebooted them nodes, but this didn't help.
Does someone have a hint?
Yours, bbk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The unit file tells me: ``` # cat /var/lib/ceph/6d0ecf22-9155-4684-971a-2f6cde8628c8/mgr.pamir.ajvbug/unit.run set -e /usr/bin/install -d -m0770 -o 167 -g 167 /var/run/ceph/6d0ecf22-9155-4684-971a-2f6cde8628c8 # mgr.pamir.ajvbug ! /usr/bin/podman rm -f ceph-6d0ecf22-9155-4684-971a-2f6cde8628c8-mgr.pamir.ajvbug 2> /dev/null ! /usr/bin/podman rm -f ceph-6d0ecf22-9155-4684-971a-2f6cde8628c8-mgr-pamir-ajvbug 2> /dev/null ! /usr/bin/podman rm -f --storage ceph-6d0ecf22-9155-4684-971a-2f6cde8628c8-mgr-pamir-ajvbug 2> /dev/null ! /usr/bin/podman rm -f --storage ceph-6d0ecf22-9155-4684-971a-2f6cde8628c8-mgr.pamir.ajvbug 2> /dev/null /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-mgr --init --name ceph-6d0ecf22-9155-4684-971a-2f6cde8628c8-mgr-pamir-ajvbug -d --log-driver journald --conmon-pidfile /run/ceph-6d0ecf22-9155-4684-971a-2f6cde8628c8@mgr.pamir.ajvbug.service-pid --cidfile /run/ceph-6d0ecf22-9155-4684-971a-2f6cde8628c8@mgr.pamir.ajvbug.service-cid --cgroups=split -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a -e NODE_NAME=pamir -e CEPH_USE_RANDOM_NONCE=1 -e TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728 -v /var/run/ceph/6d0ecf22-9155-4684-971a-2f6cde8628c8:/var/run/ceph:z -v /var/log/ceph/6d0ecf22-9155-4684-971a-2f6cde8628c8:/var/log/ceph:z -v /var/lib/ceph/6d0ecf22-9155-4684-971a-2f6cde8628c8/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/6d0ecf22-9155-4684-971a-2f6cde8628c8/mgr.pamir.ajvbug:/var/lib/ceph/mgr/ceph-pamir.ajvbug:z -v /var/lib/ceph/6d0ecf22-9155-4684-971a-2f6cde8628c8/mgr.pamir.ajvbug/config:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a -n mgr.pamir.ajvbug -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' ``` The module cephadm seems to be still enabled already: ``` # ceph mgr module enable cephadm module 'cephadm' is already enabled ``` The current active mgr seems to have the 17.2.0 image all other daemons.. are still on 16.2.11 ``` # cephadm ls |grep version "version": "16.2.11", "version": "16.2.11", "version": "16.2.11", "version": "16.2.11", "version": "16.2.11", "version": "16.2.11", "version": "16.2.11", "version": "16.2.11", "version": "16.2.11", "version": "17.2.0", "version": "", "version": "16.2.11", "version": "16.2.11", "version": "16.2.11", ```
Sorry, my replies from the webinterface didn't yet went trough. Thank you both! With your help i was able to get it up and running on 17.2.5. Yours, bbk On Tue, 2023-03-14 at 09:44 -0400, Adam King wrote:
That's very odd, I haven't seen this before. What container image is the upgraded mgr running on (to know for sure, can check the podman/docker run command at the end of the /var/lib/ceph/<fsid>/mgr.<mgr-id>/unit.run file on the mgr's host)? Also, could maybe try "ceph mgr module enable cephadm" to see if it does anything?
On Tue, Mar 14, 2023 at 9:23 AM bbk <bbk@nocloud.ch> wrote:
Dear List,
Today i was sucessfully upgrading with cephadm from 16.2.8 -> 16.2.9 -> 16.2.10 -> 16.2.11
Now i wanted to upgrade to 17.2.0 but after starting the upgrade with
``` # ceph orch upgrade start --ceph-version 17.2.0 ```
The orch manager module seems to be gone now and the upgrade don't seem to run.
``` # ceph orch upgrade status Error ENOENT: No orchestrator configured (try `ceph orch set backend`)
# ceph orch set backend cephadm Error ENOENT: Module not found ```
During the failed upgrade all nodes had the 16.2.11 cephadm installed.
Fortunately the cluster is still running... somehow. I installed the latest 17.2.X cephadm on all nodes and rebooted them nodes, but this didn't help.
Does someone have a hint?
Yours, bbk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 14.03.23 14:21, bbk wrote: `
# ceph orch upgrade start --ceph-version 17.2.0
I would never recommend to update to a .0 release. Why not go directly to the latest 17.2.5? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
ah.. ok, it was not clear to me that skipping minor version when doing a major upgrade was supported.
On 14.03.23 15:22, bbk@nocloud.ch wrote:
ah.. ok, it was not clear to me that skipping minor version when doing a major upgrade was supported.
You can even skip one major version when doing an upgrade. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Thank you both for your help! I got 17.2.5 running now. I still had one mgr on 16.2.11 with that the orch module was runnung and i was able to set the `cephadm` backend on the webinterface. Then i directly upgraded to 17.2.5, during the upgrade it seemed to have paused again but after disabling the telemetry mgr module it resumed and went trough. Yours, bbk
participants (4)
-
Adam King
-
bbk
-
bbk＠nocloud.ch
-
Robert Sander