Hi Tobias, have you tried to set your privat registry before starting the upgrade command. ~# ceph cephadm registry-login <registry url> <username> <password> e.g. ~# ceph cephadm registry-login harborregistry <username optional> <password optional> ~# ceph orch upgrade start --image harborregistry/quay.io/ceph/ceph:v18.2.4 This might also help to debug ~# ceph -W cephadm --watch-debug Cheers Stephan Am Do., 9. Jan. 2025 um 12:04 Uhr schrieb tobias tempel < tobias.tempel@desy.de>:
Dear Adam, thank you very much for your reply. In /var/log/ceph/cephadm.log i saw lots of entries like this
2025-01-08 10:00:22,045 7ff021d8c000 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'harborregistry/quay.io/ceph/ceph', '--timeout', '895', 'pull'] 2025-01-08 10:00:22,172 7ff021d8c000 INFO Pulling container image harborregistry/quay.io/ceph/ceph... 2025-01-08 10:00:27,176 7ff021d8c000 INFO Non-zero exit code 125 from /usr/bin/podman pull harborregistry/quay.io/ceph/ceph 2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr Trying to pull harborregistry/quay.io/ceph/ceph:latest... 2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr time="2025-01-08T10:00:22+01:00" level=warning msg="failed, retrying in 1s ... (1/3). Error: initializing source docker://harborregistry/ quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/ quay.io/ ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found" ... 2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr Error: initializing source docker://harborregistry/quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/quay.io/ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found 2025-01-08 10:00:27,177 7ff021d8c000 ERROR ERROR: Failed command: /usr/bin/podman pull harborregistry/quay.io/ceph/ceph 2025-01-08 10:01:27,459 7f5f185d0000 DEBUG --------------------------------------------------------------------------------
In the meantime i was given a hint, to ceph config set mgr container_image harborregistry/ quay.io/ceph/ceph:v18.2.4 which indeed changed things to
2025-01-08 17:12:45,952 7ffb1da9b000 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4', '--timeout', '895', 'inspect-image'] 2025-01-08 17:12:46,219 7ffb1da9b000 DEBUG /usr/bin/podman: stdout 2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,[ quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec 7e1388855ee3d00813b52c3172b0b23b37906 quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77 harborregistry/quay.io/ceph/ceph@sha256:6ac7f923aa1d23 b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 harborregistry/ quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77 ] 2025-01-08 17:12:46,649 7ffb1da9b000 DEBUG ceph: stdout ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 2025-01-08 17:12:50,852 7f21649cf000 DEBUG --------------------------------------------------------------------------------
Only to now encounter log entries
2025-01-09 00:01:20,077 7fe3a719e000 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'docker.io/ceph/daemon-base:latest-master-devel', '--timeout', '895', '_orch', 'deploy', '--fsid', 'xxxxx'] 2025-01-09 00:01:20,210 7fe3a719e000 DEBUG Loaded deploy configuration: {'fsid': 'xxxxx', 'name': 'mon.monitor0x', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_name': 'mon', 'ports': [], 'ip': None, 'deployed_by': [' quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906', ' quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for xxxxx\n[global]\n\tfsid = xxxxx\n\tmon_host = [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0] [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0] [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0]\n[mon.monitor0x]\npublic network = x.x.x.0/22\n', 'keyring': '[mon.]\n\tkey = xxxxx\n\tcaps mon = "allow *"\n', 'files': {'config': '[mon.monitor0x]\npublic network = x.x.x.0/22\n'}}} 2025-01-09 00:01:20,210 7fe3a719e000 DEBUG Determined image: ' docker.io/ceph/daemon-base:latest-master-devel' 2025-01-09 00:01:20,218 7fe3a719e000 INFO Redeploy daemon mon.monitor0x ... 2025-01-09 00:02:20,255 7fe3a719e000 INFO Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE= docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor0x -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-master-devel -c %u %g /var/lib/ceph 2025-01-09 00:02:20,255 7fe3a719e000 INFO stat: stderr Trying to pull docker.io/ceph/daemon-base:latest-master-devel... 2025-01-09 00:02:20,255 7fe3a719e000 INFO stat: stderr Error: initializing source docker://ceph/daemon-base:latest-master-devel: pinging container registry registry-1.docker.io: Get " https://registry-1.docker.io/v2/": dial tcp 54.236.113.205:443: i/o timeout 2025-01-09 00:02:20,256 7fe3a719e000 ERROR ERROR: Failed to extract uid/gid for path /var/lib/ceph: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor0x -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-master-devel -c %u %g /var/lib/ceph: Trying to pull docker.io/ceph/daemon-base:latest-master-devel... Error: initializing source docker://ceph/daemon-base:latest-master-devel: pinging container registry registry-1.docker.io: Get " https://registry-1.docker.io/v2/": dial tcp 54.236.113.205:443: i/o timeout
After that I was also directed to a config setting "mgr mgr/cephadm/default_registry" which up to now is absent from our configuration (nothing shows up in "ceph config dump | grep registry") but i yet have no idea what to set here ...
Again thank you very much, cheers, toBias
------------------------------ *From: *"Adam King" <adking@redhat.com> *To: *"Tobias Tempel" <tobias.tempel@desy.de> *Cc: *"ceph-users" <ceph-users@ceph.io> *Sent: *Wednesday, 8 January, 2025 20:15:51 *Subject: *[ceph-users] Re: ceph orch upgrade tries to pull latest?
It looks like the "resource not found" message is being directly output by podman. Is there anything in the cephadm.log (/var/log/ceph/cephadm.log) on one of the hosts where this is happening that says what podman command cephadm was running that hit this error?
On Wed, Jan 8, 2025 at 5:27 AM tobias tempel <tobias.tempel@desy.de> wrote:
Dear all, i'm trying to cephadm-upgrade in an airgapped environment from 18.2.2 to 18.2.4 ... yet to no avail. local image registry is a harbor instance, I start the upgrade process with
ceph orch upgrade start --image harborregistry/ quay.io/ceph/ceph:v18.2.4
and status looks good
ceph orch upgrade status { "target_image": "harborregistry/quay.io/ceph/ceph:v18.2.4", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false }
In the cephadm log i can see messages like
cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4', '--timeout', '895', 'inspect-image']
which is fine (works on the commandline), but also
2025-01-08 10:33:53,911 7f9c66d50000 INFO /usr/bin/podman: stderr Error: initializing source docker://harborregistry/quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/quay.io/ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found
so for some reason cephadm keeps trying to pull tag "latest" - which i did not specify - and this fails ... again and again and again. what am i missing? can anyone give me a hint, where to look at?
Thank you very much, cheers, toBias
PS: ceph config get mgr WHO MASK LEVEL OPTION VALUE RO mgr basic container_image harborregistry/quay.io/ceph/ceph * mgr advanced mgr/cephadm/container_image_alertmanager harborregistry/quay.io/prometheus/alertmanager * mgr advanced mgr/cephadm/container_image_base harborregistry/quay.io/ceph/ceph mgr advanced mgr/cephadm/container_image_grafana harborregistry/quay.io/ceph/ceph-grafana * mgr advanced mgr/cephadm/container_image_node_exporter harborregistry/quay.io/prometheus/node-exporter * mgr advanced mgr/cephadm/container_image_prometheus harborregistry/quay.io/prometheus/prometheus *
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io