Hi Stephan, hi Adam, thank you for reading my questions and for your suggestions. I did what you suggested, but unfortunately in the meantime the cluster in question degraded to the point of un-manageability, because for reasons incomprehensible for me, cephadm began trying to redeploy and reconfigure daemons (monitors, managers, osds) which failed ... due to, well, failing image registry access. One monitor completely died, could not be redeployed, or removed and re-added, and i only could revive it in a manual old school pre cephadmin non container way. This 95 osds cluster is used for development and for the evaluation of specific use case performance and does not contain valuable data - i deem it more costly to try to repair it, than to start a new one from scratch. However this situation once more fuels my assumption, that there is way too much abstraction and automatism in the cephadm containerization way of modern CEPH. Once things go wrong, it's gotten harder to understand, what is happening ... and in this case to prevent it from even getting worse. Again many thanks for your support. I'm happy, that this community exists. Cheers, toBias From: "Stephan Hohn" <stephan@gridscale.io> To: "Tobias Tempel" <tobias.tempel@desy.de> Cc: "Adam King" <adking@redhat.com>, "ceph-users" <ceph-users@ceph.io> Sent: Thursday, 9 January, 2025 12:31:43 Subject: [ceph-users] Re: ceph orch upgrade tries to pull latest? Hi Tobias, have you tried to set your privat registry before starting the upgrade command. ~# ceph cephadm registry-login <registry url> <username> <password> e.g. ~# ceph cephadm registry-login harborregistry <username optional> <password optional> ~# ceph orch upgrade start --image harborregistry/quay.io/ceph/ceph:v18.2.4 This might also help to debug ~# ceph -W cephadm --watch-debug Cheers Stephan Am Do., 9. Jan. 2025 um 12:04 Uhr schrieb tobias tempel < tobias.tempel@desy.de>:
Dear Adam, thank you very much for your reply. In /var/log/ceph/cephadm.log i saw lots of entries like this
2025-01-08 10:00:22,045 7ff021d8c000 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'harborregistry/quay.io/ceph/ceph', '--timeout', '895', 'pull'] 2025-01-08 10:00:22,172 7ff021d8c000 INFO Pulling container image harborregistry/quay.io/ceph/ceph... 2025-01-08 10:00:27,176 7ff021d8c000 INFO Non-zero exit code 125 from /usr/bin/podman pull harborregistry/quay.io/ceph/ceph 2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr Trying to pull harborregistry/quay.io/ceph/ceph:latest... 2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr time="2025-01-08T10:00:22+01:00" level=warning msg="failed, retrying in 1s ... (1/3). Error: initializing source docker://harborregistry/ quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/ quay.io/ ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found" ... 2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr Error: initializing source docker://harborregistry/quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/quay.io/ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found 2025-01-08 10:00:27,177 7ff021d8c000 ERROR ERROR: Failed command: /usr/bin/podman pull harborregistry/quay.io/ceph/ceph 2025-01-08 10:01:27,459 7f5f185d0000 DEBUG --------------------------------------------------------------------------------
In the meantime i was given a hint, to ceph config set mgr container_image harborregistry/ quay.io/ceph/ceph:v18.2.4 which indeed changed things to
2025-01-08 17:12:45,952 7ffb1da9b000 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4', '--timeout', '895', 'inspect-image'] 2025-01-08 17:12:46,219 7ffb1da9b000 DEBUG /usr/bin/podman: stdout 2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,[ quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec 7e1388855ee3d00813b52c3172b0b23b37906 quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77 harborregistry/quay.io/ceph/ceph@sha256:6ac7f923aa1d23 b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 harborregistry/ quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77 ] 2025-01-08 17:12:46,649 7ffb1da9b000 DEBUG ceph: stdout ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 2025-01-08 17:12:50,852 7f21649cf000 DEBUG --------------------------------------------------------------------------------
Only to now encounter log entries
2025-01-09 00:01:20,077 7fe3a719e000 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'docker.io/ceph/daemon-base:latest-master-devel', '--timeout', '895', '_orch', 'deploy', '--fsid', 'xxxxx'] 2025-01-09 00:01:20,210 7fe3a719e000 DEBUG Loaded deploy configuration: {'fsid': 'xxxxx', 'name': 'mon.monitor0x', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_name': 'mon', 'ports': [], 'ip': None, 'deployed_by': [' quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906', ' quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for xxxxx\n[global]\n\tfsid = xxxxx\n\tmon_host = [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0] [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0] [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0]\n[mon.monitor0x]\npublic network = x.x.x.0/22\n', 'keyring': '[mon.]\n\tkey = xxxxx\n\tcaps mon = "allow *"\n', 'files': {'config': '[mon.monitor0x]\npublic network = x.x.x.0/22\n'}}} 2025-01-09 00:01:20,210 7fe3a719e000 DEBUG Determined image: ' docker.io/ceph/daemon-base:latest-master-devel' 2025-01-09 00:01:20,218 7fe3a719e000 INFO Redeploy daemon mon.monitor0x ... 2025-01-09 00:02:20,255 7fe3a719e000 INFO Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE= docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor0x -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-master-devel -c %u %g /var/lib/ceph 2025-01-09 00:02:20,255 7fe3a719e000 INFO stat: stderr Trying to pull docker.io/ceph/daemon-base:latest-master-devel... 2025-01-09 00:02:20,255 7fe3a719e000 INFO stat: stderr Error: initializing source docker://ceph/daemon-base:latest-master-devel: pinging container registry registry-1.docker.io: Get " https://registry-1.docker.io/v2/": dial tcp 54.236.113.205:443: i/o timeout 2025-01-09 00:02:20,256 7fe3a719e000 ERROR ERROR: Failed to extract uid/gid for path /var/lib/ceph: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor0x -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-master-devel -c %u %g /var/lib/ceph: Trying to pull docker.io/ceph/daemon-base:latest-master-devel... Error: initializing source docker://ceph/daemon-base:latest-master-devel: pinging container registry registry-1.docker.io: Get " https://registry-1.docker.io/v2/": dial tcp 54.236.113.205:443: i/o timeout
After that I was also directed to a config setting "mgr mgr/cephadm/default_registry" which up to now is absent from our configuration (nothing shows up in "ceph config dump | grep registry") but i yet have no idea what to set here ...
Again thank you very much, cheers, toBias
------------------------------ *From: *"Adam King" <adking@redhat.com> *To: *"Tobias Tempel" <tobias.tempel@desy.de> *Cc: *"ceph-users" <ceph-users@ceph.io> *Sent: *Wednesday, 8 January, 2025 20:15:51 *Subject: *[ceph-users] Re: ceph orch upgrade tries to pull latest?
It looks like the "resource not found" message is being directly output by podman. Is there anything in the cephadm.log (/var/log/ceph/cephadm.log) on one of the hosts where this is happening that says what podman command cephadm was running that hit this error?
On Wed, Jan 8, 2025 at 5:27 AM tobias tempel <tobias.tempel@desy.de> wrote:
Dear all, i'm trying to cephadm-upgrade in an airgapped environment from 18.2.2 to 18.2.4 ... yet to no avail. local image registry is a harbor instance, I start the upgrade process with
ceph orch upgrade start --image harborregistry/ quay.io/ceph/ceph:v18.2.4
and status looks good
ceph orch upgrade status { "target_image": "harborregistry/quay.io/ceph/ceph:v18.2.4", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false }
In the cephadm log i can see messages like
cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4', '--timeout', '895', 'inspect-image']
which is fine (works on the commandline), but also
2025-01-08 10:33:53,911 7f9c66d50000 INFO /usr/bin/podman: stderr Error: initializing source docker://harborregistry/quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/quay.io/ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found
so for some reason cephadm keeps trying to pull tag "latest" - which i did not specify - and this fails ... again and again and again. what am i missing? can anyone give me a hint, where to look at?
Thank you very much, cheers, toBias
PS: ceph config get mgr WHO MASK LEVEL OPTION VALUE RO mgr basic container_image harborregistry/quay.io/ceph/ceph * mgr advanced mgr/cephadm/container_image_alertmanager harborregistry/quay.io/prometheus/alertmanager * mgr advanced mgr/cephadm/container_image_base harborregistry/quay.io/ceph/ceph mgr advanced mgr/cephadm/container_image_grafana harborregistry/quay.io/ceph/ceph-grafana * mgr advanced mgr/cephadm/container_image_node_exporter harborregistry/quay.io/prometheus/node-exporter * mgr advanced mgr/cephadm/container_image_prometheus harborregistry/quay.io/prometheus/prometheus *
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io