I managed to determine the cause after adding more debugging: https://github.com/containers/podman/issues/8444 https://github.com/pmoogi-redhat/podman/commit/5cbfbd724cc49194d20f7e9b3de3e... https://github.com/containers/podman/pull/8642 This would appear to impact podman 2.2.0. On Sun, Dec 20, 2020 at 10:23 AM David Orman <ormandj@corenode.com> wrote:

Hi,

We have an Octopus installation using cephadm/containers in podman. We're trying to upgrade from 15.2.7 to 15.2.8. The mgrs upgrade successfully, but the we then get a failure that cephadm failed to pull an image:

[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image

Failed to pull docker.io/ceph/ceph:v15.2.8 on ceph02:

Looking at this server, and trying to run the cephadm pull manually:

root@ceph02:~# cephadm pull

Using recent ceph image docker.io/ceph/ceph:v15.2.8

Pulling container image docker.io/ceph/ceph:v15.2.8...

Traceback (most recent call last):

File "/usr/sbin/cephadm", line 6111, in <module>

r = args.func()

File "/usr/sbin/cephadm", line 1381, in _infer_image

return func()

File "/usr/sbin/cephadm", line 2676, in command_pull

return command_inspect_image()

File "/usr/sbin/cephadm", line 1381, in _infer_image

return func()

File "/usr/sbin/cephadm", line 2716, in command_inspect_image

info_from = get_image_info_from_inspect(out.strip(), args.image)

File "/usr/sbin/cephadm", line 2727, in get_image_info_from_inspect

image_id, digests = out.split(',', 1)

ValueError: not enough values to unpack (expected 2, got 1)

root@ceph02:~# cephadm version

Using recent ceph image docker.io/ceph/ceph:v15.2.8

ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)

root@ceph02:~# podman images

REPOSITORY TAG IMAGE ID CREATED SIZE

docker.io/ceph/ceph v15.2.8 5553b0cb212c 3 days ago 965 MB

docker.io/ceph/ceph v15.2.7 2bc420ddb175 2 weeks ago 979 MB

docker.io/ceph/ceph v15 2bc420ddb175 2 weeks ago 979 MB

docker.io/prom/node-exporter v0.18.1 e5a616e4b9cf 18 months ago 24.3 MB

root@ceph02:~#

We've upgraded this cluster multiple times prior without issue, so this appear to be unique to the 15.2.7 -> 15.2.8 upgrade. If we pass --image with docker.io/ceph/ceph:v15.2.8, the same behavior occurs. It seems like cephadm in the 15.2.8 image isn't playing nicely with the pull command, for some reason.

Interestingly, I did a fresh install of v15.2.8 on one server, and after the installation, trying a 'cephadm pull' also fails on that 15.2.8 setup, so it seems like the pull functionality may be broken on these images.