[ceph-users] Re: Issues upgrading from ceph 15.2.7 to 15.2.8 related to cephadm pull

Hi, We have an Octopus installation using cephadm/containers in podman. We're trying to upgrade from 15.2.7 to 15.2.8. The mgrs upgrade successfully, but the we then get a failure that cephadm failed to pull an image: [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image Failed to pull docker.io/ceph/ceph:v15.2.8 on ceph02: Looking at this server, and trying to run the cephadm pull manually: root@ceph02:~# cephadm pull Using recent ceph image docker.io/ceph/ceph:v15.2.8 Pulling container image docker.io/ceph/ceph:v15.2.8... Traceback (most recent call last): File "/usr/sbin/cephadm", line 6111, in <module> r = args.func() File "/usr/sbin/cephadm", line 1381, in _infer_image return func() File "/usr/sbin/cephadm", line 2676, in command_pull return command_inspect_image() File "/usr/sbin/cephadm", line 1381, in _infer_image return func() File "/usr/sbin/cephadm", line 2716, in command_inspect_image info_from = get_image_info_from_inspect(out.strip(), args.image) File "/usr/sbin/cephadm", line 2727, in get_image_info_from_inspect image_id, digests = out.split(',', 1) ValueError: not enough values to unpack (expected 2, got 1) root@ceph02:~# cephadm version Using recent ceph image docker.io/ceph/ceph:v15.2.8 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable) root@ceph02:~# podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/ceph/ceph v15.2.8 5553b0cb212c 3 days ago 965 MB docker.io/ceph/ceph v15.2.7 2bc420ddb175 2 weeks ago 979 MB docker.io/ceph/ceph v15 2bc420ddb175 2 weeks ago 979 MB docker.io/prom/node-exporter v0.18.1 e5a616e4b9cf 18 months ago 24.3 MB root@ceph02:~# We've upgraded this cluster multiple times prior without issue, so this appear to be unique to the 15.2.7 -> 15.2.8 upgrade. If we pass --image with docker.io/ceph/ceph:v15.2.8, the same behavior occurs. It seems like cephadm in the 15.2.8 image isn't playing nicely with the pull command, for some reason. Interestingly, I did a fresh install of v15.2.8 on one server, and after the installation, trying a 'cephadm pull' also fails on that 15.2.8 setup, so it seems like the pull functionality may be broken on these images.