Pull failed on cluster upgrade
Dear Ceph users, during an upgrade from 18.2.2 to 18.2.4 the image pull from Dockerhub failed on one machine running a monitor daemon, while it succeeded on the previous ones. # ceph orch upgrade status { "target_image": "snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "mgr" ], "progress": "3/152 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true } # ceph health detail [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image failed to pull snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 on host aka On the affected host I can see the image has been correctly pulled, I can even delete and re-pull it manually: $ sudo docker pull snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 docker.io/snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1: Pulling from snack14/ceph-wizard c4df4d1fcd03: Pull complete 676ca14fffd6: Pull complete cde0e2cfc7c9: Pull complete Digest: sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 Status: Downloaded newer image for snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 docker.io/snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 so I believe I didn't hit Dockerhub pull limits. Yet the upgrade fails again in the same way when I resume it. I noticed that the docker logs show these warnings in coincidence with a failed pull triggered by the upgrade: Aug 05 18:24:28 aka dockerd[1564]: time="2024-08-05T18:24:28.087629058+02:00" level=warning msg="canonical references cannot be resolved: snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1" Aug 05 18:24:29 aka dockerd[1564]: time="2024-08-05T18:24:29.332143728+02:00" level=warning msg="reference for unknown type: " digest="sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1" remote="docker.io/snack14/ceph-wizard@sha256:b1994328eb078778> but only the second one appears when doing a successful manual pull, so I suspect only the first one is representative of the problem. The docker version on the affected machine is 26.1.3. Thanks in advance for any help, Nicola
On 05.08.24 18:38, Nicola Mori wrote:
docker.io/snack14/ceph-wizard
This is not an official container image. The images from the Ceph project are on quay.io/ceph/ceph. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
I think I found the problem. Setting the cephadm log level to debug and then watching the logs during the upgrade: ceph config set mgr mgr/cephadm/log_to_cluster_level debug ceph -W cephadm --watch-debug I found this line just before the error: ceph: stderr Fatal glibc error: CPU does not support x86-64-v2 The same error comes out if I try to launch the container manually on the culprit machine, so I'd say it's a dead end. What I don't understand is how such a big change like dismissing support for an old architecture has been introduced in a point release (18.2.2 works fine), but maybe it's just me that's missing some basic info about the versioning scheme of Ceph. Anyway, now I'm stuck with 3 daemons running 18.2.4 and the others still on 18.2.2. The cluster looks happy and I see no malfunctioning, can I leave it in this state with no risk? If not, is it safe to rollback the 3 upgraded daemons to the previous version? Thanks again, Nicola
What operating system/distribution are you running? What hardware? David On Tue, Aug 6, 2024, at 02:20, Nicola Mori wrote:
I think I found the problem. Setting the cephadm log level to debug and then watching the logs during the upgrade:
ceph config set mgr mgr/cephadm/log_to_cluster_level debug ceph -W cephadm --watch-debug
I found this line just before the error:
ceph: stderr Fatal glibc error: CPU does not support x86-64-v2
The same error comes out if I try to launch the container manually on the culprit machine, so I'd say it's a dead end. What I don't understand is how such a big change like dismissing support for an old architecture has been introduced in a point release (18.2.2 works fine), but maybe it's just me that's missing some basic info about the versioning scheme of Ceph.
Anyway, now I'm stuck with 3 daemons running 18.2.4 and the others still on 18.2.2. The cluster looks happy and I see no malfunctioning, can I leave it in this state with no risk? If not, is it safe to rollback the 3 upgraded daemons to the previous version? Thanks again,
Nicola
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Attachments: * smime.p7s
If you're using VMs, https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/6X6QIEMWDYS... might be relevant On Tue, Aug 6, 2024 at 3:21 AM Nicola Mori <mori@fi.infn.it> wrote:
I think I found the problem. Setting the cephadm log level to debug and then watching the logs during the upgrade:
ceph config set mgr mgr/cephadm/log_to_cluster_level debug ceph -W cephadm --watch-debug
I found this line just before the error:
ceph: stderr Fatal glibc error: CPU does not support x86-64-v2
The same error comes out if I try to launch the container manually on the culprit machine, so I'd say it's a dead end. What I don't understand is how such a big change like dismissing support for an old architecture has been introduced in a point release (18.2.2 works fine), but maybe it's just me that's missing some basic info about the versioning scheme of Ceph.
Anyway, now I'm stuck with 3 daemons running 18.2.4 and the others still on 18.2.2. The cluster looks happy and I see no malfunctioning, can I leave it in this state with no risk? If not, is it safe to rollback the 3 upgraded daemons to the previous version? Thanks again,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Unfortunately I'm on bare metal, with very old hardware so I cannot do much. I'd try to build a Ceph image based on Rocky Linux 8 if I could get the Dockerfile of the current image to start with, but I've not been able to find it. Can you please help me with this? Cheers, Nicola
Hi,
On 7 Aug 2024, at 10:31, Nicola Mori <mori@fi.infn.it> wrote:
Unfortunately I'm on bare metal, with very old hardware so I cannot do much. I'd try to build a Ceph image based on Rocky Linux 8 if I could get the Dockerfile of the current image to start with, but I've not been able to find it. Can you please help me with this?
You can try luck with packages, if I understood the problem correctly [1] Apparently this is a problem of our time. Now the developer writes software for the container, and the fact that under the container there is hardware that the container is not happy with is "well, buy new hardware" k [1] https://github.com/ceph/ceph-build/pull/2272
On 8/7/24 09:40, Konstantin Shalygin wrote:
Hi,
On 7 Aug 2024, at 10:31, Nicola Mori <mori@fi.infn.it> wrote:
Unfortunately I'm on bare metal, with very old hardware so I cannot do much. I'd try to build a Ceph image based on Rocky Linux 8 if I could get the Dockerfile of the current image to start with, but I've not been able to find it. Can you please help me with this?
You can try luck with packages, if I understood the problem correctly [1] Apparently this is a problem of our time. Now the developer writes software for the container, and the fact that under the container there is hardware that the container is not happy with is "well, buy new hardware"
It would be very helpful for Ceph admins if the upgrade routines first check if an upgrade is supported by the underlying hardware: $ ceph orch upgrade start --ceph-version <version> should fail in case of unsupported hw. Just an idea.... Dietmar
Thank you Konstantin, as it was foreseeable this problem didn't hit just me. So I hope the build of images based on CentOS Stream 8 will be resumed. Otherwise I'll try to build myself. Nicola
In the end I built up an image based on Ubuntu 22.04 which does not mandate x86-64-v2. I installed the official Ceph packages and hacked here and there (e.g. it was necessary to set the uid and gid of the Ceph user and group identical to those used by the CentOS Stream 8 image to avoid to mess with filesystem permissions), now I'm upgrading the cluster and hopefully it will end up well. I hope that this way, if Red Hat will continue to provide Ceph packages for Ubuntu, I will be able to upgrade to Ceph 19 and beyond since also Ubuntu 24.04 does not need x86-64-v2. I'd have to figure out how to upgrade cephadm on the hosts (which run Rocky Linux 8) now that the official build is discontinued, but I'll cope with that later. Thanks to everybody for the help; if anyone is interested I can share the Dockerfile. Cheers, Nicola
The upgrade ended successfully, but now the cluster reports this error: MDS_CLIENTS_BROKEN_ROOTSQUASH: 1 MDS report clients with broken root_squash implementation From what I understood this is due to a new feature meant to fix a bug in the root_squash implementation, and that will be released with version 19. I didn't find anything about a backport to 18.2.4. Can someone share some info please? Especially about if and how it can be fixed. Thanks in advance, Nicola
Hi Nicola, You might want to post in the ceph-dev list about this or discuss it with devs in the ceph-devel slack channel for quicker help. Bests, Frédéric. ________________________________ De : Nicola Mori <mori@fi.infn.it> Envoyé : mercredi 21 août 2024 15:52 À : ceph-users@ceph.io Objet : [ceph-users] Re: Pull failed on cluster upgrade The upgrade ended successfully, but now the cluster reports this error: MDS_CLIENTS_BROKEN_ROOTSQUASH: 1 MDS report clients with broken root_squash implementation From what I understood this is due to a new feature meant to fix a bug in the root_squash implementation, and that will be released with version 19. I didn't find anything about a backport to 18.2.4. Can someone share some info please? Especially about if and how it can be fixed. Thanks in advance, Nicola
Hi, I haven't dealt with this myself yet, but the docs [0] state:
A bug was discovered in root_squash which would potentially lose changes made by a client restricted with root_squash caps. The fix required a change to the protocol and a client upgrade is required.
This is a HEALTH_ERR warning because of the danger of inconsistency and lost data. It is recommended to either upgrade your clients, discontinue using root_squash in the interim, or silence the warning if desired.
To evict and permanently block broken clients from connecting to the cluster, set the required_client_feature bit client_mds_auth_caps.
Do you have the option to upgrade your client(s) as well? [0] https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-clients-broken-r... Zitat von Nicola Mori <mori@fi.infn.it>:
The upgrade ended successfully, but now the cluster reports this error:
MDS_CLIENTS_BROKEN_ROOTSQUASH: 1 MDS report clients with broken root_squash implementation
From what I understood this is due to a new feature meant to fix a bug in the root_squash implementation, and that will be released with version 19. I didn't find anything about a backport to 18.2.4. Can someone share some info please? Especially about if and how it can be fixed. Thanks in advance,
Nicola
participants (8)
-
Adam King
-
David Orman
-
Dietmar Rieder
-
Eugen Block
-
Frédéric Nass
-
Konstantin Shalygin
-
Nicola Mori
-
Robert Sander