We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine. I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials. When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image". But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [ "harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551" ] } I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password So what's wrong? Currently using 18.2.4.
You could try setting `ceph config set mgr mgr/cephadm/use_repo_digest false` and give it another go. I've seen some issues in the past with using the image digest with local repos. On Thu, Apr 24, 2025 at 10:15 AM Sake Ceph <ceph@paulusma.eu> wrote:
We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine.
I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials.
When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image".
But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [ "harbor.url/project/ceph/ceph@sha256 :848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551" ] }
I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password
So what's wrong? Currently using 18.2.4. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Did not work, still the same error.
Op 24-04-2025 17:01 CEST schreef Adam King <adking@redhat.com>:
You could try setting `ceph config set mgr mgr/cephadm/use_repo_digest false` and give it another go. I've seen some issues in the past with using the image digest with local repos.
On Thu, Apr 24, 2025 at 10:15 AM Sake Ceph <ceph@paulusma.eu> wrote:
We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine.
I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials.
When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image".
But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [ "harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551" ] }
I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password
So what's wrong? Currently using 18.2.4. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Did you run 'ceph cephadm registry-login' or only 'sudo cephadm -v registry-login --registry-json login'? Because the first would login only the local node into the registry, the latter does it for all nodes in the host list. Zitat von Sake Ceph <ceph@paulusma.eu>:
We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine.
I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials.
When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image".
But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [
"harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551" ] }
I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password
So what's wrong? Currently using 18.2.4. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That isn't a option: $ cephadm -v registry-login --registry-json login ERROR: cephadm should be run as root So I enabled temporarily anonymous access for the project at the moment. But I noticed something in the logs of Harbor. All the ceph images got pulled via anonymous user, but one pull for promtail (deployed on all nodes) was done with the provided credentials, the rest via anonymous. The same for loki which is installed on two admin nodes, one of which I tried adding the new credentials. But all ceph pulls where done via anonymous access. The strange part is that the first is done as ceph:v18.2.6-20250417 and the rest via ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551
Op 24-04-2025 18:57 CEST schreef Eugen Block <eblock@nde.ag>:
Did you run 'ceph cephadm registry-login' or only 'sudo cephadm -v registry-login --registry-json login'? Because the first would login only the local node into the registry, the latter does it for all nodes in the host list.
Zitat von Sake Ceph <ceph@paulusma.eu>:
We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine.
I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials.
When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image".
But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [
"harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551" ] }
I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password
So what's wrong? Currently using 18.2.4. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What I wanted to know is, did you execute the login remotely for all hosts by running sudo ceph cephadm -v registry-login --registry-json login or only sudo cephadm -v registry-login --registry-json login to login only the host where you issued that command? But since you mention that all hosts have already pulled the image it's not that relevant anymore. Zitat von Sake Ceph <ceph@paulusma.eu>:
That isn't a option: $ cephadm -v registry-login --registry-json login ERROR: cephadm should be run as root
So I enabled temporarily anonymous access for the project at the moment. But I noticed something in the logs of Harbor. All the ceph images got pulled via anonymous user, but one pull for promtail (deployed on all nodes) was done with the provided credentials, the rest via anonymous. The same for loki which is installed on two admin nodes, one of which I tried adding the new credentials. But all ceph pulls where done via anonymous access. The strange part is that the first is done as ceph:v18.2.6-20250417 and the rest via ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551
Op 24-04-2025 18:57 CEST schreef Eugen Block <eblock@nde.ag>:
Did you run 'ceph cephadm registry-login' or only 'sudo cephadm -v registry-login --registry-json login'? Because the first would login only the local node into the registry, the latter does it for all nodes in the host list.
Zitat von Sake Ceph <ceph@paulusma.eu>:
We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine.
I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials.
When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image".
But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [
"harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551"
] }
I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password
So what's wrong? Currently using 18.2.4. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I shouldn't have read your email last night and thinking I knew what to do. So I opened the shell (sudo /sbin/cephadm shell) and used there "cephadm registry-login". # ceph cephadm registry-login -i creds.json registry login scheduled That looks promising, no logging, so waited a couple of minutes. Started the upgrade again (it's a staging environment) and it's working :D I feel such a noob...
Op 25-04-2025 10:37 CEST schreef Eugen Block <eblock@nde.ag>:
What I wanted to know is, did you execute the login remotely for all hosts by running
sudo ceph cephadm -v registry-login --registry-json login
or only
sudo cephadm -v registry-login --registry-json login
to login only the host where you issued that command? But since you mention that all hosts have already pulled the image it's not that relevant anymore.
Zitat von Sake Ceph <ceph@paulusma.eu>:
That isn't a option: $ cephadm -v registry-login --registry-json login ERROR: cephadm should be run as root
So I enabled temporarily anonymous access for the project at the moment. But I noticed something in the logs of Harbor. All the ceph images got pulled via anonymous user, but one pull for promtail (deployed on all nodes) was done with the provided credentials, the rest via anonymous. The same for loki which is installed on two admin nodes, one of which I tried adding the new credentials. But all ceph pulls where done via anonymous access. The strange part is that the first is done as ceph:v18.2.6-20250417 and the rest via ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551
Op 24-04-2025 18:57 CEST schreef Eugen Block <eblock@nde.ag>:
Did you run 'ceph cephadm registry-login' or only 'sudo cephadm -v registry-login --registry-json login'? Because the first would login only the local node into the registry, the latter does it for all nodes in the host list.
Zitat von Sake Ceph <ceph@paulusma.eu>:
We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine.
I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials.
When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image".
But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [
"harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551"
] }
I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password
So what's wrong? Currently using 18.2.4. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ah, glad it worked for you. :-) Zitat von Sake Ceph <ceph@paulusma.eu>:
I shouldn't have read your email last night and thinking I knew what to do.
So I opened the shell (sudo /sbin/cephadm shell) and used there "cephadm registry-login". # ceph cephadm registry-login -i creds.json registry login scheduled
That looks promising, no logging, so waited a couple of minutes.
Started the upgrade again (it's a staging environment) and it's working :D I feel such a noob...
Op 25-04-2025 10:37 CEST schreef Eugen Block <eblock@nde.ag>:
What I wanted to know is, did you execute the login remotely for all hosts by running
sudo ceph cephadm -v registry-login --registry-json login
or only
sudo cephadm -v registry-login --registry-json login
to login only the host where you issued that command? But since you mention that all hosts have already pulled the image it's not that relevant anymore.
Zitat von Sake Ceph <ceph@paulusma.eu>:
That isn't a option: $ cephadm -v registry-login --registry-json login ERROR: cephadm should be run as root
So I enabled temporarily anonymous access for the project at the moment. But I noticed something in the logs of Harbor. All the ceph images got pulled via anonymous user, but one pull for promtail (deployed on all nodes) was done with the provided credentials, the rest via anonymous. The same for loki which is installed on two admin nodes, one of which I tried adding the new credentials. But all ceph pulls where done via anonymous access. The strange part is that the first is done as ceph:v18.2.6-20250417 and the rest via
ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551
Op 24-04-2025 18:57 CEST schreef Eugen Block <eblock@nde.ag>:
Did you run 'ceph cephadm registry-login' or only 'sudo cephadm -v registry-login --registry-json login'? Because the first would login only the local node into the registry, the latter does it for all nodes in the host list.
Zitat von Sake Ceph <ceph@paulusma.eu>:
We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine.
I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials.
When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image".
But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image
harbor.url/project/ceph/ceph:v18.2.6-20250417 pull
Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [
"harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551"
] }
I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password
So what's wrong? Currently using 18.2.4. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Adam King
-
Eugen Block
-
Sake Ceph