...

I currently try to adopt our stage cluster, some hosts just pull strange images.

root@0cc47a6df330:/var/lib/containers/storage/overlay-images# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a532c37ebe42 docker.io/ceph/daemon-base:latest-master-devel -n mgr.0cc47a6df3... 2 minutes ago Up 2 minutes ago ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df330-fxrfyl

root@0cc47a6df330:~# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.0cc47a6df14e.vqizdz 0cc47a6df14e.f00f.gridscale.dev *:9283 running (3m) 3m ago 3m 10.8M - 16.2.11 de4b0b384ad4 00b02cd82a1c mgr.0cc47a6df330.iijety 0cc47a6df330.f00f.gridscale.dev *:9283 running (5s) 2s ago 4s 10.5M - 17.0.0-7183-g54142666 75e3d7089cea 662c6baa097e mgr.0cc47aad8ce8 0cc47aad8ce8.f00f.gridscale.dev running (65m) 8m ago 60m 553M - 17.2.6 22cd8daf4d70 8145c63fdc44

Any idea what I need to do to change that?

I want to get some things cleared up. What is the version you are running? I see three different ceph versions active now. I see you are running a podman ps command, but see docker images pulled. AFAIK podman needs a different IMAGE than docker ... or do you have a mixed setup?

Podman does not need different images. I think lots of CO use the docker image format. Afaik podman is mostly a fork of docker

On 15-09-2023 09:21, Boris Behrens wrote:

Hi Stefan,

the cluster is running 17.6.2 through the board. The mentioned container with other version don't show in the ceph -s or ceph verions. It looks like it is host related. One host get the correct 17.2.6 images, one get the 16.2.11 images and the third one uses the 7.0.0-7183-g54142666 (whatever this is) images.

root@0cc47a6df330:~# ceph config-key get config/global/container_image Error ENOENT:

root@0cc47a6df330:~# ceph config-key list |grep container_image     "config-history/12/+mgr.0cc47a6df14e/container_image",     "config-history/13/+mgr.0cc47aad8ce8/container_image",     "config/mgr.0cc47a6df14e/container_image",     "config/mgr.0cc47aad8ce8/container_image",

I've tried to set the detault image to ceph config-key set config/global/container_image quay.io/ceph/ceph:v17.2.6@sha256:6b0a24e3146d4723700ce6579d40e6016b2c63d9bf90422653f2d4caa49be232 <http://quay.io/ceph/ceph:v17.2.6@sha256:6b0a24e3146d4723700ce6579d40e6016b2c63d9bf90422653f2d4caa49be232> But I can not redeploy the mgr daemons, because there is no standby daemon.

root@0cc47a6df330:~# ceph orch redeploy mgr Error EINVAL: Unable to schedule redeploy for mgr.0cc47aad8ce8: No standby MGR

But there should be: root@0cc47a6df330:~# ceph orch ps NAME                     HOST                             PORTS   STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION    IMAGE ID  CONTAINER ID mgr.0cc47a6df14e.iltiot  0cc47a6df14e  *:9283  running (23s)    22s ago   2m    10.6M        -  16.2.11    de4b0b384ad4  0f31a162fa3e mgr.0cc47aad8ce8         0cc47aad8ce8          running (16h)     8m ago  16h     591M        -  17.2.6     22cd8daf4d70  8145c63fdc44

I guess that one of the managers is not working correctly (probably the 16.2.11 version). IIRC I have changed the image reference for a container (systemd unit files) once, when I managed to redeploy all containers with a non-working image (test setup). so first make sure what manager is actually running, then try to fix the other one by editing the relevant config for that container (point it to the same image as the running container). Pull necessary image first if need be. After you've got a standby manager up and running, you can redeploy the necessary daemons. Be careful ... there are commands that redeploy all daemons at the same time, you don't want to do that normally ;-).

root@0cc47a6df330:~# ceph orch ls NAME  PORTS  RUNNING  REFRESHED  AGE  PLACEMENT mgr              2/2  8m ago     19h  0cc47a6df14e;0cc47a6df330;0cc47aad8ce8

I've also remove podman and containerd, kill all directories and then do a fresh reinstall of podman, which also did not work. It's also strange that the daemons with the wonky version got an extra suffix.

If I would now how, I would happily nuke the whole orchestrator, podman and everything that goes along with it, and start over. In the end it is not that hard to start some mgr/mon daemons without podman, so I would be back to a classical cluster. I tried this yesterday, but the daemons still use that very strange images and I just don't understand why.

I could just nuke the whole dev cluster, wipe all disks and start fresh after reinstalling the hosts, but as I have to adopt 17 clusters to the orchestrator, I rather get some learnings from the not working thing :)

There is actually a cephadm "kill it with fire" option to do that for you, but yeah, make sure you know how to fix it when things do not go according to plan. It all magically works, until it doesn't ;-). Good luck, and keep us updated with any further challenges / progress. Gr. Stefan

Hi, someone else had a similar issue [1], to set the global container image you can run: $ ceph config set global container_image my-registry:5000/ceph/ceph:v17.2.6 I usually change that as soon as a cluster is up and running or after an upgrade so there's no risk of pulling wrong container images (I assume in your case the local cephadm versions on the hosts differ and therefore each one pulls a different default image hard-coded in the cephadm binary). You should probably be able to start a mgr daemon by changing the unit.run file temporarily and replace "CONTAINER_IMAGE" with a correct image version (stop the pod first): CONTAINER_IMAGE=my-registry/ceph/ceph-quincy@v17.2.6 (this is just an example). The same line contains another image reference which you should change. Then restart that pod (e. g. with systemctl), hopefully you'll have a MGR up and running to be able to use the orchestrator again. This procedure helped me in the past. Regards, Eugen [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/THAH2JFQNB7B... Zitat von Stefan Kooman <stefan@bit.nl>:

On 15-09-2023 10:25, Stefan Kooman wrote:

...

...

I could just nuke the whole dev cluster, wipe all disks and start fresh after reinstalling the hosts, but as I have to adopt 17 clusters to the orchestrator, I rather get some learnings from the not working thing 🙂

There is actually a cephadm "kill it with fire" option to do that for you, but yeah, make sure you know how to fix it when things do not go according to plan. It all magically works, until it doesn't 😉.

cephadm rm-cluster --fsid your-fsid-here --force

... ss a last resort (short of wipefs / shred on all disks).

Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

aha found it. The mon store seemed not to assimilate the ceph config. We changed it and now it works: # ceph config dump |grep auth global advanced auth_client_required none * global advanced auth_cluster_required none * global advanced auth_service_required none Am Fr., 15. Sept. 2023 um 13:01 Uhr schrieb Boris Behrens <bb@kervyn.de>:

Oh, we found the issue. A very old update was stuck in the pipeline. We canceled it and then the correct images got pulled.

Now on to the next issue. Daemons that start have problems talking to the cluster

# podman logs 72248bafb0d3 2023-09-15T10:47:30.740+0000 7f2943559700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [1] but i only support [1] 2023-09-15T10:47:30.740+0000 7f294ac601c0 -1 mgr init Authentication failed, did you specify a mgr ID with a valid keyring? Error in initialization: (13) Permission denied

When we add the following lines to the mgr config and restart the daemon, it works flawlessly auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx

Did I miss some config value that needs to be set?

Trying the same with a new mon, will not work. 2023-09-15T10:59:28.960+0000 7fc851a77700 -1 mon.0cc47a6df330@-1(probing) e0 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported 2023-09-15T10:59:32.164+0000 7fc851a77700 -1 mon.0cc47a6df330@-1(probing) e0 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported 2023-09-15T10:59:38.568+0000 7fc851a77700 -1 mon.0cc47a6df330@-1(probing) e0 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported

I added the mon via: ceph orch daemon add mon FQDN:[IPv6_address]

Am Fr., 15. Sept. 2023 um 09:21 Uhr schrieb Boris Behrens <bb@kervyn.de>:

...

Hi Stefan,

the cluster is running 17.6.2 through the board. The mentioned container with other version don't show in the ceph -s or ceph verions. It looks like it is host related. One host get the correct 17.2.6 images, one get the 16.2.11 images and the third one uses the 7.0.0-7183-g54142666 (whatever this is) images.

root@0cc47a6df330:~# ceph config-key get config/global/container_image Error ENOENT:

root@0cc47a6df330:~# ceph config-key list |grep container_image "config-history/12/+mgr.0cc47a6df14e/container_image", "config-history/13/+mgr.0cc47aad8ce8/container_image", "config/mgr.0cc47a6df14e/container_image", "config/mgr.0cc47aad8ce8/container_image",

I've tried to set the detault image to ceph config-key set config/global/container_image quay.io/ceph/ceph:v17.2.6@sha256:6b0a24e3146d4723700ce6579d40e6016b2c63d9bf90422653f2d4caa49be232 But I can not redeploy the mgr daemons, because there is no standby daemon.

root@0cc47a6df330:~# ceph orch redeploy mgr Error EINVAL: Unable to schedule redeploy for mgr.0cc47aad8ce8: No standby MGR

But there should be: root@0cc47a6df330:~# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.0cc47a6df14e.iltiot 0cc47a6df14e *:9283 running (23s) 22s ago 2m 10.6M - 16.2.11 de4b0b384ad4 0f31a162fa3e mgr.0cc47aad8ce8 0cc47aad8ce8 running (16h) 8m ago 16h 591M - 17.2.6 22cd8daf4d70 8145c63fdc44

root@0cc47a6df330:~# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 2/2 8m ago 19h 0cc47a6df14e;0cc47a6df330;0cc47aad8ce8

I've also remove podman and containerd, kill all directories and then do a fresh reinstall of podman, which also did not work. It's also strange that the daemons with the wonky version got an extra suffix.

If I would now how, I would happily nuke the whole orchestrator, podman and everything that goes along with it, and start over. In the end it is not that hard to start some mgr/mon daemons without podman, so I would be back to a classical cluster. I tried this yesterday, but the daemons still use that very strange images and I just don't understand why.

I could just nuke the whole dev cluster, wipe all disks and start fresh after reinstalling the hosts, but as I have to adopt 17 clusters to the orchestrator, I rather get some learnings from the not working thing :)

Am Fr., 15. Sept. 2023 um 08:26 Uhr schrieb Stefan Kooman <stefan@bit.nl

...

:

...

...

Hi, I currently try to adopt our stage cluster, some hosts just pull strange images.

root@0cc47a6df330:/var/lib/containers/storage/overlay-images# podman

On 14-09-2023 17:49, Boris Behrens wrote: ps

...

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a532c37ebe42 docker.io/ceph/daemon-base:latest-master-devel -n mgr.0cc47a6df3... 2 minutes ago Up 2 minutes ago ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df330-fxrfyl

root@0cc47a6df330:~# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.0cc47a6df14e.vqizdz 0cc47a6df14e.f00f.gridscale.dev *:9283 running (3m) 3m ago 3m 10.8M - 16.2.11 de4b0b384ad4 00b02cd82a1c mgr.0cc47a6df330.iijety 0cc47a6df330.f00f.gridscale.dev *:9283 running (5s) 2s ago 4s 10.5M - 17.0.0-7183-g54142666 75e3d7089cea 662c6baa097e mgr.0cc47aad8ce8 0cc47aad8ce8.f00f.gridscale.dev running (65m) 8m ago 60m 553M - 17.2.6 22cd8daf4d70 8145c63fdc44

Any idea what I need to do to change that?

I want to get some things cleared up. What is the version you are running? I see three different ceph versions active now. I see you are running a podman ps command, but see docker images pulled. AFAIK podman needs a different IMAGE than docker ... or do you have a mixed setup?

What does "ceph config-key get config/global/container_image" give you?

ceph config-key list |grep container_image should give you a list (including config-history) where you can see what has been configured before.

cephadm logs might give a clue as well.

You can configure the IMAGE version / type that you want by setting the key and redeploy affected containers: For example (18.1.2):

ceph config-key set config/global/container_image

quay.io/ceph/ceph:v18.1.2@sha256:82a380c8127c42da406b7ce1281c2f3c0a86d4ba04b1f4b5f8d1036b8c24784f

Gr. Stefan

-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.

-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.

-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.