aha found it. The mon store seemed not to assimilate the ceph config. We changed it and now it works: # ceph config dump |grep auth global advanced auth_client_required none * global advanced auth_cluster_required none * global advanced auth_service_required none Am Fr., 15. Sept. 2023 um 13:01 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Oh, we found the issue. A very old update was stuck in the pipeline. We canceled it and then the correct images got pulled.
Now on to the next issue. Daemons that start have problems talking to the cluster
# podman logs 72248bafb0d3 2023-09-15T10:47:30.740+0000 7f2943559700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [1] but i only support [1] 2023-09-15T10:47:30.740+0000 7f294ac601c0 -1 mgr init Authentication failed, did you specify a mgr ID with a valid keyring? Error in initialization: (13) Permission denied
When we add the following lines to the mgr config and restart the daemon, it works flawlessly auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx
Did I miss some config value that needs to be set?
Trying the same with a new mon, will not work. 2023-09-15T10:59:28.960+0000 7fc851a77700 -1 mon.0cc47a6df330@-1(probing) e0 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported 2023-09-15T10:59:32.164+0000 7fc851a77700 -1 mon.0cc47a6df330@-1(probing) e0 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported 2023-09-15T10:59:38.568+0000 7fc851a77700 -1 mon.0cc47a6df330@-1(probing) e0 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported
I added the mon via: ceph orch daemon add mon FQDN:[IPv6_address]
Am Fr., 15. Sept. 2023 um 09:21 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi Stefan,
the cluster is running 17.6.2 through the board. The mentioned container with other version don't show in the ceph -s or ceph verions. It looks like it is host related. One host get the correct 17.2.6 images, one get the 16.2.11 images and the third one uses the 7.0.0-7183-g54142666 (whatever this is) images.
root@0cc47a6df330:~# ceph config-key get config/global/container_image Error ENOENT:
root@0cc47a6df330:~# ceph config-key list |grep container_image "config-history/12/+mgr.0cc47a6df14e/container_image", "config-history/13/+mgr.0cc47aad8ce8/container_image", "config/mgr.0cc47a6df14e/container_image", "config/mgr.0cc47aad8ce8/container_image",
I've tried to set the detault image to ceph config-key set config/global/container_image quay.io/ceph/ceph:v17.2.6@sha256:6b0a24e3146d4723700ce6579d40e6016b2c63d9bf90422653f2d4caa49be232 But I can not redeploy the mgr daemons, because there is no standby daemon.
root@0cc47a6df330:~# ceph orch redeploy mgr Error EINVAL: Unable to schedule redeploy for mgr.0cc47aad8ce8: No standby MGR
But there should be: root@0cc47a6df330:~# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.0cc47a6df14e.iltiot 0cc47a6df14e *:9283 running (23s) 22s ago 2m 10.6M - 16.2.11 de4b0b384ad4 0f31a162fa3e mgr.0cc47aad8ce8 0cc47aad8ce8 running (16h) 8m ago 16h 591M - 17.2.6 22cd8daf4d70 8145c63fdc44
root@0cc47a6df330:~# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 2/2 8m ago 19h 0cc47a6df14e;0cc47a6df330;0cc47aad8ce8
I've also remove podman and containerd, kill all directories and then do a fresh reinstall of podman, which also did not work. It's also strange that the daemons with the wonky version got an extra suffix.
If I would now how, I would happily nuke the whole orchestrator, podman and everything that goes along with it, and start over. In the end it is not that hard to start some mgr/mon daemons without podman, so I would be back to a classical cluster. I tried this yesterday, but the daemons still use that very strange images and I just don't understand why.
I could just nuke the whole dev cluster, wipe all disks and start fresh after reinstalling the hosts, but as I have to adopt 17 clusters to the orchestrator, I rather get some learnings from the not working thing :)
Am Fr., 15. Sept. 2023 um 08:26 Uhr schrieb Stefan Kooman <stefan@bit.nl
:
Hi, I currently try to adopt our stage cluster, some hosts just pull strange images.
root@0cc47a6df330:/var/lib/containers/storage/overlay-images# podman
On 14-09-2023 17:49, Boris Behrens wrote: ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a532c37ebe42 docker.io/ceph/daemon-base:latest-master-devel -n mgr.0cc47a6df3... 2 minutes ago Up 2 minutes ago ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df330-fxrfyl
root@0cc47a6df330:~# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.0cc47a6df14e.vqizdz 0cc47a6df14e.f00f.gridscale.dev *:9283 running (3m) 3m ago 3m 10.8M - 16.2.11 de4b0b384ad4 00b02cd82a1c mgr.0cc47a6df330.iijety 0cc47a6df330.f00f.gridscale.dev *:9283 running (5s) 2s ago 4s 10.5M - 17.0.0-7183-g54142666 75e3d7089cea 662c6baa097e mgr.0cc47aad8ce8 0cc47aad8ce8.f00f.gridscale.dev running (65m) 8m ago 60m 553M - 17.2.6 22cd8daf4d70 8145c63fdc44
Any idea what I need to do to change that?
I want to get some things cleared up. What is the version you are running? I see three different ceph versions active now. I see you are running a podman ps command, but see docker images pulled. AFAIK podman needs a different IMAGE than docker ... or do you have a mixed setup?
What does "ceph config-key get config/global/container_image" give you?
ceph config-key list |grep container_image should give you a list (including config-history) where you can see what has been configured before.
cephadm logs might give a clue as well.
You can configure the IMAGE version / type that you want by setting the key and redeploy affected containers: For example (18.1.2):
ceph config-key set config/global/container_image
quay.io/ceph/ceph:v18.1.2@sha256:82a380c8127c42da406b7ce1281c2f3c0a86d4ba04b1f4b5f8d1036b8c24784f
Gr. Stefan
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.