Hi, Yes there are operation that you have to do by yourself. The upgrade only rotate the mon keys automatically. I tried to clarify this on this post : https://pepitedata.com/ceph-tentacle-squid-security-release/ Where you'll find some code snippet to help like: ``` wait_for_active_clean() { local timeout=${1:-1200} waited=0 not_clean while true; do # scrubbing is normal background activity; block only on states that are # not a superset of active+clean (degraded, undersized, peering, ...) not_clean=$(ceph -s -f json | jq '[.pgmap.pgs_by_state[] | select(.state_name | (startswith("active+clean") | not))] | length') [[ "${not_clean}" -eq 0 ]] && { echo " active+clean (scrubbing ignored)."; return 0; } (( waited >= timeout )) && { echo " TIMEOUT after ${timeout}s, ${not_clean} PG group(s) still not active+clean" >&2; return 1; } echo " waiting for active+clean (${not_clean} PG group(s) not clean, ${waited}s elapsed)..." sleep 10 waited=$(( waited + 10 )) done } hostlist=$(ceph orch ps --daemon-type osd -f json | jq -r '.[].hostname' | sort -u) for host in $hostlist; do echo "=== ${host} ===" ceph orch host ok-to-stop ${host} || continue ceph orch host maintenance enter ${host} || continue failed_osds="" for osdid in $(ceph orch ps --hostname=${host} --daemon-type osd -f json | jq -r '.[].daemon_id'); do echo "--- osd.${osdid} ---" newkey=$(ceph auth rotate --key-type=aes256k osd.${osdid} | grep -m1 '^[[:space:]]*key = ' | sed -E 's/^[[:space:]]*key = //') label_written=0 if [[ -n "${newkey}" ]]; then # maintenance enter does not guarantee the OSD container has released the # device yet (bluefs/rocksdb close can lag a few seconds); retry on busy for attempt in 1 2 3 4 5; do if ssh -i /tmp/cephadm_id_rsa -o StrictHostKeyChecking=accept-new ${cephadm_user}@${host} "sudo cephadm --image ${image} shell --name osd.${osdid} -- bash -c 'dev=$(readlink -f /var/lib/ceph/osd/ceph-${osdid}/block); ceph-bluestore-tool --dev $dev set-label-key --key osd_key -v ${newkey}'"; then label_written=1 break fi echo " device busy, retrying osd.${osdid} in 5s (attempt ${attempt})..." >&2 sleep 5 done fi if [[ ${label_written} -eq 0 ]]; then echo " !! FAILED rotating osd.${osdid} (keyring/label mismatch), it will not come back up on its own" >&2 failed_osds="${failed_osds} ${osdid}" fi done if [[ -n "${failed_osds}" ]]; then echo "!! osd(s) left in a bad state on ${host}: ${failed_osds} - fix manually before exiting maintenance" >&2 break fi ceph orch host maintenance exit ${host} || { echo "!! maintenance exit failed on ${host}" >&2; break; } maintenance exit should start every daemon on the host; confirm it did, since a failed rotation above leaves the OSD down for good for osdid in $(ceph orch ps --hostname=${host} --daemon-type osd -f json | jq -r '.[].daemon_id'); do status=$(ceph orch ps --hostname=${host} --daemon-type osd --daemon-id=${osdid} --refresh -f json | jq -r '.[0].status_desc') if [[ "${status}" != "running" ]]; then echo " osd.${osdid} is '${status}' after maintenance exit, forcing start" >&2 ceph orch daemon start osd.${osdid} fi done ceph orch ps --hostname=${host} --daemon-type osd --refresh wait_for_active_clean || break done ``` On 11/2/26 16:04, Robert Sander wrote:

Hi,

after upgrading a cephadm managed cluster to 20.2.4 the keys for MON, MGR, OSD and MDS are rotated to the new aes256k format.

What has to be done for the ceph-exporter, crash, nfs, rgw, smb and other services managed by cephadm and the bootstrap keys?

A "ceph orch redeploy service_type" does not update the keys.

Regards

Yes, in my opinion, OSDs are the most difficult to rotate, whereas the other components can be rotated relatively easily. It is also necessary to anticipate a period of user downtime for Kubernetes PVC creation, as this involves the RBD client keys used by the Ceph-CSI driver on a Kubernetes cluster. On 11/2/26 16:20, Robert Sander wrote:

Hi,

Am 02.11.26 um 4:09 PM schrieb Julien Laurenceau:

...

Yes there are operation that you have to do by yourself. The upgrade only rotate the mon keys automatically I tried to clarify this on this post : https://pepitedata.com/ceph- tentacle-squid-security-release/

Where you'll find some code snippet to help like:

This only covers OSDs which are handled automatically by cephadm.

Why is there no automatic key rotation for RGW, NFS, crash, ceph-exporter and other services managed by cephadm?

I now have to deal with this list manually?

[WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 51 auth client entities with insecure key types     entity client.admin using insecure key type: aes     entity client.bootstrap-mds using insecure key type: aes     entity client.bootstrap-mgr using insecure key type: aes     entity client.bootstrap-osd using insecure key type: aes     entity client.bootstrap-rbd using insecure key type: aes     entity client.bootstrap-rbd-mirror using insecure key type: aes     entity client.bootstrap-rgw using insecure key type: aes     entity client.ceph-exporter.test-cephadm using insecure key type: aes     entity client.ceph-exporter.test-mds01 using insecure key type: aes     entity client.ceph-exporter.test-mds02 using insecure key type: aes     entity client.ceph-exporter.test-mds03 using insecure key type: aes     entity client.ceph-exporter.test-mon01 using insecure key type: aes     entity client.ceph-exporter.test-mon02 using insecure key type: aes     entity client.ceph-exporter.test-mon03 using insecure key type: aes     entity client.ceph-exporter.test-osd01 using insecure key type: aes     entity client.ceph-exporter.test-osd02 using insecure key type: aes     entity client.ceph-exporter.test-osd03 using insecure key type: aes     entity client.ceph-exporter.test-osd04 using insecure key type: aes     entity client.ceph-exporter.test-osd05 using insecure key type: aes     entity client.ceph-exporter.test-osd06 using insecure key type: aes     entity client.ceph-exporter.test-osd07 using insecure key type: aes     entity client.ceph-exporter.test-osd08 using insecure key type: aes     entity client.ceph-exporter.test-osd09 using insecure key type: aes     entity client.crash.test-cephadm using insecure key type: aes     entity client.crash.test-mds01 using insecure key type: aes     entity client.crash.test-mds02 using insecure key type: aes     entity client.crash.test-mds03 using insecure key type: aes     entity client.crash.test-mon01 using insecure key type: aes     entity client.crash.test-mon02 using insecure key type: aes     entity client.crash.test-mon03 using insecure key type: aes     entity client.crash.test-osd01 using insecure key type: aes     entity client.crash.test-osd02 using insecure key type: aes     entity client.crash.test-osd03 using insecure key type: aes     entity client.crash.test-osd04 using insecure key type: aes     entity client.crash.test-osd05 using insecure key type: aes     entity client.crash.test-osd06 using insecure key type: aes     entity client.crash.test-osd07 using insecure key type: aes     entity client.crash.test-osd08 using insecure key type: aes     entity client.crash.test-osd09 using insecure key type: aes     entity client.nfs.nfs01.0.4.test-mon02.wgitjt using insecure key type: aes     entity client.nfs.nfs01.0.4.test-mon02.wgitjt-rgw using insecure key type: aes     entity client.nfs.nfs01.1.7.test-mon01.fvhlxu using insecure key type: aes     entity client.nfs.nfs01.1.7.test-mon01.fvhlxu-rgw using insecure key type: aes     entity client.nfs.nfs01.2.0.test-mon03.ydsjfi using insecure key type: aes     entity client.nfs.nfs01.2.0.test-mon03.ydsjfi-rgw using insecure key type: aes     entity client.nfs.nfs01.test.8d14d02e using insecure key type: aes     entity client.rgw.test.test-mds01.wgifhi using insecure key type: aes     entity client.rgw.test.test-mds02.lebwqv using insecure key type: aes     entity client.rgw.test.test-mds03.jzkseo using insecure key type: aes     entity client.shares-test using insecure key type: aes     entity client.shares-test-lock using insecure key type: aes

client.admin and the two client.shares* keys I understand, these are to be treated manually. But the rest could be rotated automatically IMHO.

Regards

When using managed OSD the keys are not rotated automatically On 11/2/26 16:33, Julien Laurenceau wrote:

Yes, in my opinion, OSDs are the most difficult to rotate, whereas the other components can be rotated relatively easily.

It is also necessary to anticipate a period of user downtime for Kubernetes PVC creation, as this involves the RBD client keys used by the Ceph-CSI driver on a Kubernetes cluster.

On 11/2/26 16:20, Robert Sander wrote:

...

Hi,

Am 02.11.26 um 4:09 PM schrieb Julien Laurenceau:

...

Yes there are operation that you have to do by yourself. The upgrade only rotate the mon keys automatically I tried to clarify this on this post : https://pepitedata.com/ceph- tentacle-squid-security-release/

Where you'll find some code snippet to help like:

This only covers OSDs which are handled automatically by cephadm.

Why is there no automatic key rotation for RGW, NFS, crash, ceph-exporter and other services managed by cephadm?

I now have to deal with this list manually?

[WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 51 auth client entities with insecure key types     entity client.admin using insecure key type: aes     entity client.bootstrap-mds using insecure key type: aes     entity client.bootstrap-mgr using insecure key type: aes     entity client.bootstrap-osd using insecure key type: aes     entity client.bootstrap-rbd using insecure key type: aes     entity client.bootstrap-rbd-mirror using insecure key type: aes     entity client.bootstrap-rgw using insecure key type: aes     entity client.ceph-exporter.test-cephadm using insecure key type: aes     entity client.ceph-exporter.test-mds01 using insecure key type: aes     entity client.ceph-exporter.test-mds02 using insecure key type: aes     entity client.ceph-exporter.test-mds03 using insecure key type: aes     entity client.ceph-exporter.test-mon01 using insecure key type: aes     entity client.ceph-exporter.test-mon02 using insecure key type: aes     entity client.ceph-exporter.test-mon03 using insecure key type: aes     entity client.ceph-exporter.test-osd01 using insecure key type: aes     entity client.ceph-exporter.test-osd02 using insecure key type: aes     entity client.ceph-exporter.test-osd03 using insecure key type: aes     entity client.ceph-exporter.test-osd04 using insecure key type: aes     entity client.ceph-exporter.test-osd05 using insecure key type: aes     entity client.ceph-exporter.test-osd06 using insecure key type: aes     entity client.ceph-exporter.test-osd07 using insecure key type: aes     entity client.ceph-exporter.test-osd08 using insecure key type: aes     entity client.ceph-exporter.test-osd09 using insecure key type: aes     entity client.crash.test-cephadm using insecure key type: aes     entity client.crash.test-mds01 using insecure key type: aes     entity client.crash.test-mds02 using insecure key type: aes     entity client.crash.test-mds03 using insecure key type: aes     entity client.crash.test-mon01 using insecure key type: aes     entity client.crash.test-mon02 using insecure key type: aes     entity client.crash.test-mon03 using insecure key type: aes     entity client.crash.test-osd01 using insecure key type: aes     entity client.crash.test-osd02 using insecure key type: aes     entity client.crash.test-osd03 using insecure key type: aes     entity client.crash.test-osd04 using insecure key type: aes     entity client.crash.test-osd05 using insecure key type: aes     entity client.crash.test-osd06 using insecure key type: aes     entity client.crash.test-osd07 using insecure key type: aes     entity client.crash.test-osd08 using insecure key type: aes     entity client.crash.test-osd09 using insecure key type: aes     entity client.nfs.nfs01.0.4.test-mon02.wgitjt using insecure key type: aes     entity client.nfs.nfs01.0.4.test-mon02.wgitjt-rgw using insecure key type: aes     entity client.nfs.nfs01.1.7.test-mon01.fvhlxu using insecure key type: aes     entity client.nfs.nfs01.1.7.test-mon01.fvhlxu-rgw using insecure key type: aes     entity client.nfs.nfs01.2.0.test-mon03.ydsjfi using insecure key type: aes     entity client.nfs.nfs01.2.0.test-mon03.ydsjfi-rgw using insecure key type: aes     entity client.nfs.nfs01.test.8d14d02e using insecure key type: aes     entity client.rgw.test.test-mds01.wgifhi using insecure key type: aes     entity client.rgw.test.test-mds02.lebwqv using insecure key type: aes     entity client.rgw.test.test-mds03.jzkseo using insecure key type: aes     entity client.shares-test using insecure key type: aes     entity client.shares-test-lock using insecure key type: aes

client.admin and the two client.shares* keys I understand, these are to be treated manually. But the rest could be rotated automatically IMHO.

Regards

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, Am 02.11.26 um 4:23 PM schrieb Eugen Block:

those are all client keyrings, cephadm doesn't touch them. It would have to assume which keyrings for which clients are safe to rotate which could be potentially dangerous, so I assume that's why it skips them entirely.

IMHO this is not what a Ceph admin would expect. Cephadm manages these services. There is not even a documented way to rotate the keys for an NFS service manually for example. https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u... tells me that cephadm does everything except for client keys. But there is nothing in section 9 "Rotate other client keys" about the services managed by cephadm that use client keys. I assume I have to ceph auth rotate --key-type=aes256k client.crash.test-cephadm ceph orch daemon redeploy crash.test-cephadm for each of them? Why is this not implemented in the orchestrator? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin