There's a thread on Slack (https://ceph-storage.slack.com/archives/C1HFU4JK1/p1789057907366559?thread_ts=1789050849.632739&cid=C1HFU4JK1) where someone reported the same (or very similar) for a large cluster with more than 1000 OSDs. I proposed this: ---snip--- ... I would run the upgrade for a single OSD again: ceph orch upgrade start --version 19.2.6 --daemon-types osd --limit 1 The only thing is, versions 19.2.6 and 20.2.4 are "broken" in the sense of applying the provided limit in a reasonable way. Most likely, it would redeploy an entire OSD host instead of a single OSD. I did a lot tests with staggered upgrades recently and this is what I observed. Anyway, that would at least reduce the upgrade time window and hopefully rotate keyrings properly. There's also a --host parameter to limit upgrades to a host, but apparently there's a limit of 16 OSDs at a time. My expectation is that this would only rotate and redeploy the OSDs (since they already have been upgraded there's almost no delay with image pull etc.). The only issue with this staggered upgrade approach is that each time you issue the upgrade command (let's say you run it as a batch per host or whatever) it will redeploy mgr and mon daemons as well and rotate their keys again since it's part of the upgrade procedure. So my approach would be to do only run one staggered upgrade (--daemon-types osd --limit 1 ) see if that does the trick for that batch and then start the upgrade again for daemon-types osd but without a limit parameter which would then (hopefully) rorate redeploy the remaining OSDs. It could still end up in a timeout since there are so many OSDs, hard to tell. (I don't even know right now if it's actually a timeout somewhere.) ---snip--- In the mentioned thread Greg confirmed that he observed a keyring rotation for the OSDs. But it will probably take some more time until it's finished, he reported that it took 13 days on the first attempt. Zitat von Frank Ederveen <ceph-users@ceph.io>:
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both.
It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains.
All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd.
Is there going to be a new release soon, or some scripting which fixes things?
health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ...
Thanks, Frank
________________________________ From: Eugen Block <ceph-users@ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
Zitat von Brent Kennedy <ceph-users@ceph.io>:
Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters.
Regards, -Brent
Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
-----Original Message----- From: Patrick Donnelly <ceph-users@ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad <samdto987@gmail.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Hello,
On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users@ceph.io> wrote:
Hi,
While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available?
Step (9) of [1] says:
"Then copy and import the key to each machine using that client.$ID credential."
[1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u...
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io