Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both. It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains. All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd. Is there going to be a new release soon, or some scripting which fixes things? health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ... Thanks, Frank ________________________________ From: Eugen Block <ceph-users@ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. Zitat von Brent Kennedy <ceph-users@ceph.io>:
Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters.
Regards, -Brent
Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
-----Original Message----- From: Patrick Donnelly <ceph-users@ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad <samdto987@gmail.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Hello,
On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users@ceph.io> wrote:
Hi,
While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available?
Step (9) of [1] says:
"Then copy and import the key to each machine using that client.$ID credential."
[1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u...
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There's a thread on Slack (https://ceph-storage.slack.com/archives/C1HFU4JK1/p1789057907366559?thread_ts=1789050849.632739&cid=C1HFU4JK1) where someone reported the same (or very similar) for a large cluster with more than 1000 OSDs. I proposed this: ---snip--- ... I would run the upgrade for a single OSD again: ceph orch upgrade start --version 19.2.6 --daemon-types osd --limit 1 The only thing is, versions 19.2.6 and 20.2.4 are "broken" in the sense of applying the provided limit in a reasonable way. Most likely, it would redeploy an entire OSD host instead of a single OSD. I did a lot tests with staggered upgrades recently and this is what I observed. Anyway, that would at least reduce the upgrade time window and hopefully rotate keyrings properly. There's also a --host parameter to limit upgrades to a host, but apparently there's a limit of 16 OSDs at a time. My expectation is that this would only rotate and redeploy the OSDs (since they already have been upgraded there's almost no delay with image pull etc.). The only issue with this staggered upgrade approach is that each time you issue the upgrade command (let's say you run it as a batch per host or whatever) it will redeploy mgr and mon daemons as well and rotate their keys again since it's part of the upgrade procedure. So my approach would be to do only run one staggered upgrade (--daemon-types osd --limit 1 ) see if that does the trick for that batch and then start the upgrade again for daemon-types osd but without a limit parameter which would then (hopefully) rorate redeploy the remaining OSDs. It could still end up in a timeout since there are so many OSDs, hard to tell. (I don't even know right now if it's actually a timeout somewhere.) ---snip--- In the mentioned thread Greg confirmed that he observed a keyring rotation for the OSDs. But it will probably take some more time until it's finished, he reported that it took 13 days on the first attempt. Zitat von Frank Ederveen <ceph-users@ceph.io>:
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both.
It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains.
All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd.
Is there going to be a new release soon, or some scripting which fixes things?
health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ...
Thanks, Frank
________________________________ From: Eugen Block <ceph-users@ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
Zitat von Brent Kennedy <ceph-users@ceph.io>:
Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters.
Regards, -Brent
Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
-----Original Message----- From: Patrick Donnelly <ceph-users@ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad <samdto987@gmail.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Hello,
On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users@ceph.io> wrote:
Hi,
While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available?
Step (9) of [1] says:
"Then copy and import the key to each machine using that client.$ID credential."
[1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u...
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I think the problem are the lockbox keys, not the OSD keys themselves (and the bootstrap and crash keys, but those are easy to solve). The lockbox keys are used to obtain the dmcrypt secret for accessing encrypted OSD devices. And as far as I have followed the discussion on the mailing list, not much was said about them at all. They seem to be ignored by various parties, since they are not used in the network communication. A user from IRC has put together a little script to handle these keys. It is available at https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate Thx to sur5r for creating the script. I've successfully used to script on our ~900 OSD cluster. Works like charm, but you need to ensure that: - the admin key is available on all hosts - OSDs are stopped prior to running the script (check with "ps -u ceph") Background: the lockbox key is stored in LVM tags; so rotating the key also requires updating the tags. For both data and block device if applicable. Since key rotation only changes the encoding and representation of the key, the key itself is untouched. You do not need to reencrypt your OSDs. Caveat: no clue whether the script works with cephadm or whether cephadm should handle lockbox key rotation itself Best regards, Burkhard On 9/15/26 08:32, Eugen Block wrote:
There's a thread on Slack (https://ceph-storage.slack.com/archives/C1HFU4JK1/p1789057907366559?thread_ts=1789050849.632739&cid=C1HFU4JK1) where someone reported the same (or very similar) for a large cluster with more than 1000 OSDs. I proposed this:
---snip--- ... I would run the upgrade for a single OSD again:
ceph orch upgrade start --version 19.2.6 --daemon-types osd --limit 1
The only thing is, versions 19.2.6 and 20.2.4 are "broken" in the sense of applying the provided limit in a reasonable way. Most likely, it would redeploy an entire OSD host instead of a single OSD. I did a lot tests with staggered upgrades recently and this is what I observed. Anyway, that would at least reduce the upgrade time window and hopefully rotate keyrings properly. There's also a --host parameter to limit upgrades to a host, but apparently there's a limit of 16 OSDs at a time. My expectation is that this would only rotate and redeploy the OSDs (since they already have been upgraded there's almost no delay with image pull etc.). The only issue with this staggered upgrade approach is that each time you issue the upgrade command (let's say you run it as a batch per host or whatever) it will redeploy mgr and mon daemons as well and rotate their keys again since it's part of the upgrade procedure. So my approach would be to do only run one staggered upgrade (--daemon-types osd --limit 1 ) see if that does the trick for that batch and then start the upgrade again for daemon-types osd but without a limit parameter which would then (hopefully) rorate redeploy the remaining OSDs. It could still end up in a timeout since there are so many OSDs, hard to tell. (I don't even know right now if it's actually a timeout somewhere.) ---snip---
In the mentioned thread Greg confirmed that he observed a keyring rotation for the OSDs. But it will probably take some more time until it's finished, he reported that it took 13 days on the first attempt.
Zitat von Frank Ederveen <ceph-users@ceph.io>:
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both.
It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains.
All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd.
Is there going to be a new release soon, or some scripting which fixes things?
health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ...
Thanks, Frank
________________________________ From: Eugen Block <ceph-users@ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
Zitat von Brent Kennedy <ceph-users@ceph.io>:
Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters.
Regards, -Brent
Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
-----Original Message----- From: Patrick Donnelly <ceph-users@ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad <samdto987@gmail.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Hello,
On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users@ceph.io> wrote:
Hi,
While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available?
Step (9) of [1] says:
"Then copy and import the key to each machine using that client.$ID credential."
[1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u...
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. rer. nat. Burkhard Linke Bioinformatics and Systems Biology Justus-Liebig-University Giessen 35392 Giessen, Germany Phone: (+49) (0)641 9935810
Oh right, good point, I overlooked the lockbox keyrings. I had upgraded test clusters with encrypted OSDs but didn’t get to properly test the key rotation before my vacation. I will definitely look into the script when I return to work. :-) Zitat von Burkhard Linke <ceph-users@ceph.io>:
Hi,
I think the problem are the lockbox keys, not the OSD keys themselves (and the bootstrap and crash keys, but those are easy to solve).
The lockbox keys are used to obtain the dmcrypt secret for accessing encrypted OSD devices. And as far as I have followed the discussion on the mailing list, not much was said about them at all. They seem to be ignored by various parties, since they are not used in the network communication.
A user from IRC has put together a little script to handle these keys. It is available at https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate
Thx to sur5r for creating the script.
I've successfully used to script on our ~900 OSD cluster. Works like charm, but you need to ensure that:
- the admin key is available on all hosts
- OSDs are stopped prior to running the script (check with "ps -u ceph")
Background: the lockbox key is stored in LVM tags; so rotating the key also requires updating the tags. For both data and block device if applicable. Since key rotation only changes the encoding and representation of the key, the key itself is untouched. You do not need to reencrypt your OSDs.
Caveat: no clue whether the script works with cephadm or whether cephadm should handle lockbox key rotation itself
Best regards,
Burkhard
On 9/15/26 08:32, Eugen Block wrote:
There's a thread on Slack (https://ceph-storage.slack.com/archives/C1HFU4JK1/p1789057907366559?thread_ts=1789050849.632739&cid=C1HFU4JK1) where someone reported the same (or very similar) for a large cluster with more than 1000 OSDs. I proposed this:
---snip--- ... I would run the upgrade for a single OSD again:
ceph orch upgrade start --version 19.2.6 --daemon-types osd --limit 1
The only thing is, versions 19.2.6 and 20.2.4 are "broken" in the sense of applying the provided limit in a reasonable way. Most likely, it would redeploy an entire OSD host instead of a single OSD. I did a lot tests with staggered upgrades recently and this is what I observed. Anyway, that would at least reduce the upgrade time window and hopefully rotate keyrings properly. There's also a --host parameter to limit upgrades to a host, but apparently there's a limit of 16 OSDs at a time. My expectation is that this would only rotate and redeploy the OSDs (since they already have been upgraded there's almost no delay with image pull etc.). The only issue with this staggered upgrade approach is that each time you issue the upgrade command (let's say you run it as a batch per host or whatever) it will redeploy mgr and mon daemons as well and rotate their keys again since it's part of the upgrade procedure. So my approach would be to do only run one staggered upgrade (--daemon-types osd --limit 1 ) see if that does the trick for that batch and then start the upgrade again for daemon-types osd but without a limit parameter which would then (hopefully) rorate redeploy the remaining OSDs. It could still end up in a timeout since there are so many OSDs, hard to tell. (I don't even know right now if it's actually a timeout somewhere.) ---snip---
In the mentioned thread Greg confirmed that he observed a keyring rotation for the OSDs. But it will probably take some more time until it's finished, he reported that it took 13 days on the first attempt.
Zitat von Frank Ederveen <ceph-users@ceph.io>:
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both.
It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains.
All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd.
Is there going to be a new release soon, or some scripting which fixes things?
health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ...
Thanks, Frank
________________________________ From: Eugen Block <ceph-users@ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
Zitat von Brent Kennedy <ceph-users@ceph.io>:
Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters.
Regards, -Brent
Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
-----Original Message----- From: Patrick Donnelly <ceph-users@ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad <samdto987@gmail.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Hello,
On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users@ceph.io> wrote:
Hi,
While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available?
Step (9) of [1] says:
"Then copy and import the key to each machine using that client.$ID credential."
[1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u... -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. rer. nat. Burkhard Linke Bioinformatics and Systems Biology Justus-Liebig-University Giessen 35392 Giessen, Germany Phone: (+49) (0)641 9935810
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I really hope there will be a 20.2.5 release soon where cephadm takes care of everything; it seems to .4 release may have been a bit premature. We won't be upgrading any more clusters to 20.2.4 until this is sorted. ________________________________ From: Eugen Block <ceph-users@ceph.io> Sent: 15 September 2026 20:55:56 To: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 Oh right, good point, I overlooked the lockbox keyrings. I had upgraded test clusters with encrypted OSDs but didn’t get to properly test the key rotation before my vacation. I will definitely look into the script when I return to work. :-) Zitat von Burkhard Linke <ceph-users@ceph.io>:
Hi,
I think the problem are the lockbox keys, not the OSD keys themselves (and the bootstrap and crash keys, but those are easy to solve).
The lockbox keys are used to obtain the dmcrypt secret for accessing encrypted OSD devices. And as far as I have followed the discussion on the mailing list, not much was said about them at all. They seem to be ignored by various parties, since they are not used in the network communication.
A user from IRC has put together a little script to handle these keys. It is available at https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate
Thx to sur5r for creating the script.
I've successfully used to script on our ~900 OSD cluster. Works like charm, but you need to ensure that:
- the admin key is available on all hosts
- OSDs are stopped prior to running the script (check with "ps -u ceph")
Background: the lockbox key is stored in LVM tags; so rotating the key also requires updating the tags. For both data and block device if applicable. Since key rotation only changes the encoding and representation of the key, the key itself is untouched. You do not need to reencrypt your OSDs.
Caveat: no clue whether the script works with cephadm or whether cephadm should handle lockbox key rotation itself
Best regards,
Burkhard
On 9/15/26 08:32, Eugen Block wrote:
There's a thread on Slack (https://ceph-storage.slack.com/archives/C1HFU4JK1/p1789057907366559?thread_ts=1789050849.632739&cid=C1HFU4JK1) where someone reported the same (or very similar) for a large cluster with more than 1000 OSDs. I proposed this:
---snip--- ... I would run the upgrade for a single OSD again:
ceph orch upgrade start --version 19.2.6 --daemon-types osd --limit 1
The only thing is, versions 19.2.6 and 20.2.4 are "broken" in the sense of applying the provided limit in a reasonable way. Most likely, it would redeploy an entire OSD host instead of a single OSD. I did a lot tests with staggered upgrades recently and this is what I observed. Anyway, that would at least reduce the upgrade time window and hopefully rotate keyrings properly. There's also a --host parameter to limit upgrades to a host, but apparently there's a limit of 16 OSDs at a time. My expectation is that this would only rotate and redeploy the OSDs (since they already have been upgraded there's almost no delay with image pull etc.). The only issue with this staggered upgrade approach is that each time you issue the upgrade command (let's say you run it as a batch per host or whatever) it will redeploy mgr and mon daemons as well and rotate their keys again since it's part of the upgrade procedure. So my approach would be to do only run one staggered upgrade (--daemon-types osd --limit 1 ) see if that does the trick for that batch and then start the upgrade again for daemon-types osd but without a limit parameter which would then (hopefully) rorate redeploy the remaining OSDs. It could still end up in a timeout since there are so many OSDs, hard to tell. (I don't even know right now if it's actually a timeout somewhere.) ---snip---
In the mentioned thread Greg confirmed that he observed a keyring rotation for the OSDs. But it will probably take some more time until it's finished, he reported that it took 13 days on the first attempt.
Zitat von Frank Ederveen <ceph-users@ceph.io>:
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both.
It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains.
All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd.
Is there going to be a new release soon, or some scripting which fixes things?
health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ...
Thanks, Frank
________________________________ From: Eugen Block <ceph-users@ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
Zitat von Brent Kennedy <ceph-users@ceph.io>:
Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters.
Regards, -Brent
Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
-----Original Message----- From: Patrick Donnelly <ceph-users@ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad <samdto987@gmail.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Hello,
On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users@ceph.io> wrote:
Hi,
While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available?
Step (9) of [1] says:
"Then copy and import the key to each machine using that client.$ID credential."
[1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u... -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. rer. nat. Burkhard Linke Bioinformatics and Systems Biology Justus-Liebig-University Giessen 35392 Giessen, Germany Phone: (+49) (0)641 9935810
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have tested this script in a small cephadm cluster. Encrypted OSDs but no separate db/wal device. The script manages to rotate the client.osd-lockbox.{uuid} keyrings in the cluster and on the osd LVM tags. But you currently need to update the `/var/lib/ceph/{fsid}/osd.x/lockbox.keyring` file manually. On 9/16/26 12:00 PM, Frank Ederveen wrote:
I really hope there will be a 20.2.5 release soon where cephadm takes care of everything; it seems to .4 release may have been a bit premature.
We won't be upgrading any more clusters to 20.2.4 until this is sorted.
________________________________ From: Eugen Block<ceph-users@ceph.io> Sent: 15 September 2026 20:55:56 To:ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Oh right, good point, I overlooked the lockbox keyrings. I had upgraded test clusters with encrypted OSDs but didn’t get to properly test the key rotation before my vacation. I will definitely look into the script when I return to work. :-)
Zitat von Burkhard Linke<ceph-users@ceph.io>:
Hi,
I think the problem are the lockbox keys, not the OSD keys themselves (and the bootstrap and crash keys, but those are easy to solve).
The lockbox keys are used to obtain the dmcrypt secret for accessing encrypted OSD devices. And as far as I have followed the discussion on the mailing list, not much was said about them at all. They seem to be ignored by various parties, since they are not used in the network communication.
A user from IRC has put together a little script to handle these keys. It is available at https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate
Thx to sur5r for creating the script.
I've successfully used to script on our ~900 OSD cluster. Works like charm, but you need to ensure that:
- the admin key is available on all hosts
- OSDs are stopped prior to running the script (check with "ps -u ceph")
Background: the lockbox key is stored in LVM tags; so rotating the key also requires updating the tags. For both data and block device if applicable. Since key rotation only changes the encoding and representation of the key, the key itself is untouched. You do not need to reencrypt your OSDs.
Caveat: no clue whether the script works with cephadm or whether cephadm should handle lockbox key rotation itself
Best regards,
Burkhard
On 9/15/26 08:32, Eugen Block wrote:
There's a thread on Slack (https://ceph-storage.slack.com/archives/C1HFU4JK1/p1789057907366559?thread_ts=1789050849.632739&cid=C1HFU4JK1) where someone reported the same (or very similar) for a large cluster with more than 1000 OSDs. I proposed this:
---snip--- ... I would run the upgrade for a single OSD again:
ceph orch upgrade start --version 19.2.6 --daemon-types osd --limit 1
The only thing is, versions 19.2.6 and 20.2.4 are "broken" in the sense of applying the provided limit in a reasonable way. Most likely, it would redeploy an entire OSD host instead of a single OSD. I did a lot tests with staggered upgrades recently and this is what I observed. Anyway, that would at least reduce the upgrade time window and hopefully rotate keyrings properly. There's also a --host parameter to limit upgrades to a host, but apparently there's a limit of 16 OSDs at a time. My expectation is that this would only rotate and redeploy the OSDs (since they already have been upgraded there's almost no delay with image pull etc.). The only issue with this staggered upgrade approach is that each time you issue the upgrade command (let's say you run it as a batch per host or whatever) it will redeploy mgr and mon daemons as well and rotate their keys again since it's part of the upgrade procedure. So my approach would be to do only run one staggered upgrade (--daemon-types osd --limit 1 ) see if that does the trick for that batch and then start the upgrade again for daemon-types osd but without a limit parameter which would then (hopefully) rorate redeploy the remaining OSDs. It could still end up in a timeout since there are so many OSDs, hard to tell. (I don't even know right now if it's actually a timeout somewhere.) ---snip---
In the mentioned thread Greg confirmed that he observed a keyring rotation for the OSDs. But it will probably take some more time until it's finished, he reported that it took 13 days on the first attempt.
Zitat von Frank Ederveen<ceph-users@ceph.io>:
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both.
It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains.
All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd.
Is there going to be a new release soon, or some scripting which fixes things?
health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ...
Thanks, Frank
________________________________ From: Eugen Block<ceph-users@ceph.io> Sent: 31 August 2026 07:44:50 To:ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
Zitat von Brent Kennedy<ceph-users@ceph.io>:
Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters.
Regards, -Brent
Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
-----Original Message----- From: Patrick Donnelly<ceph-users@ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad<samdto987@gmail.com> Cc:ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Hello,
On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad<ceph-users@ceph.io> wrote:
Hi,
While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? Step (9) of [1] says:
"Then copy and import the key to each machine using that client.$ID credential."
[1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u... -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io -- Dr. rer. nat. Burkhard Linke Bioinformatics and Systems Biology Justus-Liebig-University Giessen 35392 Giessen, Germany Phone: (+49) (0)641 9935810
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hi, For those interested. I've tested this script on OSDs with encryption and db_device set. In my previous message I mention that you need to update 'lockbox.keyring' manually. This is not needed, you only need to move/delete the file and run a reconfig on the osd daemon. My testing is like this: * Copy admin.keyring & ceph-osd-lockbox-key-rotate to OSD node, in /var/lib/ceph/{fsid}/. * Put host in maintenance to stop the osd daemons * cephadm shell --mount /var/lib/ceph/{fsid} (in cephadm shell) * copy admin.keyring to /etc/ceph/ceph.client.admin.keyring * copy config from one osd daemon (cp /mnt/osd.10/config /etc/ceph/ceph.conf) * /mnt/ceph-osd-lockbox-key-rotate (select y/n on the osds you want to rotate) * for the osds with new keys, move the 'lockbox.keyring' file. (on mgr) * ceph orch daemon reconfig osd.X # this will update the lockbox.keyring with the new key AND start the osd daemon On 9/25/26 2:35 PM, Andreas Eriksson wrote:
I have tested this script in a small cephadm cluster.
Encrypted OSDs but no separate db/wal device.
The script manages to rotate the client.osd-lockbox.{uuid} keyrings in the cluster and on the osd LVM tags. But you currently need to update the `/var/lib/ceph/{fsid}/osd.x/lockbox.keyring` file manually.
On 9/16/26 12:00 PM, Frank Ederveen wrote:
I really hope there will be a 20.2.5 release soon where cephadm takes care of everything; it seems to .4 release may have been a bit premature.
We won't be upgrading any more clusters to 20.2.4 until this is sorted.
________________________________ From: Eugen Block<ceph-users@ceph.io> Sent: 15 September 2026 20:55:56 To:ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Oh right, good point, I overlooked the lockbox keyrings. I had upgraded test clusters with encrypted OSDs but didn’t get to properly test the key rotation before my vacation. I will definitely look into the script when I return to work. :-)
Zitat von Burkhard Linke<ceph-users@ceph.io>:
Hi,
I think the problem are the lockbox keys, not the OSD keys themselves (and the bootstrap and crash keys, but those are easy to solve).
The lockbox keys are used to obtain the dmcrypt secret for accessing encrypted OSD devices. And as far as I have followed the discussion on the mailing list, not much was said about them at all. They seem to be ignored by various parties, since they are not used in the network communication.
A user from IRC has put together a little script to handle these keys. It is available at https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate
Thx to sur5r for creating the script.
I've successfully used to script on our ~900 OSD cluster. Works like charm, but you need to ensure that:
- the admin key is available on all hosts
- OSDs are stopped prior to running the script (check with "ps -u ceph")
Background: the lockbox key is stored in LVM tags; so rotating the key also requires updating the tags. For both data and block device if applicable. Since key rotation only changes the encoding and representation of the key, the key itself is untouched. You do not need to reencrypt your OSDs.
Caveat: no clue whether the script works with cephadm or whether cephadm should handle lockbox key rotation itself
Best regards,
Burkhard
On 9/15/26 08:32, Eugen Block wrote:
There's a thread on Slack (https://ceph-storage.slack.com/archives/C1HFU4JK1/p1789057907366559?thread_ts=1789050849.632739&cid=C1HFU4JK1) where someone reported the same (or very similar) for a large cluster with more than 1000 OSDs. I proposed this:
---snip--- ... I would run the upgrade for a single OSD again:
ceph orch upgrade start --version 19.2.6 --daemon-types osd --limit 1
The only thing is, versions 19.2.6 and 20.2.4 are "broken" in the sense of applying the provided limit in a reasonable way. Most likely, it would redeploy an entire OSD host instead of a single OSD. I did a lot tests with staggered upgrades recently and this is what I observed. Anyway, that would at least reduce the upgrade time window and hopefully rotate keyrings properly. There's also a --host parameter to limit upgrades to a host, but apparently there's a limit of 16 OSDs at a time. My expectation is that this would only rotate and redeploy the OSDs (since they already have been upgraded there's almost no delay with image pull etc.). The only issue with this staggered upgrade approach is that each time you issue the upgrade command (let's say you run it as a batch per host or whatever) it will redeploy mgr and mon daemons as well and rotate their keys again since it's part of the upgrade procedure. So my approach would be to do only run one staggered upgrade (--daemon-types osd --limit 1 ) see if that does the trick for that batch and then start the upgrade again for daemon-types osd but without a limit parameter which would then (hopefully) rorate redeploy the remaining OSDs. It could still end up in a timeout since there are so many OSDs, hard to tell. (I don't even know right now if it's actually a timeout somewhere.) ---snip---
In the mentioned thread Greg confirmed that he observed a keyring rotation for the OSDs. But it will probably take some more time until it's finished, he reported that it took 13 days on the first attempt.
Zitat von Frank Ederveen<ceph-users@ceph.io>:
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both.
It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains.
All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd.
Is there going to be a new release soon, or some scripting which fixes things?
health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ...
Thanks, Frank
________________________________ From: Eugen Block<ceph-users@ceph.io> Sent: 31 August 2026 07:44:50 To:ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys.
Zitat von Brent Kennedy<ceph-users@ceph.io>:
Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters.
Regards, -Brent
Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
-----Original Message----- From: Patrick Donnelly<ceph-users@ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad<samdto987@gmail.com> Cc:ceph-users@ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
Hello,
On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad<ceph-users@ceph.io> wrote: > Hi, > > While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these > versions introduce the secure key type, I have a clarification > regarding client key rotation. > For the client.admin key, we need to rotate the key and then > import/update the keyring. > For the other client keys, do we only need to rotate the key and > verify that the key has changed? Is there no need to import/update > the keyring for the other clients, since there is no local keyring > available? Step (9) of [1] says:
"Then copy and import the key to each machine using that client.$ID credential."
[1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-u... -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io -- Dr. rer. nat. Burkhard Linke Bioinformatics and Systems Biology Justus-Liebig-University Giessen 35392 Giessen, Germany Phone: (+49) (0)641 9935810
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Andreas Eriksson
-
Burkhard Linke
-
Eugen Block
-
Frank Ederveen