Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
Yeah, I manually replaced the old key in each hosts authorized_keys file (just before I saw your email!). Then it went away. This is a recurring sort of theme with cephadm though. It is supposed to abstract away many of the implementation details. But then the documentation doesn't cover it, so you have to guess what it should have done, and then fix it manually... Thanks, Chris On 26/08/2026 13:43, Eugen Block wrote:
Hi,
did you add the new pub key to all the host's authorized_keys?
clear-key and then generate-key is the correct way.
Zitat von Chris Palmer <ceph-users@ceph.io>:
I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and now it is broken:
The note below points towards the SSH Configuration section. That doesn't have anything specific about rotation, so I had to make some guesses.
It says that a new SSH key can be generated with "ceph cephadm generate-key". It doesn't give any error, but using "ceph cephadm get-pub-key" suggests it hasn't actually done anything (I still get the original ssh-rsa key). The doc probably needs clarifying.
So I tried "ceph cephadm clear-key" which worked, and then "ceph cephadm generate-key" again. This time I can see a new ssh-ed25519 key. I then failed all MGRs.
Now however I have this for every host:
root@c1:~# ceph cephadm check-host c2 check-host failed: Failed to connect to c2 at address (192.168.nnn.162): Failed to connect to c2 (192.168.nnn.162). Permission denied Log: Opening SSH connection to 192.168.nnn.162, port 22 [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 [conn=9] Local address: 192.168.nnn.161, port 35514 [conn=9] Peer address: 192.168.nnn.162, port 22 [conn=9] Beginning auth for user root [conn=9] Auth failed for user root [conn=9] Connection failure: Permission denied [conn=9] Aborting connection
Have I done this incorrectly, or is the documentation wrong? What is the correct way to rectify this?
Thanks, Chris
On 19/08/2026 17:07, Patrick Donnelly wrote:
Today, August 19th, 2026, the Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components.
This is the sixth backport release in the Squid series.
This is the fourth backport release in the Tentacle series.
Additional details will be posted on the Ceph website/blog: https://ceph.io/en/news/blog/
We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible.
## Critical Upgrade Steps
The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at:
https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.ht...
These steps are for package-based deployments of Ceph. In other sections, the document also includes new information about key rotation procedures, monitor authentication controls, and emergency recovery procedures.
Deployments using cephadm will automate the process except for client keys. You may notice that Cephadm spends more time than normal on the upgrade after updating all daemon images. This is due to a new process rotating the OSD and MDS keys. Please be patient. There are plans to improve this in the future.
Deployments using Rook will also automate rotation of some client keys with some exclusions. Look to the Rook project’s announcement for more details.
Client and kernel upgrades are recommended to support aes256k. Note that upstream Linux kernel client support began in kernel 7.0 and has been backported to CentOS Stream 9 and 10. Check with your distribution vendor for backported key support before rotating client keys used by the kernel.
As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go. These are documented beginning here:
https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#au...
The CephX documentation has been brought up to date and vetted for accuracy. Those changes are live here:
https://docs.ceph.com/en/latest/architecture/index.html#high-availability-au...
Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 requests with `host` and `x-amz-` headers not included in the signed subset. Unfortunately, the REST client used in multisite was generating such improperly signed requests.
If you are running multisite, you must set the ``rgw_sigv4_insecure`` option to true before you begin to upgrade. After all clusters are upgraded, set the option to ``false`` again.
Regarding the fix for CVE-2026-50152, formal guidance on rotating all secrets stored in the Monitor config-key store will be forthcoming. In the meantime, operators of cephadm-managed clusters are strongly encouraged to rotate the cephadm SSH key, as an established process exists:
See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration
Rotating other secrets is advised but the process is not yet established. Operators should assess their cluster's potential exposure to determine if immediate Monitor config-key secret rotation is required for their environment.
## CVE Fixes
* CVE-2025-30156 is an authentication bypass in CephX caused by misuse of AES-CBC.
See: https://docs.ceph.com/en/latest/security/CVE-2025-30156
* CVE-2026-39944 shares the unauthenticated-encryption root cause of CVE-2025-30156, but applies it to RGW's STS session tokens resulting in improper verification of a cryptographic signature.
See: https://docs.ceph.com/en/latest/security/CVE-2026-39944
* CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor subscription handler.
See: https://docs.ceph.com/en/latest/security/CVE-2026-50152
* CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 cryptographic signatures in RGW's SigV4 verifier.
See: https://docs.ceph.com/en/latest/security/CVE-2026-54330
## Other Media
A Cephalocon 2025 talk covers some of the details surrounding the new CephX key upgrade process.
https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-p...
## Getting Ceph ------------ * Source at https://github.com/ceph/ceph * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * For upgrade with cephadm, see: https://docs.ceph.com/en/latest/cephadm/upgrade/ * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I agree that there's always some room for improvement regarding the documentation. But at the same time the docs can't cover every single aspect of operating a cluster, it's just impossible. In this specific case I would argue that an operator who has used cephadm to bootstrap and expand a cluster should be familiar with some of the details, like the ssh key deployment. For example, the page "host management" --> "adding hosts" contains the info about adding the pub-key to the authorized_keys file: https://docs.ceph.com/en/latest/cephadm/host-management/#adding-hosts But if you feel the docs are incomplete, feel free to create a PR or get in touch with Emmanuel Ameh. :-) Regards, Eugen Zitat von Chris Palmer <chris.palmer@idnet.com>: https://docs.ceph.com/en/latest/cephadm/host-management/#enabling-sudo-harde...
Yeah, I manually replaced the old key in each hosts authorized_keys file (just before I saw your email!). Then it went away.
This is a recurring sort of theme with cephadm though. It is supposed to abstract away many of the implementation details. But then the documentation doesn't cover it, so you have to guess what it should have done, and then fix it manually...
Thanks, Chris
On 26/08/2026 13:43, Eugen Block wrote:
Hi,
did you add the new pub key to all the host's authorized_keys?
clear-key and then generate-key is the correct way.
Zitat von Chris Palmer <ceph-users@ceph.io>:
I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and now it is broken:
The note below points towards the SSH Configuration section. That doesn't have anything specific about rotation, so I had to make some guesses.
It says that a new SSH key can be generated with "ceph cephadm generate-key". It doesn't give any error, but using "ceph cephadm get-pub-key" suggests it hasn't actually done anything (I still get the original ssh-rsa key). The doc probably needs clarifying.
So I tried "ceph cephadm clear-key" which worked, and then "ceph cephadm generate-key" again. This time I can see a new ssh-ed25519 key. I then failed all MGRs.
Now however I have this for every host:
root@c1:~# ceph cephadm check-host c2 check-host failed: Failed to connect to c2 at address (192.168.nnn.162): Failed to connect to c2 (192.168.nnn.162). Permission denied Log: Opening SSH connection to 192.168.nnn.162, port 22 [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 [conn=9] Local address: 192.168.nnn.161, port 35514 [conn=9] Peer address: 192.168.nnn.162, port 22 [conn=9] Beginning auth for user root [conn=9] Auth failed for user root [conn=9] Connection failure: Permission denied [conn=9] Aborting connection
Have I done this incorrectly, or is the documentation wrong? What is the correct way to rectify this?
Thanks, Chris
On 19/08/2026 17:07, Patrick Donnelly wrote:
Today, August 19th, 2026, the Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components.
This is the sixth backport release in the Squid series.
This is the fourth backport release in the Tentacle series.
Additional details will be posted on the Ceph website/blog: https://ceph.io/en/news/blog/
We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible.
## Critical Upgrade Steps
The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at:
https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.ht... These steps are for package-based deployments of Ceph. In other sections, the document also includes new information about key rotation procedures, monitor authentication controls, and emergency recovery procedures.
Deployments using cephadm will automate the process except for client keys. You may notice that Cephadm spends more time than normal on the upgrade after updating all daemon images. This is due to a new process rotating the OSD and MDS keys. Please be patient. There are plans to improve this in the future.
Deployments using Rook will also automate rotation of some client keys with some exclusions. Look to the Rook project’s announcement for more details.
Client and kernel upgrades are recommended to support aes256k. Note that upstream Linux kernel client support began in kernel 7.0 and has been backported to CentOS Stream 9 and 10. Check with your distribution vendor for backported key support before rotating client keys used by the kernel.
As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go. These are documented beginning here:
https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#au... The CephX documentation has been brought up to date and vetted for accuracy. Those changes are live here:
https://docs.ceph.com/en/latest/architecture/index.html#high-availability-au... Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 requests with `host` and `x-amz-` headers not included in the signed subset. Unfortunately, the REST client used in multisite was generating such improperly signed requests.
If you are running multisite, you must set the ``rgw_sigv4_insecure`` option to true before you begin to upgrade. After all clusters are upgraded, set the option to ``false`` again.
Regarding the fix for CVE-2026-50152, formal guidance on rotating all secrets stored in the Monitor config-key store will be forthcoming. In the meantime, operators of cephadm-managed clusters are strongly encouraged to rotate the cephadm SSH key, as an established process exists:
See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration
Rotating other secrets is advised but the process is not yet established. Operators should assess their cluster's potential exposure to determine if immediate Monitor config-key secret rotation is required for their environment.
## CVE Fixes
* CVE-2025-30156 is an authentication bypass in CephX caused by misuse of AES-CBC.
See: https://docs.ceph.com/en/latest/security/CVE-2025-30156
* CVE-2026-39944 shares the unauthenticated-encryption root cause of CVE-2025-30156, but applies it to RGW's STS session tokens resulting in improper verification of a cryptographic signature.
See: https://docs.ceph.com/en/latest/security/CVE-2026-39944
* CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor subscription handler.
See: https://docs.ceph.com/en/latest/security/CVE-2026-50152
* CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 cryptographic signatures in RGW's SigV4 verifier.
See: https://docs.ceph.com/en/latest/security/CVE-2026-54330
## Other Media
A Cephalocon 2025 talk covers some of the details surrounding the new CephX key upgrade process.
https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-p... ## Getting Ceph ------------ * Source at https://github.com/ceph/ceph * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * For upgrade with cephadm, see: https://docs.ceph.com/en/latest/cephadm/upgrade/ * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, the docs were updated to reflect the requirement of clearing the key first before generating it: https://github.com/ceph/ceph/pull/71933 Zitat von Chris Palmer <chris.palmer@idnet.com>:
Yeah, I manually replaced the old key in each hosts authorized_keys file (just before I saw your email!). Then it went away.
This is a recurring sort of theme with cephadm though. It is supposed to abstract away many of the implementation details. But then the documentation doesn't cover it, so you have to guess what it should have done, and then fix it manually...
Thanks, Chris
On 26/08/2026 13:43, Eugen Block wrote:
Hi,
did you add the new pub key to all the host's authorized_keys?
clear-key and then generate-key is the correct way.
Zitat von Chris Palmer <ceph-users@ceph.io>:
I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and now it is broken:
The note below points towards the SSH Configuration section. That doesn't have anything specific about rotation, so I had to make some guesses.
It says that a new SSH key can be generated with "ceph cephadm generate-key". It doesn't give any error, but using "ceph cephadm get-pub-key" suggests it hasn't actually done anything (I still get the original ssh-rsa key). The doc probably needs clarifying.
So I tried "ceph cephadm clear-key" which worked, and then "ceph cephadm generate-key" again. This time I can see a new ssh-ed25519 key. I then failed all MGRs.
Now however I have this for every host:
root@c1:~# ceph cephadm check-host c2 check-host failed: Failed to connect to c2 at address (192.168.nnn.162): Failed to connect to c2 (192.168.nnn.162). Permission denied Log: Opening SSH connection to 192.168.nnn.162, port 22 [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 [conn=9] Local address: 192.168.nnn.161, port 35514 [conn=9] Peer address: 192.168.nnn.162, port 22 [conn=9] Beginning auth for user root [conn=9] Auth failed for user root [conn=9] Connection failure: Permission denied [conn=9] Aborting connection
Have I done this incorrectly, or is the documentation wrong? What is the correct way to rectify this?
Thanks, Chris
On 19/08/2026 17:07, Patrick Donnelly wrote:
Today, August 19th, 2026, the Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components.
This is the sixth backport release in the Squid series.
This is the fourth backport release in the Tentacle series.
Additional details will be posted on the Ceph website/blog: https://ceph.io/en/news/blog/
We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible.
## Critical Upgrade Steps
The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at:
https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.ht... These steps are for package-based deployments of Ceph. In other sections, the document also includes new information about key rotation procedures, monitor authentication controls, and emergency recovery procedures.
Deployments using cephadm will automate the process except for client keys. You may notice that Cephadm spends more time than normal on the upgrade after updating all daemon images. This is due to a new process rotating the OSD and MDS keys. Please be patient. There are plans to improve this in the future.
Deployments using Rook will also automate rotation of some client keys with some exclusions. Look to the Rook project’s announcement for more details.
Client and kernel upgrades are recommended to support aes256k. Note that upstream Linux kernel client support began in kernel 7.0 and has been backported to CentOS Stream 9 and 10. Check with your distribution vendor for backported key support before rotating client keys used by the kernel.
As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go. These are documented beginning here:
https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#au... The CephX documentation has been brought up to date and vetted for accuracy. Those changes are live here:
https://docs.ceph.com/en/latest/architecture/index.html#high-availability-au... Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 requests with `host` and `x-amz-` headers not included in the signed subset. Unfortunately, the REST client used in multisite was generating such improperly signed requests.
If you are running multisite, you must set the ``rgw_sigv4_insecure`` option to true before you begin to upgrade. After all clusters are upgraded, set the option to ``false`` again.
Regarding the fix for CVE-2026-50152, formal guidance on rotating all secrets stored in the Monitor config-key store will be forthcoming. In the meantime, operators of cephadm-managed clusters are strongly encouraged to rotate the cephadm SSH key, as an established process exists:
See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration
Rotating other secrets is advised but the process is not yet established. Operators should assess their cluster's potential exposure to determine if immediate Monitor config-key secret rotation is required for their environment.
## CVE Fixes
* CVE-2025-30156 is an authentication bypass in CephX caused by misuse of AES-CBC.
See: https://docs.ceph.com/en/latest/security/CVE-2025-30156
* CVE-2026-39944 shares the unauthenticated-encryption root cause of CVE-2025-30156, but applies it to RGW's STS session tokens resulting in improper verification of a cryptographic signature.
See: https://docs.ceph.com/en/latest/security/CVE-2026-39944
* CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor subscription handler.
See: https://docs.ceph.com/en/latest/security/CVE-2026-50152
* CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 cryptographic signatures in RGW's SigV4 verifier.
See: https://docs.ceph.com/en/latest/security/CVE-2026-54330
## Other Media
A Cephalocon 2025 talk covers some of the details surrounding the new CephX key upgrade process.
https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-p... ## Getting Ceph ------------ * Source at https://github.com/ceph/ceph * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * For upgrade with cephadm, see: https://docs.ceph.com/en/latest/cephadm/upgrade/ * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Chris Palmer
-
Eugen Block