- ceph-users - lists.ceph.io

Handling key rotation for proxmox , openstack and ceph csi clients
by Burkhard Linke 21 Aug '26

21 Aug '26

Hi, since the latest ceph releases introduce a new key type to fix a security vulnerability, the release notes recommend to rotate the client keys and switch to the new key algorithm. This might not be possible for client that use a shared key for multiple hosts, e.g. proxmox, openstack and ceph-csi in k8s clusters. The recommended way to handle this is using host specific keys, but this it currently not an option for the mentioned clients. I would like to collect the various methods for handling the client side of the key rotation in the thread as a reference for other users. We did not upgrade yet, so the following steps are not tested yet. But we did something similar in the past in in different scenario, so it should work [tm]. So: use at own risk - Proxmox and Openstack Both usually use qemu, and qemu has support for performing a data live migration. In the case of proxmox I would propose to define a new rbd pool, using the same setup (crush etc.) as the original data pool. Then define a new storage backend in proxmox using the new pool and a ceph client with the new key algorithm. After testing the pool you should be able to migrate existing disks to the new pool. The migration option is available in the hardware overview of each virtual machine; select the disk, and the "Disk action" menu will allow to move the storage to a different backend. As mentioned above this is a live migration, so the running instance will not be affected (except for slightly higher I/O latencies). Since proxmox offers an API and CLI utilities, you should also be able to automate this process if you are running a large number of instances. Openstack supports live data migration for cinder volumes. You need to define a second volume type, again with a new pool and a new ceph client. The pool has to be configured on all controllers , which should be performed by your deployment tool. The "openstack volume set" command can be used to trigger the migration; I'm not sure whether it supports an option for waiting until the migration is done. In the worst case you'll have to poll the volume state. The might be some restriction, e.g. snapshot presence, which might prevent a volume from being migrated. Automation should also be possible as script or command line. We did a similar migration in the past, migrating from one ceph cluster to another; we had to update some stuff in the cinder database afterwards since they were still referring to the old cluster. You might want to test this first... Advantage: - live migration, no downtime necessary for most instances - automation possible Disadvantage: - physical data migration to new pool aka data is being copied - ceph-csi RBD and cephfs via csi in k8s clusters do not use qemu or other layers of indirection; rbds are mapped directly to the host and mounted, cephfs volume are mounted. CSI supports cloning and volume migration, so similar approaches might be possible. I haven't done this in the past, so I cannot comment on this or recommend any approach. Feedback from experiences users is highly welcome. Best regards, Burkhard Linke

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Nico Schottelius 21 Aug '26

21 Aug '26

Good morning list, some of you already mentioned the issue in relation to proxmox, but virtually all VPS/VM hostings are affected that are running qemu/kvm. The situation is even worse, if I see it correctly: a) All VPS/VMs need to be restarted To be able to rotate keys, aside from the almost impossible flow to actually doing the rotation (more to that below), this will affect every consumer of VMs. The usual setup is that all hypervisors are using the same key. b) Key support in qemu/kvm From my understanding, qemu/kvm + librbd does at the moment not have support for the new key type. Thus there will be major waiting periods until the packages have been updated and eventually make it back into the distros. If they are being done at all, depending on your setup there might be old qemu versions which cannot easily be upgraded. c) Key rotation Assuming that the support actually has landed in qemu/kvm (and that can still take months), one will need to rotate potentially update/edit generade libvirtd xml that contains the reference to the ceph key or the key itself. I'm not saying anything cannot be done, but the way how this vulnerabilty is solved will cause major headache around the industry. BR, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Dan van der Ster 21 Aug '26

21 Aug '26

Heads up: large clusters might crash during the first steps of the upgrade to v19.2.6. See https://tracker.ceph.com/issues/77610#note-9 On a cluster with ~1400 OSDs, upgrading from 18.2.8 to 19.2.6, the squid mgr's crash when reading the osdmap .mgr pool read_balance_score = inf. Workaround is to increase the number of pgs on the .mgr pool to 8 or 16. ( Patrick is this why you merged this a few hours ago? https://github.com/ceph/ceph/pull/60063 ) Cheers, Dan [1] 2026-08-21T01:37:37.329+0000 7a754d6b0640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'devicehealth' while running on mgr.xxxx.zgsszn: Expecting value: line 1 column 2403 (char 2402) 2026-08-21T01:37:37.329+0000 7a754d6b0640 -1 devicehealth.serve: 2026-08-21T01:37:37.329+0000 7a754d6b0640 -1 Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 408, in serve self._do_serve() File "/usr/share/ceph/mgr/mgr_module.py", line 551, in check return func(self, *args, **kwargs) File "/usr/share/ceph/mgr/devicehealth/module.py", line 364, in _do_serve if self.db_ready() and self.enable_monitoring: File "/usr/share/ceph/mgr/mgr_module.py", line 1314, in db_ready return self.db is not None File "/usr/share/ceph/mgr/mgr_module.py", line 1326, in db self._db = self.open_db() File "/usr/share/ceph/mgr/mgr_module.py", line 1293, in open_db if not self.pool_exists(self.MGR_POOL_NAME): File "/usr/share/ceph/mgr/mgr_module.py", line 1120, in pool_exists pools = [p['pool_name'] for p in self.get('osd_map')['pools']] File "/usr/share/ceph/mgr/mgr_module.py", line 1444, in get obj = json.loads(obj) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 2403 (char 2402) On Wed, Aug 19, 2026 at 9:05 AM Patrick Donnelly <ceph-users(a)ceph.io> wrote: > > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Burkhard Linke 21 Aug '26

21 Aug '26

Hi, On 8/20/26 21:35, Patrick Donnelly wrote: >> A challenge to that approach is that Proxmox PVE defines external RBD >> pools at the datacenter level using a single user/key pair. This then >> becomes the storage pool on each individual node in the proxmox >> cluster. I believe ceph-csi K8 storage class also defines this once >> cluster wide. > I understand but the only way to change things is to spread the word. > If the key is that challenging to rotate, then I would suggest leaving > it as aes and push for changes in the product or deployment > technology. Rook developers are already aware this should be reworked. So you expect every user running Proxmox PVE, K8s with ceph csi or Openstack (which is usually more than just a handful of hosts) to figure out how to adopt their deployment? Which is not possible at this time for the mentioned cases. Most users will probably welcome a more detailed documentation on how to handle key rotation _and_ especially how to manage the client side in this scenario. As was already mentioned before, most of us cannot simply shutdown all virtual machines in Proxmox or Openstack for a key rotation.... Best regards, Burkhard Linke

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Jan Kasprzak 21 Aug '26

21 Aug '26

Hi William and all, I ran into the same, thanks for the suggested fix. FWIW, the readlink calls in your code are supefluous, one can simply use the symlink itself (after all, this is the purpose of symlinks :-): OSD_ID=123 KEY=`ceph auth get-key osd.$OSD_ID` ssh root@... "ceph-bluestore-tool set-label-key --dev /var/lib/ceph/osd/ceph-$OSD_ID/block -k osd_key -v $KEY" -Yenya William Xiong wrote: > Hi Patrick, > > Regarding the fix for CVE-2025-30156 > > We've discovered that the bluestore label still contains the old key after following instructions at > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > Old key shows up as per: > ceph-bluestore-tool show-label --dev "$(readlink -f /var/lib/ceph/osd/ceph-X/block)" | grep osd_key > > > If we reboot the osd servers, the new key is overwritten with the old key in /var/lib/ceph/osd/ceph-${OSD_ID}/keyring causing osd auth failure > osd service doesnt start > Can simulate this without reboot via `umount /var/lib/ceph/osd/ceph-${OSD_ID}/` and reactivate via `ceph-volume lvm activate --all` > We are running a non cephadm cluster. > > > Need to update the label as well after rotating the key: > ceph-bluestore-tool set-label-key --dev "$(readlink -f /var/lib/ceph/osd/ceph-${OSD_ID}/block)" -k osd_key -v $NEW_KEY > > Kind Regards, > > > William Xiong | Infrastructure Engineer - Storage > > Research Computing Services > > The University of Melbourne, Victoria 3010 Australia > > E: william.xiong(a)unimelb.edu.au<mailto:william.xiong@unimelb.edu.au> > > ________________________________ > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Thursday, 20 August 2026 2:08 AM > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released > > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Stephan Hohn 21 Aug '26

21 Aug '26

I ran into a problem upgrading encrypted OSDs with cephadm on a hardened node where /tmp is mounted with noexec. The rotate helper script will fail on such setups. podman run --privileged --entrypoint /tmp/cryptsetup_action.sh \ -v /tmp/cephadm-osd-14-rotate-helper.sh:/tmp/cryptsetup_action.sh <image> → ERROR (catatonit:2): failed to exec pid1: Permission denied → ERROR: Got error rotating osd keyring while using cryptsetup tool → cephadm exited with an error code: 1 A temporary remount of /tmp which allowed exec solved it for me. BR Stephan Hohn IT Technical Leader stephan(a)gridscale.io gridscale GmbH Oskar-Jäger-Str. 173 50825 Köln www.gridscale.io HRB 97235 Amtsgericht Köln Vertreten durch: Marc Dittmann, Yaniv Fdida Am Mi., 19. Aug. 2026 um 18:08 Uhr schrieb Patrick Donnelly < ceph-users(a)ceph.io>: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: > https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 21 Aug '26

21 Aug '26

Hi Jan, On Thu, Aug 20, 2026 at 10:54 AM Jan Kasprzak <ceph-users(a)ceph.io> wrote: > > Hi all, > > Patrick Donnelly wrote: > > Today, August 19th, 2026, the Ceph project has released Tentacle > > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > > components. > > I have just upgraded one of my cluster from 20.2.3 to 20.2.4. > The documentation seems to be clear (thanks!) and worked for me > with the exception of step 4: > > > 4. Confirm the AUTH_INSECURE_SERVICE_KEY_TYPE is cleared. > > > > ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_SERVICE_KEY_TYPE") | not' > > > > output gives false. > > > > If it outputs true, there is another daemon that needs to be upgraded. > > Check the output of ceph health detail. > > I might have done something wrong, but if I remember it correctly, > at this point I still got 4 AUTH_INSECURE_SERVICE_KEY_TYPE keys, > and had to do step 5 "ceph mon set auth_service_cipher aes256k" first > and then step 6 "ceph auth wipe-rotating-service-keys". Only then > the AUTH_INSECURE_SERVICE_KEY_TYPE disappeared from ceph health detail > (together with AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE). AUTH_INSECURE_SERVICE_KEY_TYPE indicates that the service, e.g. osd.0, is using an aes key to authenticate with the Monitors. That can only be addressed by rotating that service's key. The other health error is AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE which is addressed by the auth_service_cipher monitor setting (required) and wiping the rotating key database (optional). > One question, though: is it possible to have a client with two keys, > aes and aes256? One of my cluster is mainly used for RBD with client.libvirt > user, and I have hundreds of running VMs, which I cannot restart at once. > So I would like to _add_ an aes256 key to the client.libvirt user, > and incrementally restart the qemu processes which use this secret. No, you cannot have two keys. I think a lesson (for everyone) here is that each node should have its own client key. It became prevalent to use the same key everywhere for simplicity but it makes deployments and updates more challenging. > Or what is the recommended way of gradually replacing a client key? Create new keys for each node. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Maximilian Stinsky-Damke 21 Aug '26

21 Aug '26

Hi Adam, I tried following your guidance regarding rotating an OSD lockbox key via ceph auth rotate and then redeploying the OSD via ceph orch redeploy in a test cluster. It seems that cephadm does not update the lockbox keyring during a redeploy, which results in the OSDs failing to start with the new keyring because they cannot unlock their dm-crypt volumes. To restore my test cluster, I had to manually update the lockbox.keyring for each OSD with the new key. Is this a bug in cephadm, or am I missing something here? Best regards, Max ________________________________ From: Adam King <ceph-users(a)ceph.io> Sent: 20 August 2026 15:46 To: Pedro Macedo <pmacedo(a)pmacedo.com> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released Warning: This email is from an unusual correspondent. Warning: Make sure this is someone you trust. That command got removed for the time being as it used an old way of doing the key rotation that wasn't built to work with the changes for the cve and seemed to be broken for a number of daemon types. Since the name would make people want to use it it was decided to just temporarily disables it so we can fix it and reintroduce it in a new version as an easier way to rotate the keys. For any keys tied to a cephadm deployed daemon that aren't automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using nfs exports managed through the nfs module the export's keyrings also need to be rotated. `ceph auth rotate <export-keyring-entity> --key-type aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. Between rotating the keyrings for the nfs cluster/daemons and rotating the keyring for the individual export(s) those export(s) are unavailable iirc. On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > > On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: > > Hi, > > > > I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The > following cephadm managed client keys were not rotated automatically: > > > > client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) > > client.ceph-exporter. > > client.crash. > > client.osd-lockbox.. > > client.rgw. > > > > The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* > this command does not exist. > > Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing > there, but present on 20.2.3. > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Fsquid%2F… > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph%2Fbl… > > > > What is the correct way to rotate these client keys via cephadm as part > of the cve fix? > > > > Thanks for any guidance. > > Best Regards > > Max > > > > ________________________________ > > From: Patrick Donnelly <ceph-users(a)ceph.io> > > Sent: 19 August 2026 18:08 > > To: ceph-users <ceph-users(a)ceph.io> > > Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 > released > > > > Warning: This email is from an unusual correspondent. > > Warning: Make sure this is someone you trust. > > > > Today, August 19th, 2026, the Ceph project has released Tentacle > > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > > components. > > > > This is the sixth backport release in the Squid series. > > > > This is the fourth backport release in the Tentacle series. > > > > Additional details will be posted on the Ceph website/blog: > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… > > > > We strongly recommend that all Ceph operators upgrade to one of these > > releases as soon as possible. > > > > > > ## Critical Upgrade Steps > > > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > > This is the first time Ceph has introduced a new key type for CephX > > credentials. Therefore, a new procedure exists for upgrading and > > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > > cluster, the operator should familiarize themselves with the procedure > > documented at: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > These steps are for package-based deployments of Ceph. In other > > sections, the document also includes new information about key > > rotation procedures, monitor authentication controls, and emergency > > recovery procedures. > > > > Deployments using cephadm will automate the process except for client > > keys. You may notice that Cephadm spends more time than normal on the > > upgrade after updating all daemon images. This is due to a new process > > rotating the OSD and MDS keys. Please be patient. There are plans to > > improve this in the future. > > > > Deployments using Rook will also automate rotation of some client keys > > with some exclusions. Look to the Rook project’s announcement for more > > details. > > > > Client and kernel upgrades are recommended to support aes256k. Note > > that upstream Linux kernel client support began in kernel 7.0 and has > > been backported to CentOS Stream 9 and 10. Check with your > > distribution vendor for backported key support before rotating client > > keys used by the kernel. > > > > As part of upgrading a Ceph cluster, six new health warnings and > > errors will be generated. This is normal. As part of the process to > > upgrade/rotate entity keys, you will address the warnings and errors > > as you go. These are documented beginning here: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > The CephX documentation has been brought up to date and vetted for > > accuracy. Those changes are live here: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > > requests with `host` and `x-amz-` headers not included in the signed > > subset. Unfortunately, the REST client used in multisite was > > generating such improperly signed requests. > > > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > > option to true before you begin to upgrade. After all clusters are > > upgraded, set the option to ``false`` again. > > > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > > secrets stored in the Monitor config-key store will be forthcoming. In > > the meantime, operators of cephadm-managed clusters are strongly > > encouraged to rotate the cephadm SSH key, as an established process > > exists: > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > Rotating other secrets is advised but the process is not yet > > established. Operators should assess their cluster's potential > > exposure to determine if immediate Monitor config-key secret rotation > > is required for their environment. > > > > > > ## CVE Fixes > > > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > > of AES-CBC. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > > in improper verification of a cryptographic signature. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > > subscription handler. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > > cryptographic signatures in RGW's SigV4 verifier. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > ## Other Media > > > > A Cephalocon 2025 talk covers some of the details surrounding the new > > CephX key upgrade process. > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… > > > > ## Getting Ceph > > ------------ > > * Source at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… > > * v19.2.6 Tarball at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > > * v20.2.4 Tarball at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > > * Containers at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… > > * For packages, see > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * For upgrade with cephadm, see: > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > > > -- > > Patrick Donnelly, Ph.D. > > He / Him / His > > Red Hat Partner Engineer > > IBM, Inc. > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > This message has been checked by Libraesva ESG and is found to be clean. > > Follow this link to report as spam/bad: > > https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad > > Follow this link to blocklist sender: > > https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- This message has been checked by Libraesva ESG and is found to be clean. Follow this link to report as spam/bad: https://mx10.wiit.cloud/action/4hQl8t5ktzzTkbn/report-as-bad Follow this link to blocklist sender: https://mx10.wiit.cloud/action/4hQl8t5ktzzTkbn/blocklist

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 20 Aug '26

20 Aug '26

On Thu, Aug 20, 2026 at 3:51 PM Salatiel Filho <ceph-users(a)ceph.io> wrote: > entity client.bootstrap-mds using insecure key type: aes > entity client.bootstrap-mgr using insecure key type: aes > entity client.bootstrap-osd using insecure key type: aes > entity client.bootstrap-rbd using insecure key type: aes > entity client.bootstrap-rbd-mirror using insecure key type: aes > entity client.bootstrap-rgw using insecure key type: aes Bootstrap keys should be safe to delete. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Question on so-called tenant level roles in tentacle
by Manuel Rios - EDH 20 Aug '26

20 Aug '26

I don’t know how this patch become online without an auto-rotation... -----Mensaje original----- De: Casey Bodley <ceph-users(a)ceph.io> Enviado el: Thursday, August 20, 2026 7:15 PM Para: Christopher Durham <caduceus42(a)aol.com> CC: ceph-users(a)ceph.io Asunto: [ceph-users] Re: Question on so-called tenant level roles in tentacle On Wed, Aug 19, 2026 at 5:20 PM Christopher Durham <ceph-users(a)ceph.io> wrote: > > > The release notes for ceph tentacle, at: > https://docs.ceph.com/en/latest/releases/tentacle/ > state the following > "RGW: The User Account feature introduced in Squid provides first-class support for IAM APIs and policy. Our preliminary STS support was based on tenants, and exposed some IAM APIs to admins only. This tenant-level IAM functionality is now deprecated in favor of accounts. While we’ll continue to support the tenant feature itself for namespace isolation, the following features will be removed no sooner than the V release: Tenant-level IAM APIs including CreateRole, PutRolePolicy and PutUserPolicy, Use of tenant names instead of accounts in IAM policy documents, Interpretation of IAM policy without cross-account policy evaluation, S3 API support for cross-tenant names such as Bucket=’tenant:bucketname’ STS Lite and sts:GetSessionToken." > > Does the 'V' release mean the Vampire release? Does this statement mean that those API calls will be available in all tentacle releases and umbrella, but maybe not vampire? that's right, Chris > I need to have the CreateRole,PutRolePolicy et al available in tentacle and do a planned transition to IAM users/roles. Any documentation on how to do this? I have STS roles for various groups of users to gain temporary access to solme number of buckets and I want to be sure my upgrade to tentacle will be ok. (I am on 19.2.5 squid). We are not using tenants but buckets owned by various s3 users and roles to give temporary access to buckets by other users. We have been doing this since either nautilus or octopus and it has worked well for us. (We do not use the gui for role management but have our own python scripts to add users/buckets to roles). https://docs.ceph.com/en/squid/radosgw/account/#migrate-an-existing-user-in… shows how to migrate existing users into an account, but not how to migrate existing roles. you'll need to create new roles in accounts and delete the old ones because the initial sts implementation treated the tenant like an account, the most direct mapping would be to create one account per tenant (in your case just the global/unnamed tenant) and migrate all of that tenant's users into it. you could then recreate the roles in that account, with role policy that names the same buckets and user principals. however, all buckets would be owned by the account and no users would have any access by default. so you'd have to use iam policy to control all permissions, not just to facilitate sharing between users/buckets if there are distinct groups of ownership/sharing, it would make more sense to create separate accounts for each. users within each account could be granted direct access to that account's buckets, and account roles could be used to share access to some of those buckets with other trusted accounts/users. you'd also have the option to give the account root user's credentials to someone in that group so they could use the iam apis to manage the sharing of their own resources. it's this self-service aspect of accounts that most differentiates it from the tenant-level sts support, which was an admin-only feature if you do decide to put all of the users in a single account, you might consider using iam groups rather than roles. you could use group policy to grant access to specific buckets in the account, then manage group membership with iam:AddUserToGroup/RemoveUserFromGroup. this would avoid the need for role trust policy to name all of the individual users, and those users wouldn't need to go through AssumeRole for that shared access > -Chris > > > > > > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0