- ceph-users - lists.ceph.io

Ceph Tentacle 20.2.4 Noble support
by Marek Szuba 24 Aug '26

24 Aug '26

Dear Ceph maintainers, Further to the thread about 20.2.3 Noble support earlier this month, I would like to point out that with there having been Noble packages for up to and including 20.2.0, Noble users are now stuck with Ceph versions with known security vulnerabilities. For many operators, ourselves included, a downgrade to Jammy is not possible even it it were to be done on Ceph servers themselves (as I mentioned in passing in one of my messages to this list we experienced severe I/O issues with Jammy-backed Ceph OSDs in one of our clusters, whereas another requires Noble or newer in order for Podman to be new enough for SMB containers). I would therefore STRONGLY encourage you to resume the building and publishing of packages for Noble ASAP irrespective of whether this particular version of Ubuntu has reached the "officially supported" status or not. Thank you for your consideration. -- MS

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Alexander Patrakov 22 Aug '26

22 Aug '26

On Fri, Aug 21, 2026 at 7:28 PM Stephan Hohn <ceph-users(a)ceph.io> wrote: > podman run --privileged --entrypoint /tmp/cryptsetup_action.sh \ > -v /tmp/cephadm-osd-14-rotate-helper.sh:/tmp/cryptsetup_action.sh <image> Hey, isn't this (the use of a predictably named file in /tmp, that the adversary can pre-create, e.g., as a symlink to something else) a vulnerability by itself? -- Alexander Patrakov

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by William Xiong 21 Aug '26

21 Aug '26

Hi Patrick, Regarding the fix for CVE-2025-30156 We've discovered that the bluestore label still contains the old key after following instructions at https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… Old key shows up as per: ceph-bluestore-tool show-label --dev "$(readlink -f /var/lib/ceph/osd/ceph-X/block)" | grep osd_key If we reboot the osd servers, the new key is overwritten with the old key in /var/lib/ceph/osd/ceph-${OSD_ID}/keyring causing osd auth failure > osd service doesnt start Can simulate this without reboot via `umount /var/lib/ceph/osd/ceph-${OSD_ID}/` and reactivate via `ceph-volume lvm activate --all` We are running a non cephadm cluster. Need to update the label as well after rotating the key: ceph-bluestore-tool set-label-key --dev "$(readlink -f /var/lib/ceph/osd/ceph-${OSD_ID}/block)" -k osd_key -v $NEW_KEY Kind Regards, William Xiong | Infrastructure Engineer - Storage Research Computing Services The University of Melbourne, Victoria 3010 Australia E: william.xiong(a)unimelb.edu.au<mailto:william.xiong@unimelb.edu.au> ________________________________ From: Patrick Donnelly <ceph-users(a)ceph.io> Sent: Thursday, 20 August 2026 2:08 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released Today, August 19th, 2026, the Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components. This is the sixth backport release in the Squid series. This is the fourth backport release in the Tentacle series. Additional details will be posted on the Ceph website/blog: https://ceph.io/en/news/blog We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible. ## Critical Upgrade Steps The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at: https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… These steps are for package-based deployments of Ceph. In other sections, the document also includes new information about key rotation procedures, monitor authentication controls, and emergency recovery procedures. Deployments using cephadm will automate the process except for client keys. You may notice that Cephadm spends more time than normal on the upgrade after updating all daemon images. This is due to a new process rotating the OSD and MDS keys. Please be patient. There are plans to improve this in the future. Deployments using Rook will also automate rotation of some client keys with some exclusions. Look to the Rook project’s announcement for more details. Client and kernel upgrades are recommended to support aes256k. Note that upstream Linux kernel client support began in kernel 7.0 and has been backported to CentOS Stream 9 and 10. Check with your distribution vendor for backported key support before rotating client keys used by the kernel. As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go. These are documented beginning here: https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… The CephX documentation has been brought up to date and vetted for accuracy. Those changes are live here: https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 requests with `host` and `x-amz-` headers not included in the signed subset. Unfortunately, the REST client used in multisite was generating such improperly signed requests. If you are running multisite, you must set the ``rgw_sigv4_insecure`` option to true before you begin to upgrade. After all clusters are upgraded, set the option to ``false`` again. Regarding the fix for CVE-2026-50152, formal guidance on rotating all secrets stored in the Monitor config-key store will be forthcoming. In the meantime, operators of cephadm-managed clusters are strongly encouraged to rotate the cephadm SSH key, as an established process exists: See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration Rotating other secrets is advised but the process is not yet established. Operators should assess their cluster's potential exposure to determine if immediate Monitor config-key secret rotation is required for their environment. ## CVE Fixes * CVE-2025-30156 is an authentication bypass in CephX caused by misuse of AES-CBC. See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 * CVE-2026-39944 shares the unauthenticated-encryption root cause of CVE-2025-30156, but applies it to RGW's STS session tokens resulting in improper verification of a cryptographic signature. See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor subscription handler. See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 cryptographic signatures in RGW's SigV4 verifier. See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 ## Other Media A Cephalocon 2025 talk covers some of the details surrounding the new CephX key upgrade process. https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… ## Getting Ceph ------------ * Source at https://github.com/ceph/ceph * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages * For upgrade with cephadm, see: https://docs.ceph.com/en/latest/cephadm/upgrade * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Robert Sander 21 Aug '26

21 Aug '26

Hi Patrick, Am 21.08.26 um 4:27 PM schrieb Patrick Donnelly: > Again, rotating the client keys is not necessary to address the main > vulnerabilities associated with the CVE. You may mute the > AUTH_INSECURE_CLIENT_KEY_TYPE warning on an extended basis or > permanently if it cannot be resolved for the forseeable future. Thanks for pointing this out so clearly. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Nico Schottelius 21 Aug '26

21 Aug '26

Ciao Robert, Robert Sander <ceph-users(a)ceph.io> writes: > Hi, > > Am 21.08.26 um 4:10 PM schrieb Nico Schottelius: > >> a) All VPS/VMs need to be restarted >> To be able to rotate keys, aside from the almost impossible flow to >> actually doing the rotation (more to that below), this will affect >> every consumer of VMs. The usual setup is that all hypervisors are >> using the same key. > > VM live migration should be able to do the trick here. When the new > compute node already has the new key and the librbd version to support > it. That's a very good point. However this might also need work (or hacks) in the hypervisor sphere, to support using different keys in the 2 qemu processes. By default in opennebula, the ceph key is saved in a libvirtd secret. So in the best case, the new process will read it from the local secret store. In the not so great case, opennebula (and other hypervisors) might infer the key from the previously deployed VM, instead of reading it from the local secret store. (*) >> I'm not saying anything cannot be done, but the way how this >> vulnerabilty is solved will cause major headache around the industry. > > IMHO we will see a muted health warning for the years to come. Agreed. BR, Nico (*) I'm not sure about the state of other hypervisors, but in the opennebula case keys are actually embededd in the XML VM definition (sic!) that is saved in the database (sic!!) and copied on creation of the VM (!!!) and will never every be changed. A likely working hack is to modify the key inside the XML inside the database and then trigger live migration with a small time window of uncertainty that if the VM crashes on the original host before the live migration is triggered, it will likely not restart. -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 21 Aug '26

21 Aug '26

On Fri, Aug 21, 2026 at 10:08 AM Nico Schottelius <ceph-users(a)ceph.io> wrote: > > > Good morning list, > > some of you already mentioned the issue in relation to proxmox, but > virtually all VPS/VM hostings are affected that are running qemu/kvm. > > The situation is even worse, if I see it correctly: > > a) All VPS/VMs need to be restarted > To be able to rotate keys, aside from the almost impossible flow to > actually doing the rotation (more to that below), this will affect > every consumer of VMs. The usual setup is that all hypervisors are > using the same key. > > b) Key support in qemu/kvm > From my understanding, qemu/kvm + librbd does at the moment not have > support for the new key type. Thus there will be major waiting > periods until the packages have been updated and eventually make it > back into the distros. If they are being done at all, depending on > your setup there might be old qemu versions which cannot easily be > upgraded. > > c) Key rotation > Assuming that the support actually has landed in qemu/kvm (and that > can still take months), one will need to rotate potentially > update/edit generade libvirtd xml that contains the reference to the > ceph key or the key itself. > > I'm not saying anything cannot be done, but the way how this > vulnerabilty is solved will cause major headache around the industry. Again, rotating the client keys is not necessary to address the main vulnerabilities associated with the CVE. You may mute the AUTH_INSECURE_CLIENT_KEY_TYPE warning on an extended basis or permanently if it cannot be resolved for the forseeable future. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 2

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Jan Kasprzak 21 Aug '26

21 Aug '26

Hello Patrick, regarding the step 4 of the documentation: I upgraded another cluster, and now I see the issue: > 4. Confirm the AUTH_INSECURE_SERVICE_KEY_TYPE is cleared. > > ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_SERVICE_KEY_TYPE") | not' > > output gives false. > > If it outputs true, there is another daemon that needs to be upgraded. > Check the output of ceph health detail. This is incorrect. When all the daemons have their keys rotated, then "ceph health detail" does _not_ contain AUTH_INSECURE_SERVICE_KEY_TYPE error, so the above "... | not" returns true, not false. So either remove the "| not" part from the docs, or replace false <--> true in the next two sentences. Regards, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Dmitry Melekhov 21 Aug '26

21 Aug '26

21.08.2026 18:06, Nico Schottelius пишет: > Good morning list, > > some of you already mentioned the issue in relation to proxmox, but > virtually all VPS/VM hostings are affected that are running qemu/kvm. > > By the way I don't see that CVE is critical in case of hypervisor is running on the same hosts and subnet is isolated by firewall. May be it is better to have an option to not change keys during upgrade but leave everything as it is until it is suitable time to implement more secure keys?

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Robert Sander 21 Aug '26

21 Aug '26

Hi, Am 21.08.26 um 4:10 PM schrieb Nico Schottelius: > a) All VPS/VMs need to be restarted > To be able to rotate keys, aside from the almost impossible flow to > actually doing the rotation (more to that below), this will affect > every consumer of VMs. The usual setup is that all hypervisors are > using the same key. VM live migration should be able to do the trick here. When the new compute node already has the new key and the librbd version to support it. > I'm not saying anything cannot be done, but the way how this > vulnerabilty is solved will cause major headache around the industry. IMHO we will see a muted health warning for the years to come. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 21 Aug '26

21 Aug '26

On Fri, Aug 21, 2026 at 7:40 AM Burkhard Linke <ceph-users(a)ceph.io> wrote: > > Hi, > > On 8/20/26 21:35, Patrick Donnelly wrote: > >> A challenge to that approach is that Proxmox PVE defines external RBD > >> pools at the datacenter level using a single user/key pair. This then > >> becomes the storage pool on each individual node in the proxmox > >> cluster. I believe ceph-csi K8 storage class also defines this once > >> cluster wide. > > I understand but the only way to change things is to spread the word. > > If the key is that challenging to rotate, then I would suggest leaving > > it as aes and push for changes in the product or deployment > > technology. Rook developers are already aware this should be reworked. > > So you expect every user running Proxmox PVE, K8s with ceph csi or > Openstack (which is usually more than just a handful of hosts) to figure > out how to adopt their deployment? Which is not possible at this time > for the mentioned cases. I'm not sure what you mean. I think the docs are clear that it may not be possible to rotate client keys to the stronger key type due to various technical hurdles. That is okay. The CVE is mitigated by updating the core Ceph daemons. > Most users will probably welcome a more detailed documentation on how to > handle key rotation _and_ especially how to manage the client side in > this scenario. As was already mentioned before, most of us cannot simply > shutdown all virtual machines in Proxmox or Openstack for a key rotation.... https://docs.ceph.com/en/latest/security/CVE-2025-30156/ "Client/kernel upgrades are recommended to support the new key type but not required to resolve the most serious aspects of the security vulnerability." and https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… "If you cannot rotate a particular client key yet, you may prefer to mute the health warning until you can complete upgrading all of the client keys. We expect this to be typical situation for some clusters." How would you like the docs improved? -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0