August 2026 - ceph-users

Maintenance Notification: Mailing List Migration 9/4
by David Galloway 04 Sep '26

04 Sep '26

Hello, Just a quick heads up that the mailing lists will be migrating back into the Sepia lab on 9/4 at 10am Eastern time. Aside from no longer needing to pay AWS for the service, additional benefits we’ll see from this migration include: * Upgrade from mailman 3.3.1 to 3.3.10. This version includes a task runner that will automatically clean up stuck list subscriptions. This was the main cause for the Web UI hanging and going down over the past few years. * Postgres DB migration from 9.6 to highly available Crunchy DB based on Postgres 16 * Lab-wide SMTP relay * Modern anti-spam stack (postscreen, rspamd) * In-house DKIM signing (mail-tester.com rates us a 8.9/10 score) * Mailing alias (vs. an entire list) support * Proper reverse DNS (helps our spam score) The migration itself shouldn’t take more than 30 minutes. It’s already been tested and everything that can be staged ahead of time is already ready. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway(a)ibm.com IBM

1 3

umbrella 21.1.1 RC2 QE validation status
by Yuri Weinstein 03 Sep '26

03 Sep '26

Details of this release are summarized here: https://tracker.ceph.com/issues/78413 Release Notes - email version for RCs https://github.com/ceph/ceph/pull/70240#issuecomment-5232852937 @Neha Ojha <nojha(a)redhat.com> @Patrick Donnelly <pdonnell(a)redhat.com> LRC upgrade - TBD PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: PASSED rados: Kamoltat, Radek, Nitzan, Afreen (dashboard), Travis (rook) rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane, Guillaume rbd: Ilya krbd: Ilya nvmeof: PASSED upgrade: Kamoltat upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: PASSED upgrade-clients:client-upgrade-squid-umbrella-umbrella: PASSED perf-basic: PASSED crimson-rados: Matan powercycle: Brad (same as before) TIA

1 1

Python version for cephadm orchestrator
by Robert Sander 31 Aug '26

31 Aug '26

Hi, is there a possibility to set the Python interpreter path that the cephadm orchestrator uses on the nodes? We currently deal with a cluster that runs on SLES 15.7 where the default Python version is still 3.6 and /usr/bin/python3 points to that version. Python 3.11 is available as /usr/bin/python3.11 Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

19.2.6 garbage collection: RGWGC::send_split_chain - send chain returned error: -28
by Manuel Rios - EDH 31 Aug '26

31 Aug '26

Hi , Since last patch 19.2.6 squid (stable), when we delete a bucket with the common cli: radosgw-admin bucket rm --bucket=XXXXX --purge-objects We get this error: 2026-08-31T08:10:45.237+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:10:47.394+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:10:59.289+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:11:21.806+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:11:23.422+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 2026-08-31T08:11:31.995+0000 753eebf64900 0 garbage collection: RGWGC::send_split_chain - send chain returned error: -28 Know error or new bug? ADVERTENCIA LEGAL: Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado. Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. [Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Flor ecoTECH]No imprimas si no es necesario. Protejamos el Medio Ambiente.

1 0

Re: Affects of high pg count on OSD peering
by Elias Carter 31 Aug '26

31 Aug '26

Regarding osd_heartbeat_interval, I set it to 1 in our production clusters and did some tests with marking OSDs as out/in. From my tests, I did not see any noticeable difference on client side IOs between osd_heartbeat_interval=6 and osd_heartbeat_interval=1. In both cases p99 RBD latency increased by about 40% for ~10 seconds after marking OSDs as out/in. Elias On Wed, Aug 19, 2026 at 9:17 AM Elias Carter <ceph-users(a)ceph.io> wrote: > > Can you share these values? > > bash-5.1$ ceph config get osd osd_max_pg_per_osd_hard_ratio > 3.000000 > bash-5.1$ ceph config get mon mon_max_pg_per_osd > 600 > > On Tue, Aug 18, 2026 at 11:00 PM Eugen Block <eblock(a)nde.ag> wrote: > > > Can you share these values? > > > > osd_max_pg_per_osd_hard_ratio > > mon_max_pg_per_osd > > > > Zitat von Elias Carter <elias(a)dropbox.com>: > > > > >> Oh, in your first message you wrote you have 300 PGs per OSD, which I > > > didn’t find that high. Now you wrote 600 PGs per OSD, which is it? > > > > > > My mistake, our target is 300 PGs per OSD and the maximum is 600 PGs > per > > > OSD. Currently, our clusters are hovering around the target of 300 PGs > > per > > > OSD. > > > > > >> Can you share some details about the OSDs (size)? > > > > > > We are running 14TiB NVMe SSD drives (PCIe 4). Our largest cluster > right > > > now has approximately 400 OSDs. > > > > > > Regarding tuning osd_heartbeat_interval, I did gradually lower it from > 6 > > to > > > 1 and found that packets per second increased by about 50% on the OSDs. > > > Although we are only seeing tens of thousands of packets per second per > > > OSD, these nodes have NICs rated for 100 million packets per second, so > > we > > > are not even close to hitting limits there. So far I don't see any > > cluster > > > instability caused by setting osd_heartbeat_interval to 1 in our > > clusters. > > > My assumption is that the number of heartbeat packets will increase > > > quadratically with the number of OSDs in the cluster (given 300 PGs/OSD > > > there is a good likelihood that any one OSD peers with almost every > other > > > OSD?). > > > > > > Elias > > > > > > On Tue, Aug 18, 2026 at 1:15 PM Eugen Block <eblock(a)nde.ag> wrote: > > > > > >> Oh, in your first message you wrote you have 300 PGs per OSD, which I > > >> didn’t find that high. Now you wrote 600 PGs per OSD, which is it? 😁 > > >> 600 sounds like a lot indeed, that could be one of the downsides of > > >> having too many PGs per OSD. Some time ago there was a thread here on > > >> the list about the ancient recommendation of 100 per OSD and nobody > > >> seemed to know the exact reason. Several people (including me) > > >> mentioned good experiences with around 300 to 400 PGs per OSD. Maybe > > >> you found the max value 😅 > > >> > > >> Can you share some details about the OSDs (size)? > > >> > > >> Zitat von Elias Carter <elias(a)dropbox.com>: > > >> > > >> >> the IO should not be blocked for the entire pool but only for those > > >> > repeering PGs > > >> > > > >> > Yes, the issue is that when running 600 PGs per OSD a high fraction > of > > >> PGs > > >> > re-peer simultaneously. > > >> > > > >> >> Gradually reweighting instead of setting an OSD "out" directly is a > > >> > common practice among ceph users, there should be plenty of threads > on > > >> > this list about that. The downside is of course a lot more manual > > >> > intervention and possibly (some) redundant data movement. > > >> > > > >> > Thank you for the confirmation. We run Ceph within kubernetes (rook) > > and > > >> my > > >> > plan is to write an operator which will handle gradually reweighting > > OSDs > > >> > as they are added/removed from the cluster. Rook allows setting the > > >> initial > > >> > OSD weight, so my plan is to set the initial OSD weights to 0 and > then > > >> have > > >> > the operator gradually reweight the OSDs to 1. > > >> > > > >> >> One knob to turn could be to decrease osd_heartbeat_interval > (number > > >> > of seconds between peer pings) from 6 seconds to a lower value (1 > > >> > second is the minimum, but I probably would start with 5 or 4 > > >> > seconds). > > >> > > > >> > Thank you; I will try that out. We have a mix of 100G and 400G > > networks, > > >> so > > >> > I think the increased ping frequency will be ok. I will try it out > on > > our > > >> > staging clusters first and report back if there are any interesting > > >> > findings. > > >> > > > >> > Elias > > >> > > > >> > On Tue, Aug 18, 2026 at 2:21 AM Eugen Block <ceph-users(a)ceph.io> > > wrote: > > >> > > > >> >> Hi, > > >> >> > > >> >> the IO should not be blocked for the entire pool but only for those > > >> >> repeering PGs. Warnings during that phase are expected. > > >> >> Gradually reweighting instead of setting an OSD "out" directly is a > > >> >> common practice among ceph users, there should be plenty of threads > > on > > >> >> this list about that. The downside is of course a lot more manual > > >> >> intervention and possibly (some) redundant data movement. > > >> >> > > >> >> One knob to turn could be to decrease osd_heartbeat_interval > (number > > >> >> of seconds between peer pings) from 6 seconds to a lower value (1 > > >> >> second is the minimum, but I probably would start with 5 or 4 > > >> >> seconds). That could (or probably would) increase network traffic > > >> >> significantly, so it's a trade-off. If you have a test cluster with > > >> >> similar behavior, test it there first. > > >> >> > > >> >> Regards, > > >> >> Eugen > > >> >> > > >> >> Zitat von Elias Carter via ceph-users <ceph-users(a)ceph.io>: > > >> >> > > >> >> > We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total > > PGs > > >> and > > >> >> > upwards). > > >> >> > > > >> >> > Something that I have noticed is that CRUSH changes (marking an > > OSD as > > >> >> out, > > >> >> > or a new OSD joining the cluster) can cause many hundreds of PGs > to > > >> remap > > >> >> > and peer. During this time we get PG_AVAILABILITY health check > > >> warnings > > >> >> as > > >> >> > the OSDs need to peer with their new PGs before they can serve > IOs. > > >> RADOS > > >> >> > tail latency suffers because IOs sent to a peering PG are blocked > > >> until > > >> >> > peering completes. > > >> >> > > > >> >> > I believe that the more PGs we have, the more PGs need to peer, > and > > >> the > > >> >> > longer IOs are blocked for the pool. > > >> >> > > > >> >> > Does anyone have suggestions on how to deal with PG peering > > blocking > > >> IOs? > > >> >> > > > >> >> > I was thinking about gradually re-weighting OSDs instead of > marking > > >> them > > >> >> > out or in, but that can result in multiple redundant data moves. > > >> >> > > > >> >> > Elias > > >> >> > _______________________________________________ > > >> >> > ceph-users mailing list -- ceph-users(a)ceph.io > > >> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> >> > > >> >> > > >> >> _______________________________________________ > > >> >> ceph-users mailing list -- ceph-users(a)ceph.io > > >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> >> > > >> > > >> > > >> > > >> > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

RGW fix for CVE-2026-54330 can break some S3 clients
by Sascha Lucas 31 Aug '26

31 Aug '26

Hi, it seems not spread into this list yet? If you have upgraded to ceph 20.2.4/19.2.6 some S3 clients like restic[1] or php[2] can break due to a new introduced ceph BUG: https://tracker.ceph.com/issues/79674 Affected users can temporary disable the fix at runtime without any restarts: $ ceph config set client.rgw rgw_sigv4_insecure true until 20.2.5/19.2.7 gets releases. Thanks, Sascha. [1] https://github.com/restic/restic/issues/22033 [2] https://lp.zeroservices.eu/articles/ceph-sigv4-when-a-cve-fix-breaks-your-c…

1 0

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Eugen Block 31 Aug '26

31 Aug '26

If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. Zitat von Brent Kennedy <ceph-users(a)ceph.io>: > Question, sort of related. If upgrading from squid to tentacle, > will it automatically try to rotate keys are part of the cluster > upgrade or is that still a manual process? We would like to avoid > the key rotation for now as we have some proxmox nodes using ceph > storage in some of the clusters. > > Regards, > -Brent > > Existing Clusters: > US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > > > > -----Original Message----- > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Saturday, August 29, 2026 10:38 AM > To: Saif Mohammad <samdto987(a)gmail.com> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Clarification on Client Key Rotation > During Upgrade to 19.2.6/20.2.4 > > Hello, > > On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: >> >> Hi, >> >> While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these >> versions introduce the secure key type, I have a clarification >> regarding client key rotation. >> For the client.admin key, we need to rotate the key and then >> import/update the keyring. >> For the other client keys, do we only need to rotate the key and >> verify that the key has changed? Is there no need to import/update >> the keyring for the other clients, since there is no local keyring >> available? > > Step (9) of [1] says: > > "Then copy and import the key to each machine using that client.$ID > credential." > > [1] > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Brent Kennedy 30 Aug '26

30 Aug '26

Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters. Regards, -Brent Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds -----Original Message----- From: Patrick Donnelly <ceph-users(a)ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad <samdto987(a)gmail.com> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 Hello, On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: > > Hi, > > While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. > For the client.admin key, we need to rotate the key and then import/update the keyring. > For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? Step (9) of [1] says: "Then copy and import the key to each machine using that client.$ID credential." [1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Patrick Donnelly 29 Aug '26

29 Aug '26

Hello, On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: > > Hi, > > While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. > For the client.admin key, we need to rotate the key and then import/update the keyring. > For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? Step (9) of [1] says: "Then copy and import the key to each machine using that client.$ID credential." [1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Missing Ubuntu 24.04 (noble) packages for Tentacle 20.2.3 and 20.2.4
by Homayoon Sadeghi 29 Aug '26

29 Aug '26

Hello, Tracker: https://tracker.ceph.com/issues/79633 Ubuntu 24.04 LTS (noble) Debian packages were published for Tentacle 20.2.1 and 20.2.2, but are absent from 20.2.3 and 20.2.4. The rolling debian-tentacle repository has also lost its noble suite. Because 20.2.4 is the hotfix for CVE-2025-30156, CVE-2026-39944, CVE-2026-50152 and CVE-2026-54330, operators running cephadm on Ubuntu 24.04 currently have no upstream path to install patched host tooling (cephadm, ceph-common, ceph-volume) at a version matching the 20.2.4 container image. download.ceph.com coverage: Release jammy noble bookworm 20.2.0 present missing (pulled) present 20.2.1 present present present 20.2.2 present present present 20.2.3 present missing present 20.2.4 present missing present debian-tentacle (rolling) present missing present Squid (19.2.2–19.2.6) never published a noble suite. Noble in Tentacle therefore only exists for 20.2.1–20.2.2. 20.2.0 noble was published and then removed on 20 Nov 2025 because ceph-mgr crashed on Python 3.12 (tracker #73930). The announcement said Ubuntu 24.04 would return in a later point release. 20.2.1 and 20.2.2 did; 20.2.3 and 20.2.4 did not. Reproduce: curl -sI https://download.ceph.com/debian-20.2.2/dists/noble/Release # HTTP/1.1 200 OK curl -sI https://download.ceph.com/debian-20.2.3/dists/noble/Release curl -sI https://download.ceph.com/debian-20.2.4/dists/noble/Release # HTTP/1.1 404 Not Found curl -s https://download.ceph.com/debian-20.2.4/dists/ curl -s https://download.ceph.com/debian-tentacle/dists/ # bookworm/ jammy/ curl -s https://download.ceph.com/debian-20.2.2/dists/noble/main/binary-amd64/Packa… \ | grep -A3 -E '^Package: (cephadm|ceph-common|ceph-volume)$' # Version: 20.2.2-1noble The Ubuntu archive only has Squid 19.2.3-0ubuntu0.24.04.3 for noble, so download.ceph.com is the only upstream Tentacle source for this distro. Jammy 20.2.4-1jammy debs are not usable on noble (ABI). This looks like a release-matrix / publish gap, not "noble cannot build": v20.2.4 commit 7f793731f1b39eb4f465e960113d2363c311b964 Jenkins ceph-dev-pipeline/8318 https://shaman.ceph.com/builds/ceph/tentacle/7f793731f1b39eb4f465e960113d23… That job built jammy (20.2.4-1jammy), bookworm, Rocky 10, CentOS 9. It did not build noble. Those other distros were copied to download.ceph.com; noble was not. v20.2.3 commit 06c2f9c35b67055a8a6fb99d1be236b3c4832ace got a late noble CI rebuild on 27 Aug (20.2.3-1noble) that was never copied to debian-20.2.3/dists/noble/. tentacle/latest/ubuntu/noble currently points at that unpublished 20.2.3 repo. Expected: 20.2.4-1noble under debian-20.2.4/dists/noble/ and debian-tentacle/dists/noble/, matching 20.2.1 and 20.2.2. Questions: 1. Was noble coverage dropped after 20.2.2, or is this a break in the release/publishing pipeline? 2. If intentional, can this be documented and announced, given 20.2.1/20.2.2 already shipped noble? 3. Can noble packages be produced for 20.2.4 out-of-band, given the CVE severity and the "upgrade as soon as possible" guidance? Thanks, Homayoon Sadeghi

1 0

Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Saif Mohammad 29 Aug '26

29 Aug '26

Hi, While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? We’re managing the cluster using cephadm. Could someone please confirm if my understanding is correct? Thanks, Mohammad Saif

1 0

Re: auth_service_cipher after 19.2.6 upgrade and other clients
by Christopher Durham 28 Aug '26

28 Aug '26

Eugen, Here is what you requested. # ceph mon dump | grep cipherdumped monmap epoch 5auth_service_cipher aesauth_allowed_ciphers aes, aes256kauth_preferred_ciphers aes I suspect that I am able set set auth_service_cipher to aes256k but wanted to be sure before I do so. -Chris On Friday, August 28, 2026 at 11:14:57 AM MDT, Eugen Block <eblock(a)nde.ag> wrote: Hi, can you share 'ceph mon dump |grep cipher'? If you use cephadm, the orchestrator should already have that done for you. The warning you're seeing will clear after a few hours automatically. If you don't use cephadm, you can safely set 'ceph mon set auth_service_cipher aes256k' because it really does only affect the core services, otherwise I'd have a problem. ;-) I upgraded our production (cephadm) cluster this week and all went well. We also can't upgrade our clients at this time, so we're all in the same boat, I guess. :-) Regards, Eugen Zitat von Christopher Durham <ceph-users(a)ceph.io>: > Hi: > After update to 19.2.6 I am so far successfully updating key types > for various daemons. I have the following messages in the output of > 'ceph health detail': > > [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth > service keys using insecure key types > rotating service keys for mon using insecure key type: aes > rotating service keys for mds using insecure key type: aes > rotating service keys for osd using insecure key type: aes > rotating service keys for mgr using insecure key type: aes > The doc says to set auth_service_cipher to aes256k. Before I change > that, I have a question: > Is this setting ONLY for mon/mds/osd/mgr? > I ask because i have several client keys that cannot be updated to > aes256k yet, such as the keys for ceph-csi in kubernetes. I want to > be sure that if update auth_service_cipher to aes256k, I am ONLY > affecting the rotating keys for mon/mds/osd/mgr. I do not want to > set auth_service_cipher to aes256k and then have the ceph-csi still > able to authenticate with older aes keys but not able to do rotating > keys after authentication ( I do not know if it does ).. Thanks for > any information > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: auth_service_cipher after 19.2.6 upgrade and other clients
by Eugen Block 28 Aug '26

28 Aug '26

Hi, can you share 'ceph mon dump |grep cipher'? If you use cephadm, the orchestrator should already have that done for you. The warning you're seeing will clear after a few hours automatically. If you don't use cephadm, you can safely set 'ceph mon set auth_service_cipher aes256k' because it really does only affect the core services, otherwise I'd have a problem. ;-) I upgraded our production (cephadm) cluster this week and all went well. We also can't upgrade our clients at this time, so we're all in the same boat, I guess. :-) Regards, Eugen Zitat von Christopher Durham <ceph-users(a)ceph.io>: > Hi: > After update to 19.2.6 I am so far successfully updating key types > for various daemons. I have the following messages in the output of > 'ceph health detail': > > [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth > service keys using insecure key types > rotating service keys for mon using insecure key type: aes > rotating service keys for mds using insecure key type: aes > rotating service keys for osd using insecure key type: aes > rotating service keys for mgr using insecure key type: aes > The doc says to set auth_service_cipher to aes256k. Before I change > that, I have a question: > Is this setting ONLY for mon/mds/osd/mgr? > I ask because i have several client keys that cannot be updated to > aes256k yet, such as the keys for ceph-csi in kubernetes. I want to > be sure that if update auth_service_cipher to aes256k, I am ONLY > affecting the rotating keys for mon/mds/osd/mgr. I do not want to > set auth_service_cipher to aes256k and then have the ceph-csi still > able to authenticate with older aes keys but not able to do rotating > keys after authentication ( I do not know if it does ).. Thanks for > any information > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

auth_service_cipher after 19.2.6 upgrade and other clients
by Christopher Durham 28 Aug '26

28 Aug '26

Hi: After update to 19.2.6 I am so far successfully updating key types for various daemons. I have the following messages in the output of 'ceph health detail': [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes The doc says to set auth_service_cipher to aes256k. Before I change that, I have a question: Is this setting ONLY for mon/mds/osd/mgr? I ask because i have several client keys that cannot be updated to aes256k yet, such as the keys for ceph-csi in kubernetes. I want to be sure that if update auth_service_cipher to aes256k, I am ONLY affecting the rotating keys for mon/mds/osd/mgr. I do not want to set auth_service_cipher to aes256k and then have the ceph-csi still able to authenticate with older aes keys but not able to do rotating keys after authentication ( I do not know if it does ).. Thanks for any information -Chris

1 0

ceph orch device zap can't wipe db/wal remainders
by Eugen Block 28 Aug '26

28 Aug '26

Hi, while I was investigating something else I noticed that the orchestrator still seems to fail wiping DB/WAL devices, it only wipes device paths like /dev/sdX. I couldn't find anything on tracker, only an old thread where I stated the same: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/H3KQGPIZXH… So the 'ceph orch device zap' command would only work for a standalone OSD. If the OSD has a separate DB/WAL you can't specify the entire device path of the fast device since it's usually the DB/WAL device for multiple OSDs. But you also can't specify the VG/LV syntax as you can with 'cephadm ceph-volume lvm zap --destroy VG/LV': host1:~ # ceph orch device zap host6 /dev/ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18 --force Error EINVAL: Device path '/dev/ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18' not found on host 'host6' Obviously, the preferred choice in such a case would be utilizing ceph-volume which can zap both VGs/LVs: host6:~ # cephadm ceph-volume lvm zap --destroy ceph-f0b8e0c6-840e-4ed5-ade3-de232e35a4b4/osd-block-6e58bd57-86ed-463b-8e8c-8ab64eb3e42f ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18 Is there anything planned for the orchestrator to support that as well? Thanks! Eugen

1 0

Re: v20.2.4 staggered upgrade
by Eugen Block 27 Aug '26

27 Aug '26

It got worse: reef1:~ # ceph orch ps --daemon-type mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.reef1.eqokui reef1 *:9283,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765 running (3h) 7m ago 23h 198M - 20.2.4 6facee348180 472d9e5cd38a mgr.reef3.wiuxme reef3 *:8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765 running (3h) 7m ago 23h 144M - 20.2.4 6facee348180 820c2422172f I started from scratch (purged the previous cluster) and upgraded from: 18.2.2 -> 18.2.5 -> 18.2.6 -> 18.2.7 -> 18.2.8 -> 19.2.0 -> 19.2.1 -> 19.2.2 -> 19.2.3 -> 19.2.4 -> 19.2.5 -> 20.2.0 -> 20.2.1 -> 20.2.3 -> 20.2.4 So these are 14 Upgrades but I'm counting 39 and 27 entries for port 8765, so that doesn't seem to correlate. I have no idea yet where that comes from :-D Maybe there were some leftovers on those VMs that somehow confused something. I have other test clusters and our production cluster as well on either 20.2.4 or 19.2.6 and can't see this effect anywhere else (yet). Zitat von Jean-Marie de Boer <ceph-users(a)ceph.io>: > I have tested a staggered upgrade from 19.2.4 to 19.2.6. > > Indeed the OSD keys got rotated multiple times, I guess it may > always happen when upgrading, even if it's a single daemon type > that's later in the sequence. It's not really a problem but it does > make everything take quite a bit longer (we have 1000+ OSD's on > production clusters and must do a staggered upgrade because we need > to manage the MDS failover). I think the devs are aware though. > > I do not observe this port multiplication however. > > mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 > running (7h) 42s ago 5w 478M - 19.2.6 > ff836bb28e7d 251c5d067daf > mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 > running (14m) 42s ago 5w 475M - 19.2.6 > ff836bb28e7d 45c38fb90d31 > mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 > running (7h) 42s ago 5w 675M - 19.2.6 > ff836bb28e7d 1d69113ae7ed > > However I have been observing some strangeness with daemons showing > non-standard status: > > mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 > running (6h) 3m ago 5w 478M - 19.2.6 > ff836bb28e7d 251c5d067daf > mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 > unknown 8m ago 5w 339M - 19.2.6 > ff836bb28e7d 98add61a7280 > mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 > running 0s ago 5w - - <unknown> > <unknown> <unknown> A node failed to finish starting its > containers without a clear reason why, it also refused to exit > maintenance, and a mon on another node failed and fixed itself > without intervention. I have not really determined a clear cause. > > Most of that fixed itself eventually although I now notice a single > crash daemon playing up. > > Of course we're making it hard because we do a staggered upgrade and > we use multiple routed subnets too :) > > Regards, > JM > > > On 8/26/26 12:21, Eugen Block wrote: >> I haven't seen anything like this yet: >> >> mgr.reef1.zfjbzv reef1 >> *:9283,8765,8765,8765,8765,8765,8765 running (2m) 109s ago >> 18h 129M - 20.2.4 6facee348180 ed9ae39b8f5f >> mgr.reef2.omwexa reef2 >> *:8765,8765,8765,8765,8765,8765,8765 running (2m) 108s ago >> 14h 167M - 20.2.4 6facee348180 b24464d5c98b >> >> I just did a staggered upgrade in a test cluster from 18.2.7 to >> 20.2.4 and I noticed that it goes through redeploying all daemons >> with a new keyring everytime although they already had been >> upgraded, for example MGR, MON and OSD when I was trying to upgrade >> MDS only (--daemon-type MDS). >> >> A 'ceph mgr fail' does not clean up those redundant port entries. >> Should I create a tracker for that or is this already known? >> >> I haven't tried the same on 19.2.6 yet. >> >> Regards, >> Eugen >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > > Jean-Marie de Boer > Specialist > > SURF | Science Park 140 | 1098 XG Amsterdam | www.surf.nl > E: jean-marie.deboer(a)surf.nl | M: +31610961787 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 2

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Anthony D'Atri 27 Aug '26

27 Aug '26

I'd no idea either > On Aug 27, 2026, at 6:12 AM, Eugen Block <ceph-users(a)ceph.io> wrote: > > Apparently I either forgot that I have or really never have used 'ceph config set --force' before. :-D

1 0

Best way to find out the client key support
by Albert Shih 27 Aug '26

27 Aug '26

Hi, Our ceph client (cephfs) mostly run Debian 12/13 with the native pkg of ceph-common. And currenlty if I check the https://download.ceph.com/debian-20.2.4 they are still not the ceph-common & co for Debian Trixie. We also have proxmox client (rdb) and openshift (rooks) client. So what's the best way to check if the client can use aes256k key ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 27 août 2026 14:22:01 CEST

1 0

Re: Rotation key after upgrade to 20.2.4
by Albert Shih 27 Aug '26

27 Aug '26

Le 27/08/2026 à 10:10:09+0000, Eugen Block a écrit Hi, > > please read all the sections in the docs ;-) > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > Although you use cephadm, step 6 is still relevant because it explains the > warning you're seeing: I read that, the problem is I just understand they are two types of keys :-( :-( > > > 6. Wipe the rotating service keys. > > Warning > > This is not recommended for most deployments. It is best to let your > > rotating service keys expire after a few hours (using default TTL). > > > Warning > > Only perform this step if all service daemons have upgraded binaries > > that understand the new cipher type. > > If you want to immediately clear the > > AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning, you can wipe the > > existing rotating service key database on the Monitors: > > > This is just a transitional state, the warning will go away after a few > hours (I'm not sure how long exactly). You're 100% right. The message just vannish. Thanks. Regards. -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 27 août 2026 14:19:14 CEST

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 27 Aug '26

27 Aug '26

Apparently I either forgot that I have or really never have used 'ceph config set --force' before. :-D Thanks for pointing it out, Boris and Chris. Zitat von Christopher Durham <ceph-users(a)ceph.io>: > Boris, et al: > What this appears to mean is that the rgw_sigv4_insecure option does > not exist in releases prior to 19.2.6/20.2.4, but using 'force' sets > it anyway, so that when a 19.2.6 or 20.2.4 version of radosgw > starts, it will see it and use it. > I suspect that if a 19.2.6 or 20.2.4 radosgw starts without this > option it will fail to do replication to say an earlier radosgw > without the signature fix on the other side of a multisite that has > not yet updated. As we always have, we plan on doing the update on > one side of the multisite on day 1, and the other side on day 2. The > force option seems to make this possible without having to modify > ceph.conf everywhere. > If I am wrong, please someone correct me. Thanks. > -ChrisOn Tuesday, August 25, 2026 at 01:49:30 AM MDT, Boris > <bb(a)kervyn.de> wrote: > > Hi Matthew, > > we've set it with: > ceph config set client.rgw rgw_sigv4_insecure true --force > > Best > Boris > > > > Am Di., 25. Aug. 2026 um 09:41 Uhr schrieb Matthew Vernon > <ceph-users(a)ceph.io>: >> >> Hi, >> >> On 19/08/2026 17:09, Patrick Donnelly wrote: >> >> > If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> > option to true before you begin to upgrade. After all clusters are >> > upgraded, set the option to ``false`` again. >> >> Stupid question - how/where does one set this, and will this be >> understood by older versions of Ceph? >> >> I ask because I have a Reef multisite setup, which should be supported >> to upgrade to Tentacle. Which I fear I'm now going to have to do in a hurry! >> >> AFAICT this is a new config setting, so I'd expect older versions of >> Ceph won't understand it... [and searching the reef docs doesn't find it] >> >> Thanks, >> >> Matthew >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend > im groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Rotation key after upgrade to 20.2.4
by Eugen Block 27 Aug '26

27 Aug '26

Hi, please read all the sections in the docs ;-) https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… Although you use cephadm, step 6 is still relevant because it explains the warning you're seeing: > 6. Wipe the rotating service keys. > Warning > This is not recommended for most deployments. It is best to let your > rotating service keys expire after a few hours (using default TTL). > Warning > Only perform this step if all service daemons have upgraded binaries > that understand the new cipher type. > If you want to immediately clear the > AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning, you can wipe the > existing rotating service key database on the Monitors: This is just a transitional state, the warning will go away after a few hours (I'm not sure how long exactly). Regards, Eugen Zitat von Albert Shih <ceph-users(a)ceph.io>: > Hi everyone > > So today I upgrade (one of) my ceph cluster (cephadm), and indeed after > upgrading some time all osd come back to life and all osd key seem to be in > aes256k > > But still with ceph health detail I've got > > [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth > service keys using insecure key types > rotating service keys for mon using insecure key type: aes > rotating service keys for mds using insecure key type: aes > rotating service keys for osd using insecure key type: aes > rotating service keys for mgr using insecure key type: aes > > When I look in the documentation > > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > section 3 it seem I had to copy the keyring directly to the nodes of my > cluster who run those service, but inside > > /var/lib/ceph/CEPHID/* > > I didn't find any keyring file related to mon/mds/osd/mgr with aes key. > > So I'm a little confuse...in the documentation it's say in a note: > > The mon. historically has not been managed by the Monitor auth > database; it exists soley in each Monitor’s keyring inside its data > directory. This suggested rotation procedure now puts the > authoritative copy in the auth database alongside other keys. The > Monitor keyring persists as a fallback or emergency key. > > so where are those rotation key ? > > Regards > > -- > Albert SHIH 🦫 🐸 > France > Heure locale/Local time: > jeu. 27 août 2026 11:50:08 CEST > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Rotation key after upgrade to 20.2.4
by Albert Shih 27 Aug '26

27 Aug '26

Hi everyone So today I upgrade (one of) my ceph cluster (cephadm), and indeed after upgrading some time all osd come back to life and all osd key seem to be in aes256k But still with ceph health detail I've got [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes When I look in the documentation https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… section 3 it seem I had to copy the keyring directly to the nodes of my cluster who run those service, but inside /var/lib/ceph/CEPHID/* I didn't find any keyring file related to mon/mds/osd/mgr with aes key. So I'm a little confuse...in the documentation it's say in a note: The mon. historically has not been managed by the Monitor auth database; it exists soley in each Monitor’s keyring inside its data directory. This suggested rotation procedure now puts the authoritative copy in the auth database alongside other keys. The Monitor keyring persists as a fallback or emergency key. so where are those rotation key ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 27 août 2026 11:50:08 CEST

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Albert Shih 27 Aug '26

27 Aug '26

Le 20/08/2026 à 19:55:02+0000, Salatiel Filho a écrit > Hi, I also miss in the docs the steps for those clients that are not > daemons? ( cephadm managed ) > > entity client.admin using insecure key type: aes They are a section (8) for the client.admin key rotation in the official documentation : https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… I didn't do it currently so no way to tell if that's work or not. Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 27 août 2026 11:48:06 CEST

1 0

Re: Ubuntu 24.04 Noble package support for Ceph 20.2.x
by Casey Bodley 27 Aug '26

27 Aug '26

according to Patrick in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/UEGCE3TKWY…, ubuntu 24.04 support will be added for tentacle in 20.2.5 On Wed, Aug 26, 2026 at 11:41 AM Martin Vobruba <ceph-users(a)ceph.io> wrote: > > Hi, > > I'd like to ask about the status of Ubuntu 24.04 Noble package support for > the Ceph 20.2.x release series. When I originally deployed the cluster, > packages for Noble were available and our clients are currently running > Ceph 20.2.2 on Ubuntu 24.04. However, I noticed that Noble packages are no > longer available for newer 20.2.x releases. I would like to upgrade the > clients to 20.2.4, primarily to get the fix for the insecure keys issue. > > Is the lack of Noble packages temporary? If so, is there any estimate for > when Ubuntu 24.04 packages for the 20.2.x branch will become available > again? Also, is there a recommended upgrade path for Noble clients > currently running 20.2.2 in the meantime? > > Thanks for any clarification. > > Best regards, > Martin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Christopher Durham 26 Aug '26

26 Aug '26

Boris, et al: What this appears to mean is that the rgw_sigv4_insecure option does not exist in releases prior to 19.2.6/20.2.4, but using 'force' sets it anyway, so that when a 19.2.6 or 20.2.4 version of radosgw starts, it will see it and use it. I suspect that if a 19.2.6 or 20.2.4 radosgw starts without this option it will fail to do replication to say an earlier radosgw without the signature fix on the other side of a multisite that has not yet updated. As we always have, we plan on doing the update on one side of the multisite on day 1, and the other side on day 2. The force option seems to make this possible without having to modify ceph.conf everywhere. If I am wrong, please someone correct me. Thanks. -ChrisOn Tuesday, August 25, 2026 at 01:49:30 AM MDT, Boris <bb(a)kervyn.de> wrote: Hi Matthew, we've set it with: ceph config set client.rgw rgw_sigv4_insecure true --force Best Boris Am Di., 25. Aug. 2026 um 09:41 Uhr schrieb Matthew Vernon <ceph-users(a)ceph.io>: > > Hi, > > On 19/08/2026 17:09, Patrick Donnelly wrote: > > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > > option to true before you begin to upgrade. After all clusters are > > upgraded, set the option to ``false`` again. > > Stupid question - how/where does one set this, and will this be > understood by older versions of Ceph? > > I ask because I have a Reef multisite setup, which should be supported > to upgrade to Tentacle. Which I fear I'm now going to have to do in a hurry! > > AFAICT this is a new config setting, so I'd expect older versions of > Ceph won't understand it... [and searching the reef docs doesn't find it] > > Thanks, > > Matthew > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 26 Aug '26

26 Aug '26

Yeah, I manually replaced the old key in each hosts authorized_keys file (just before I saw your email!). Then it went away. This is a recurring sort of theme with cephadm though. It is supposed to abstract away many of the implementation details. But then the documentation doesn't cover it, so you have to guess what it should have done, and then fix it manually... Thanks, Chris On 26/08/2026 13:43, Eugen Block wrote: > Hi, > > did you add the new pub key to all the host's authorized_keys? > > clear-key and then generate-key is the correct way. > > Zitat von Chris Palmer <ceph-users(a)ceph.io>: > >> I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and >> now it is broken: >> >> The note below points towards the SSH Configuration section. That >> doesn't have anything specific about rotation, so I had to make some >> guesses. >> >> It says that a new SSH key can be generated with "ceph cephadm >> generate-key". It doesn't give any error, but using "ceph cephadm >> get-pub-key" suggests it hasn't actually done anything (I still get >> the original ssh-rsa key). The doc probably needs clarifying. >> >> So I tried "ceph cephadm clear-key" which worked, and then "ceph >> cephadm generate-key" again. This time I can see a new ssh-ed25519 >> key. I then failed all MGRs. >> >> Now however I have this for every host: >> >> root@c1:~# ceph cephadm check-host c2 >> check-host failed: >> Failed to connect to c2 at address (192.168.nnn.162): Failed to >> connect to c2 (192.168.nnn.162). Permission denied >> Log: Opening SSH connection to 192.168.nnn.162, port 22 >> [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 >> [conn=9] Local address: 192.168.nnn.161, port 35514 >> [conn=9] Peer address: 192.168.nnn.162, port 22 >> [conn=9] Beginning auth for user root >> [conn=9] Auth failed for user root >> [conn=9] Connection failure: Permission denied >> [conn=9] Aborting connection >> >> Have I done this incorrectly, or is the documentation wrong? What is >> the correct way to rectify this? >> >> Thanks, Chris >> >> On 19/08/2026 17:07, Patrick Donnelly wrote: >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> https://ceph.io/en/news/blog/ >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >>> >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >>> >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >>> >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >>> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >>> >>> >>> ## Getting Ceph >>> ------------ >>> * Source at https://github.com/ceph/ceph >>> * v19.2.6 Tarball at >>> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >>> * v20.2.4 Tarball at >>> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >>> * Containers at https://quay.io/repository/ceph/ceph >>> * For packages, see >>> https://docs.ceph.com/en/latest/install/get-packages/ >>> * For upgrade with cephadm, see: >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Ubuntu 24.04 Noble package support for Ceph 20.2.x
by Martin Vobruba 26 Aug '26

26 Aug '26

Hi, I'd like to ask about the status of Ubuntu 24.04 Noble package support for the Ceph 20.2.x release series. When I originally deployed the cluster, packages for Noble were available and our clients are currently running Ceph 20.2.2 on Ubuntu 24.04. However, I noticed that Noble packages are no longer available for newer 20.2.x releases. I would like to upgrade the clients to 20.2.4, primarily to get the fix for the insecure keys issue. Is the lack of Noble packages temporary? If so, is there any estimate for when Ubuntu 24.04 packages for the 20.2.x branch will become available again? Also, is there a recommended upgrade path for Noble clients currently running 20.2.2 in the meantime? Thanks for any clarification. Best regards, Martin

1 0

Re: Cephadm staggered upgrade --limit parameter not always applied
by Eugen Block 26 Aug '26

26 Aug '26

Hi, I haven't found anything explaining this yet, but I also haven't looked for a while. I just retried limiting RGW upgrade (from Reef to Tentacle): reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.4 --daemon-types rgw --limit 1 But the limit is still ignored: 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:26.554+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:26.555+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef3.boujxu (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef3.boujxu (1/1) All three RGW daemons have been upgraded automatically. I'm pretty sure that it worked at some point with RGWs, in my notes from one of our customer upgrades where this worked: ceph orch upgrade start --image registry:5000/ceph/ceph:v18.2.5 --daemon-types rgw --limit 1 I can't tell when it stopped working properly. Regards, Eugen Zitat von Andreas Eriksson <norrland(a)nullbyte.se>: > Hi, > > Were you able to find anything more about this? I'm looking at > upgrading a bunch of clusters from previous major versions, but do > not want to upgrade all RGW services. And also limit the number of > osds at the time that are being upgraded, to keep the recovery > traffic lower for example. > > Br, > > On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote: >> Adding the dev list >> >> Zitat von Eugen Block <eblock(a)nde.ag>: >> >>> Hi, >>> >>> I did a quick search on tracker but couldn't find anything >>> related. A customer reported this, and I can confirm the behaviour >>> on a lab cluster. I usually perform a staggered upgrade with >>> --daemon-types and --limit, but not for all daemon types. So I >>> haven't stumbled across this yet myself, but our customer did (I >>> can reproduce with --daemon-types as well). They upgraded from >>> latest Reef to latest Squid and reported that despite providing >>> the --limit parameter, the mons were all upgraded. So I tried to >>> reproduce, and the behaviour is not really clear to me, I'll try >>> to clarify. >>> >>> # Start with MGRs >>> >>> reef1:~ # ceph orch upgrade start --image >>> quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1 >>> >>> Upgrading MGRs with limit works, but it doesn't reflect in the MGR >>> log. Usually, I expect a line like this: >>> >>> ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade >>> >>> But there is no such line in the logs. Then I upgrade the rest of the MGRs. >>> >>> # Continue with MONs >>> >>> reef1:~ # ceph orch upgrade start --image >>> quay.io/ceph/ceph:v19.2.3 --services mon --limit 1 >>> >>> This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. >>> And again, no such line in the log (Hit upgrade limit). What I >>> noticed was a MGR respawn after the first MON had been upgraded >>> successfully. Maybe some state of the upgrade progress gets lost >>> during the respawn? I then upgraded the remaining MON. Then >>> ceph-crash is upgraded successfully. >>> >>> # Upgrade OSD >>> >>> reef1:~ # ceph orch upgrade start --image >>> quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1 >>> >>> And this is the first service that actually reports the upgrade limit: >>> >>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO >>> cephadm.upgrade] Upgrade: Updating osd.0 (1/1) >>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Upgrade: Updating osd.0 (1/1) >>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] >>> Hit upgrade limit of 1. Stopping upgrade >>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Hit upgrade limit of 1. Stopping upgrade >>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO >>> cephadm.upgrade] Upgrade: Setting container_image for all nvmeof >>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Upgrade: Setting container_image for all nvmeof >>> 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO >>> cephadm.upgrade] Upgrade: Finalizing container_image settings >>> 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Upgrade: Finalizing container_image settings >>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO >>> cephadm.upgrade] Upgrade: Complete! >>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Upgrade: Complete! >>> >>> >>> This is really irritating and inconsistent: If the orchestrator >>> does honor --limit with other services than OSD, why isn't that >>> visible in the logs? And what's with the MONs? Why 2 out of 3? >>> >>> Any pointers appreciated! Not sure which Ceph versions might be >>> affected by this, I'll try out a couple more upgrade paths. >>> >>> Thanks, >>> Eugen >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

August Ceph Foundation Board Meeting
by Anthony Middleton 26 Aug '26

26 Aug '26

Hey Ceph friends, I hope you're doing well. Here is the recording of the August Ceph Foundation Board meeting: https://www.youtube.com/watch?v=e7xt4h37rSg. There are several committees community members can join, including Hardware, Ceph Days, Ceph Ambassadors, and Community Outreach. Please reach out to me if you're interested in connecting with any of the committee leaders. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: v20.2.4 staggered upgrade
by Jean-Marie de Boer 26 Aug '26

26 Aug '26

I have tested a staggered upgrade from 19.2.4 to 19.2.6. Indeed the OSD keys got rotated multiple times, I guess it may always happen when upgrading, even if it's a single daemon type that's later in the sequence. It's not really a problem but it does make everything take quite a bit longer (we have 1000+ OSD's on production clusters and must do a staggered upgrade because we need to manage the MDS failover). I think the devs are aware though. I do not observe this port multiplication however. mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 running (7h) 42s ago 5w 478M - 19.2.6 ff836bb28e7d 251c5d067daf mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 running (14m) 42s ago 5w 475M - 19.2.6 ff836bb28e7d 45c38fb90d31 mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 running (7h) 42s ago 5w 675M - 19.2.6 ff836bb28e7d 1d69113ae7ed However I have been observing some strangeness with daemons showing non-standard status: mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 running (6h) 3m ago 5w 478M - 19.2.6 ff836bb28e7d 251c5d067daf mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 unknown 8m ago 5w 339M - 19.2.6 ff836bb28e7d 98add61a7280 mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 running 0s ago 5w - - <unknown> <unknown> <unknown> A node failed to finish starting its containers without a clear reason why, it also refused to exit maintenance, and a mon on another node failed and fixed itself without intervention. I have not really determined a clear cause. Most of that fixed itself eventually although I now notice a single crash daemon playing up. Of course we're making it hard because we do a staggered upgrade and we use multiple routed subnets too :) Regards, JM On 8/26/26 12:21, Eugen Block wrote: > I haven't seen anything like this yet: > > mgr.reef1.zfjbzv reef1 *:9283,8765,8765,8765,8765,8765,8765 > running (2m) 109s ago 18h 129M - 20.2.4 > 6facee348180 ed9ae39b8f5f > mgr.reef2.omwexa reef2 *:8765,8765,8765,8765,8765,8765,8765 > running (2m) 108s ago 14h 167M - 20.2.4 > 6facee348180 b24464d5c98b > > I just did a staggered upgrade in a test cluster from 18.2.7 to 20.2.4 > and I noticed that it goes through redeploying all daemons with a new > keyring everytime although they already had been upgraded, for example > MGR, MON and OSD when I was trying to upgrade MDS only (--daemon-type MDS). > > A 'ceph mgr fail' does not clean up those redundant port entries. Should > I create a tracker for that or is this already known? > > I haven't tried the same on 19.2.6 yet. > > Regards, > Eugen > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Jean-Marie de Boer Specialist SURF | Science Park 140 | 1098 XG Amsterdam | www.surf.nl E: jean-marie.deboer(a)surf.nl | M: +31610961787

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 26 Aug '26

26 Aug '26

Hi, did you add the new pub key to all the host's authorized_keys? clear-key and then generate-key is the correct way. Zitat von Chris Palmer <ceph-users(a)ceph.io>: > I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and > now it is broken: > > The note below points towards the SSH Configuration section. That > doesn't have anything specific about rotation, so I had to make some > guesses. > > It says that a new SSH key can be generated with "ceph cephadm > generate-key". It doesn't give any error, but using "ceph cephadm > get-pub-key" suggests it hasn't actually done anything (I still get > the original ssh-rsa key). The doc probably needs clarifying. > > So I tried "ceph cephadm clear-key" which worked, and then "ceph > cephadm generate-key" again. This time I can see a new ssh-ed25519 > key. I then failed all MGRs. > > Now however I have this for every host: > > root@c1:~# ceph cephadm check-host c2 > check-host failed: > Failed to connect to c2 at address (192.168.nnn.162): Failed to > connect to c2 (192.168.nnn.162). Permission denied > Log: Opening SSH connection to 192.168.nnn.162, port 22 > [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 > [conn=9] Local address: 192.168.nnn.161, port 35514 > [conn=9] Peer address: 192.168.nnn.162, port 22 > [conn=9] Beginning auth for user root > [conn=9] Auth failed for user root > [conn=9] Connection failure: Permission denied > [conn=9] Aborting connection > > Have I done this incorrectly, or is the documentation wrong? What is > the correct way to rectify this? > > Thanks, Chris > > On 19/08/2026 17:07, Patrick Donnelly wrote: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: >> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 26 Aug '26

26 Aug '26

I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and now it is broken: The note below points towards the SSH Configuration section. That doesn't have anything specific about rotation, so I had to make some guesses. It says that a new SSH key can be generated with "ceph cephadm generate-key". It doesn't give any error, but using "ceph cephadm get-pub-key" suggests it hasn't actually done anything (I still get the original ssh-rsa key). The doc probably needs clarifying. So I tried "ceph cephadm clear-key" which worked, and then "ceph cephadm generate-key" again. This time I can see a new ssh-ed25519 key. I then failed all MGRs. Now however I have this for every host: root@c1:~# ceph cephadm check-host c2 check-host failed: Failed to connect to c2 at address (192.168.nnn.162): Failed to connect to c2 (192.168.nnn.162). Permission denied Log: Opening SSH connection to 192.168.nnn.162, port 22 [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 [conn=9] Local address: 192.168.nnn.161, port 35514 [conn=9] Peer address: 192.168.nnn.162, port 22 [conn=9] Beginning auth for user root [conn=9] Auth failed for user root [conn=9] Connection failure: Permission denied [conn=9] Aborting connection Have I done this incorrectly, or is the documentation wrong? What is the correct way to rectify this? Thanks, Chris On 19/08/2026 17:07, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0

v20.2.4 staggered upgrade
by Eugen Block 26 Aug '26

26 Aug '26

I haven't seen anything like this yet: mgr.reef1.zfjbzv reef1 *:9283,8765,8765,8765,8765,8765,8765 running (2m) 109s ago 18h 129M - 20.2.4 6facee348180 ed9ae39b8f5f mgr.reef2.omwexa reef2 *:8765,8765,8765,8765,8765,8765,8765 running (2m) 108s ago 14h 167M - 20.2.4 6facee348180 b24464d5c98b I just did a staggered upgrade in a test cluster from 18.2.7 to 20.2.4 and I noticed that it goes through redeploying all daemons with a new keyring everytime although they already had been upgraded, for example MGR, MON and OSD when I was trying to upgrade MDS only (--daemon-type MDS). A 'ceph mgr fail' does not clean up those redundant port entries. Should I create a tracker for that or is this already known? I haven't tried the same on 19.2.6 yet. Regards, Eugen

1 0

Re: Cephadm staggered upgrade --limit parameter not always applied
by Andreas Eriksson 26 Aug '26

26 Aug '26

Hi, Were you able to find anything more about this? I'm looking at upgrading a bunch of clusters from previous major versions, but do not want to upgrade all RGW services. And also limit the number of osds at the time that are being upgraded, to keep the recovery traffic lower for example. Br, On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote: > Adding the dev list > > Zitat von Eugen Block <eblock(a)nde.ag>: > >> Hi, >> >> I did a quick search on tracker but couldn't find anything related. A >> customer reported this, and I can confirm the behaviour on a lab >> cluster. I usually perform a staggered upgrade with --daemon-types >> and --limit, but not for all daemon types. So I haven't stumbled >> across this yet myself, but our customer did (I can reproduce with >> --daemon-types as well). They upgraded from latest Reef to latest >> Squid and reported that despite providing the --limit parameter, the >> mons were all upgraded. So I tried to reproduce, and the behaviour is >> not really clear to me, I'll try to clarify. >> >> # Start with MGRs >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 >> --services mgr --limit 1 >> >> Upgrading MGRs with limit works, but it doesn't reflect in the MGR >> log. Usually, I expect a line like this: >> >> ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade >> >> But there is no such line in the logs. Then I upgrade the rest of the >> MGRs. >> >> # Continue with MONs >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 >> --services mon --limit 1 >> >> This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. >> And again, no such line in the log (Hit upgrade limit). What I >> noticed was a MGR respawn after the first MON had been upgraded >> successfully. Maybe some state of the upgrade progress gets lost >> during the respawn? I then upgraded the remaining MON. Then >> ceph-crash is upgraded successfully. >> >> # Upgrade OSD >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 >> --services osd.osd.standalone --limit 1 >> >> And this is the first service that actually reports the upgrade limit: >> >> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating osd.0 (1/1) >> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating osd.0 (1/1) >> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] Hit >> upgrade limit of 1. Stopping upgrade >> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Hit upgrade limit of 1. Stopping upgrade >> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Setting container_image for all nvmeof >> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Upgrade: Setting container_image for all nvmeof >> 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Finalizing container_image settings >> 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Upgrade: Finalizing container_image settings >> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Complete! >> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Upgrade: Complete! >> >> >> This is really irritating and inconsistent: If the orchestrator does >> honor --limit with other services than OSD, why isn't that visible in >> the logs? And what's with the MONs? Why 2 out of 3? >> >> Any pointers appreciated! Not sure which Ceph versions might be >> affected by this, I'll try out a couple more upgrade paths. >> >> Thanks, >> Eugen > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Robert Sander 26 Aug '26

26 Aug '26

Hi, Am 26.08.26 um 8:44 AM schrieb Eugen Block: > according to the "OS Recommendations" [0] Ceph Tentacle is not available > for Debian 13. Umbrella will be built for Trixie. > Are you using cephadm managed clusters or package based? With cephadm > you're not that strictly tied to the installed package versions. In the end all systems need updated client libraries to handle the new key format. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 26 Aug '26

26 Aug '26

Hi, according to the "OS Recommendations" [0] Ceph Tentacle is not available for Debian 13. Umbrella will be built for Trixie. Are you using cephadm managed clusters or package based? With cephadm you're not that strictly tied to the installed package versions. Regards, Eugen [0] https://docs.ceph.com/en/latest/start/os-recommendations/ Zitat von Johan <ceph-users(a)ceph.io>: > Ok, > > on Debian 12 it was not a problem upgrading to ceph 20.2.4 using apt > and https://download.ceph.com/debian-tentacle/ > > With Debian 13 which I have at my troublesome cluster I get several > dependency-issues. > ceph-common is dependent on libicu72 which doesn't have any candidates > python3-cephfs (and others) is dependent on python3 < 3.12 but > 3.13.5-1 will be installed > etc > > Is this because I perhaps hadn't upgraded ceph-common and others > properly before upgrading Debian from 12 to 13? > > /Johan > > Den 2026-08-24 kl. 11:06, skrev Johan: >> Im not a professional computer tech but have two ceph clusters >> (cephadm, Trixie, Debian 12/13). >> >> I upgraded the least sensitive one using the dashboard. When I >> upgrade the admin client I now get >> auth: error parsing file /etc/ceph/ceph.client.admin.keyring: >> error setting modifier for [client.admin] type=key val=<key>: >> Malformed input [buffer:3] >> auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) >> Input/ output error >> >> I get the same respons when using a ordinary user to connect the filesystem. >> >> I found this issue describing the same error >> https://github.com/rook/rook/issues/18240 >> >> As they, in above issue, discussed which version/image of the >> toolbox was used I checked my version of ceph at the command line. >> I get >> ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) >> reef (stable) >> I would guess this is the problem. >> >> How do I get to run up-to-date version on the command line? >> >> /Johan >> >> >> Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> https://ceph.io/en/news/blog/ >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/ >>> index.html#upgrading-and-rotating-cephx-keys >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/ >>> index.html#auth-insecure-keys-creatable >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> https://docs.ceph.com/en/latest/architecture/index.html#high- >>> availability-authentication >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh- >>> configuration >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx- >>> keys-patrick-donnelly-ibm-yehuda-sadeh-weinraub-ubiquiti >>> >>> ## Getting Ceph >>> ------------ >>> * Source at https://github.com/ceph/ceph >>> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ ceph-19.2.6.tar.gz >>> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ ceph-20.2.4.tar.gz >>> * Containers at https://quay.io/repository/ceph/ceph >>> * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ >>> * For upgrade with cephadm, see: >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Insecure keys allowed (aes)
by Eugen Block 25 Aug '26

25 Aug '26

Hi, you can disable the old cipher by setting auth_allowed_ciphers to aes256k only: ceph mon set auth_allowed_ciphers aes256k See step 10 in the docs: https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… Regards, Eugen Zitat von "GLE, Vivien" <ceph-users(a)ceph.io>: > Hi, > > > I just upgrade (from tentacle 20.2.1 to 20.2.4) and rolled CephX > keys for a Cephadm cluster but there is still an error concerning > the use of insecure keys > > > #ceph config get mon mon_auth_allow_insecure_key > false > > > #ceph health detail > > HEALTH_WARN Monitors are configured to allow auth using insecure key types > [WRN] AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow > auth using insecure key types > insecure cipher aes allowed for auth > > > Didn't found any keys named like this in the ceph configuration but > in the ceph mon dump > > #ceph mon dump > > auth_epoch 0 > auth_service_cipher aes256k > auth_allowed_ciphers aes, aes256k <= = > auth_preferred_cipher aes256k > dumped monmap epoch 15 > > Is the only way to get rid of this error is to manipulate the mon > map ? How can I do it safely ? > > Thanks > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Insecure keys allowed (aes)
by Patrick Donnelly 25 Aug '26

25 Aug '26

Hello, On Tue, Aug 25, 2026 at 8:32 AM GLE, Vivien <ceph-users(a)ceph.io> wrote: > > Hi, > > > I just upgrade (from tentacle 20.2.1 to 20.2.4) and rolled CephX keys for a Cephadm cluster but there is still an error concerning the use of insecure keys > > > #ceph config get mon mon_auth_allow_insecure_key > false > > > #ceph health detail > > HEALTH_WARN Monitors are configured to allow auth using insecure key types > [WRN] AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types > insecure cipher aes allowed for auth > > > Didn't found any keys named like this in the ceph configuration but in the ceph mon dump Please read the documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks/#auth-insecu… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 25 Aug '26

25 Aug '26

Yes, unmanaged OSDs are of course also rotated (excerpt from an upgrade of unmanaged OSDs): 2026-08-25T12:35:03.728+0000 7f9d451b7640 0 [cephadm INFO cephadm.upgrade] Redeploying osd.5 with new keyring The unmanaged flag is basically to prevent automatic deployment of OSDs only. Zitat von Chris Palmer <ceph-users(a)ceph.io>: > Can anyone confirm whether OSDs that were adopted into cephadm but > remain unmanaged will be automatically rotated? (Currently on 20.2.3). > > On 20/08/2026 14:45, Adam King wrote: >> That command got removed for the time being as it used an old way of doing >> the key rotation that wasn't built to work with the changes for the cve and >> seemed to be broken for a number of daemon types. Since the name would make >> people want to use it it was decided to just temporarily disables it so we >> can fix it and reintroduce it in a new version as an easier way to rotate >> the keys. For any keys tied to a cephadm deployed daemon that aren't >> automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings >> during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type >> aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using >> nfs exports managed through the nfs module the export's keyrings also need >> to be rotated. `ceph auth rotate <export-keyring-entity> --key-type >> aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > >> export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. >> Between rotating the keyrings for the nfs cluster/daemons and rotating the >> keyring for the individual export(s) those export(s) are unavailable iirc. >> >> On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: >> >>> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: >>>> Hi, >>>> >>>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The >>> following cephadm managed client keys were not rotated automatically: >>>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) >>>> client.ceph-exporter. >>>> client.crash. >>>> client.osd-lockbox.. >>>> client.rgw. >>>> >>>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* >>> this command does not exist. >>> >>> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing >>> there, but present on 20.2.3. >>> >>> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… >>> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… >>>> What is the correct way to rotate these client keys via cephadm as part >>> of the cve fix? >>>> Thanks for any guidance. >>>> Best Regards >>>> Max >>>> >>>> ________________________________ >>>> From: Patrick Donnelly <ceph-users(a)ceph.io> >>>> Sent: 19 August 2026 18:08 >>>> To: ceph-users <ceph-users(a)ceph.io> >>>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 >>> released >>>> Warning: This email is from an unusual correspondent. >>>> Warning: Make sure this is someone you trust. >>>> >>>> Today, August 19th, 2026, the Ceph project has released Tentacle >>>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>>> components. >>>> >>>> This is the sixth backport release in the Squid series. >>>> >>>> This is the fourth backport release in the Tentacle series. >>>> >>>> Additional details will be posted on the Ceph website/blog: >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… >>>> We strongly recommend that all Ceph operators upgrade to one of these >>>> releases as soon as possible. >>>> >>>> >>>> ## Critical Upgrade Steps >>>> >>>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>>> This is the first time Ceph has introduced a new key type for CephX >>>> credentials. Therefore, a new procedure exists for upgrading and >>>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>>> cluster, the operator should familiarize themselves with the procedure >>>> documented at: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> These steps are for package-based deployments of Ceph. In other >>>> sections, the document also includes new information about key >>>> rotation procedures, monitor authentication controls, and emergency >>>> recovery procedures. >>>> >>>> Deployments using cephadm will automate the process except for client >>>> keys. You may notice that Cephadm spends more time than normal on the >>>> upgrade after updating all daemon images. This is due to a new process >>>> rotating the OSD and MDS keys. Please be patient. There are plans to >>>> improve this in the future. >>>> >>>> Deployments using Rook will also automate rotation of some client keys >>>> with some exclusions. Look to the Rook project’s announcement for more >>>> details. >>>> >>>> Client and kernel upgrades are recommended to support aes256k. Note >>>> that upstream Linux kernel client support began in kernel 7.0 and has >>>> been backported to CentOS Stream 9 and 10. Check with your >>>> distribution vendor for backported key support before rotating client >>>> keys used by the kernel. >>>> >>>> As part of upgrading a Ceph cluster, six new health warnings and >>>> errors will be generated. This is normal. As part of the process to >>>> upgrade/rotate entity keys, you will address the warnings and errors >>>> as you go. These are documented beginning here: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> The CephX documentation has been brought up to date and vetted for >>>> accuracy. Those changes are live here: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>>> requests with `host` and `x-amz-` headers not included in the signed >>>> subset. Unfortunately, the REST client used in multisite was >>>> generating such improperly signed requests. >>>> >>>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>>> option to true before you begin to upgrade. After all clusters are >>>> upgraded, set the option to ``false`` again. >>>> >>>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>>> secrets stored in the Monitor config-key store will be forthcoming. In >>>> the meantime, operators of cephadm-managed clusters are strongly >>>> encouraged to rotate the cephadm SSH key, as an established process >>>> exists: >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> Rotating other secrets is advised but the process is not yet >>>> established. Operators should assess their cluster's potential >>>> exposure to determine if immediate Monitor config-key secret rotation >>>> is required for their environment. >>>> >>>> >>>> ## CVE Fixes >>>> >>>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>>> of AES-CBC. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>>> in improper verification of a cryptographic signature. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>>> subscription handler. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>>> cryptographic signatures in RGW's SigV4 verifier. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> ## Other Media >>>> >>>> A Cephalocon 2025 talk covers some of the details surrounding the new >>>> CephX key upgrade process. >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… >>>> ## Getting Ceph >>>> ------------ >>>> * Source at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… >>>> * v19.2.6 Tarball at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>>> * v20.2.4 Tarball at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>>> * Containers at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… >>>> * For packages, see >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * For upgrade with cephadm, see: >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>>> >>>> -- >>>> Patrick Donnelly, Ph.D. >>>> He / Him / His >>>> Red Hat Partner Engineer >>>> IBM, Inc. >>>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> -- >>>> This message has been checked by Libraesva ESG and is found to be clean. >>>> Follow this link to report as spam/bad: >>>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad >>>> Follow this link to blocklist sender: >>>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Insecure keys allowed (aes)
by GLE, Vivien 25 Aug '26

25 Aug '26

Hi, I just upgrade (from tentacle 20.2.1 to 20.2.4) and rolled CephX keys for a Cephadm cluster but there is still an error concerning the use of insecure keys #ceph config get mon mon_auth_allow_insecure_key false #ceph health detail HEALTH_WARN Monitors are configured to allow auth using insecure key types [WRN] AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types insecure cipher aes allowed for auth Didn't found any keys named like this in the ceph configuration but in the ceph mon dump #ceph mon dump auth_epoch 0 auth_service_cipher aes256k auth_allowed_ciphers aes, aes256k <= = auth_preferred_cipher aes256k dumped monmap epoch 15 Is the only way to get rid of this error is to manipulate the mon map ? How can I do it safely ? Thanks

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 25 Aug '26

25 Aug '26

Hi, you're correct, the option doesn't exist in older clusters (I only checked one Reef and one Squid version < 19.2.6). I assume it's recommended to perform a staggered upgrade [0], and before you upgrade the RGWs, you set the config option. I guess that could have been made a bit clearer in the docs. The staggered upgrade works in this test cluster (I don't have multisite enabled): reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon-types mgr,mon,crash,osd,mds wait a bit... reef1:~ # ceph versions { "mon": { "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 3 }, "mgr": { "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 2 }, "osd": { "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 5 }, "rgw": { "ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable)": 2 }, "overall": { "ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable)": 2, "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 10 } } reef1:~ # ceph config set client.rgw rgw_sigv4_insecure true reef1:~ # reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon-types rgw And then upgrade the rest (omit daemon-types). Hope this helps! Eugen [0] https://docs.ceph.com/en/latest/cephadm/upgrade/#staggered-upgrade Zitat von Matthew Vernon <ceph-users(a)ceph.io>: > Hi, > > On 19/08/2026 17:09, Patrick Donnelly wrote: > >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. > > Stupid question - how/where does one set this, and will this be > understood by older versions of Ceph? > > I ask because I have a Reef multisite setup, which should be > supported to upgrade to Tentacle. Which I fear I'm now going to have > to do in a hurry! > > AFAICT this is a new config setting, so I'd expect older versions of > Ceph won't understand it... [and searching the reef docs doesn't > find it] > > Thanks, > > Matthew > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Boris 25 Aug '26

25 Aug '26

Hi Matthew, we've set it with: ceph config set client.rgw rgw_sigv4_insecure true --force Best Boris Am Di., 25. Aug. 2026 um 09:41 Uhr schrieb Matthew Vernon <ceph-users(a)ceph.io>: > > Hi, > > On 19/08/2026 17:09, Patrick Donnelly wrote: > > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > > option to true before you begin to upgrade. After all clusters are > > upgraded, set the option to ``false`` again. > > Stupid question - how/where does one set this, and will this be > understood by older versions of Ceph? > > I ask because I have a Reef multisite setup, which should be supported > to upgrade to Tentacle. Which I fear I'm now going to have to do in a hurry! > > AFAICT this is a new config setting, so I'd expect older versions of > Ceph won't understand it... [and searching the reef docs doesn't find it] > > Thanks, > > Matthew > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Matthew Vernon 25 Aug '26

25 Aug '26

Hi, On 19/08/2026 17:09, Patrick Donnelly wrote: > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. Stupid question - how/where does one set this, and will this be understood by older versions of Ceph? I ask because I have a Reef multisite setup, which should be supported to upgrade to Tentacle. Which I fear I'm now going to have to do in a hurry! AFAICT this is a new config setting, so I'd expect older versions of Ceph won't understand it... [and searching the reef docs doesn't find it] Thanks, Matthew

1 0

Re: Crush OSD mapping updates when OSD marked out
by Sam Goyal 25 Aug '26

25 Aug '26

> So clearly some modification to the crush map is needed or some post processing of the crush mappings are in order to account for OSDs marked out. Someone can correct me if I am wrong, but after a quick look at the code, the crushmap is not touched when an OSD is marked out. The new weight is stored in the OSDMap. When the new epoch of this map is sent to the OSDs, CRUSH calculates the new PG mappings, skipping the OSD with weight 0 (passed down from the new map). The new mappings differ from the old ones in this case, which triggers peering. > I.e. how to simulate a reweight (to 0 for being out, for any other number other than 1.0 - same problem). I.e. I have a crush map, I'd like to understand how to determine which PGs get remapped to what OSDs when I reweight an OSD from 1.0 to something lower. I think you can use osdmaptool with --test-map-pgs-dump" and "--mark-out" flag to simulate an out OSD. ( https://docs.ceph.com/en/latest/man/8/osdmaptool/#cmdoption-osdmaptool-mark… ) I don't think just the crushmap alone can achieve this. I'll let someone else confirm this. You can also simulate changing the crush weights using the "--adjust-crush-weight" flag. Thanks, Sam On Mon, 24 Aug 2026 at 15:33, Andras Pataki <ceph-users(a)ceph.io> wrote: > I'm trying to understand how 'reweighting' the OSD affects the placement > of the PGs. I.e. how to simulate a reweight (to 0 for being out, for > any other number other than 1.0 - same problem). > I.e. I have a crush map, I'd like to understand how to determine which > PGs get remapped to what OSDs when I reweight an OSD from 1.0 to > something lower. > > Andras > > > On 8/24/26 5:47 PM, Joshua Blanch wrote: > >> The "reweight" field that is set from 1.0 to 0.0 > > Have you tried reweighting to a non-zero, near-zero crush weight, e.g., > > 0.0001? That should leave it in the crush map and simulate it being > 'out'. > > > > - Joshua > > > > On Mon, 24 Aug 2026 at 14:12, Andras Pataki <ceph-users(a)ceph.io> wrote: > > > >> Hi friends of ceph, > >> > >> I'd like to better understand how ceph moves PGs off of OSDs that are > >> marked out for the purposes of simulating potential setup changes. > >> When all OSDs are in (and no upmaps are present), I can use crushtool to > >> find the PG to OSDs mappings of all PGs in the cluster given a crush map > >> (for example: "crushtool -i crushmap.compiled --test --show-mappings > >> --num-rep 3 --min-x 0 --max-x 511 --rule 3 --pool-id 3" would do this > >> for one of our pools). If I change any weights in the crush map, I can > >> recompile it, rerun crushtool to get an updated mapping, and I have > >> found this to perfectly match what ceph does when the crush map is > >> loaded into the cluster. > >> > >> What I am a bit unsure about is how to get the PG to OSD mappings when > >> an OSD is marked out. The "reweight" field that is set from 1.0 to 0.0 > >> when an OSD is marked out is not in the crush map, it is in the osdmap > >> only. So clearly some modification to the crush map is needed or some > >> post processing of the crush mappings are in order to account for OSDs > >> marked out. I originally thought that changing the OSD weight to 0.0 in > >> the crush map without modifying the parent bucket weights would do the > >> trick. While this approach does remap the correct PGs (the ones that > >> originally contain the OSD that is marked out), the new mappings are not > >> the right ones (i.e. the new mappings do not match what live ceph does). > >> > >> So my question if someone can enlighten me or point me to the right > >> direction: starting with a crush map, what do I need to do to simulate > >> the mappings for an OSD that is marked out? I.e. what changes are > >> needed beyond what crushtool --show-mappings would give and how to > >> calculate those offline (without touching a live cluster)? I've tried > >> looking at the code (OSDMap.cc) - but couldn't quite understand where > >> the adjustments are made for OSDs marked out. Any > >> pointers/hints/insights would be appreciated. > >> > >> Thanks, > >> > >> Andras > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Crush OSD mapping updates when OSD marked out
by Andras Pataki 24 Aug '26

24 Aug '26

I'm trying to understand how 'reweighting' the OSD affects the placement of the PGs. I.e. how to simulate a reweight (to 0 for being out, for any other number other than 1.0 - same problem). I.e. I have a crush map, I'd like to understand how to determine which PGs get remapped to what OSDs when I reweight an OSD from 1.0 to something lower. Andras On 8/24/26 5:47 PM, Joshua Blanch wrote: >> The "reweight" field that is set from 1.0 to 0.0 > Have you tried reweighting to a non-zero, near-zero crush weight, e.g., > 0.0001? That should leave it in the crush map and simulate it being 'out'. > > - Joshua > > On Mon, 24 Aug 2026 at 14:12, Andras Pataki <ceph-users(a)ceph.io> wrote: > >> Hi friends of ceph, >> >> I'd like to better understand how ceph moves PGs off of OSDs that are >> marked out for the purposes of simulating potential setup changes. >> When all OSDs are in (and no upmaps are present), I can use crushtool to >> find the PG to OSDs mappings of all PGs in the cluster given a crush map >> (for example: "crushtool -i crushmap.compiled --test --show-mappings >> --num-rep 3 --min-x 0 --max-x 511 --rule 3 --pool-id 3" would do this >> for one of our pools). If I change any weights in the crush map, I can >> recompile it, rerun crushtool to get an updated mapping, and I have >> found this to perfectly match what ceph does when the crush map is >> loaded into the cluster. >> >> What I am a bit unsure about is how to get the PG to OSD mappings when >> an OSD is marked out. The "reweight" field that is set from 1.0 to 0.0 >> when an OSD is marked out is not in the crush map, it is in the osdmap >> only. So clearly some modification to the crush map is needed or some >> post processing of the crush mappings are in order to account for OSDs >> marked out. I originally thought that changing the OSD weight to 0.0 in >> the crush map without modifying the parent bucket weights would do the >> trick. While this approach does remap the correct PGs (the ones that >> originally contain the OSD that is marked out), the new mappings are not >> the right ones (i.e. the new mappings do not match what live ceph does). >> >> So my question if someone can enlighten me or point me to the right >> direction: starting with a crush map, what do I need to do to simulate >> the mappings for an OSD that is marked out? I.e. what changes are >> needed beyond what crushtool --show-mappings would give and how to >> calculate those offline (without touching a live cluster)? I've tried >> looking at the code (OSDMap.cc) - but couldn't quite understand where >> the adjustments are made for OSDs marked out. Any >> pointers/hints/insights would be appreciated. >> >> Thanks, >> >> Andras >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Crush OSD mapping updates when OSD marked out
by Joshua Blanch 24 Aug '26

24 Aug '26

> The "reweight" field that is set from 1.0 to 0.0 Have you tried reweighting to a non-zero, near-zero crush weight, e.g., 0.0001? That should leave it in the crush map and simulate it being 'out'. - Joshua On Mon, 24 Aug 2026 at 14:12, Andras Pataki <ceph-users(a)ceph.io> wrote: > Hi friends of ceph, > > I'd like to better understand how ceph moves PGs off of OSDs that are > marked out for the purposes of simulating potential setup changes. > When all OSDs are in (and no upmaps are present), I can use crushtool to > find the PG to OSDs mappings of all PGs in the cluster given a crush map > (for example: "crushtool -i crushmap.compiled --test --show-mappings > --num-rep 3 --min-x 0 --max-x 511 --rule 3 --pool-id 3" would do this > for one of our pools). If I change any weights in the crush map, I can > recompile it, rerun crushtool to get an updated mapping, and I have > found this to perfectly match what ceph does when the crush map is > loaded into the cluster. > > What I am a bit unsure about is how to get the PG to OSD mappings when > an OSD is marked out. The "reweight" field that is set from 1.0 to 0.0 > when an OSD is marked out is not in the crush map, it is in the osdmap > only. So clearly some modification to the crush map is needed or some > post processing of the crush mappings are in order to account for OSDs > marked out. I originally thought that changing the OSD weight to 0.0 in > the crush map without modifying the parent bucket weights would do the > trick. While this approach does remap the correct PGs (the ones that > originally contain the OSD that is marked out), the new mappings are not > the right ones (i.e. the new mappings do not match what live ceph does). > > So my question if someone can enlighten me or point me to the right > direction: starting with a crush map, what do I need to do to simulate > the mappings for an OSD that is marked out? I.e. what changes are > needed beyond what crushtool --show-mappings would give and how to > calculate those offline (without touching a live cluster)? I've tried > looking at the code (OSDMap.cc) - but couldn't quite understand where > the adjustments are made for OSDs marked out. Any > pointers/hints/insights would be appreciated. > > Thanks, > > Andras > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Crush OSD mapping updates when OSD marked out
by Andras Pataki 24 Aug '26

24 Aug '26

Hi friends of ceph, I'd like to better understand how ceph moves PGs off of OSDs that are marked out for the purposes of simulating potential setup changes. When all OSDs are in (and no upmaps are present), I can use crushtool to find the PG to OSDs mappings of all PGs in the cluster given a crush map (for example: "crushtool -i crushmap.compiled --test --show-mappings --num-rep 3 --min-x 0 --max-x 511 --rule 3 --pool-id 3" would do this for one of our pools). If I change any weights in the crush map, I can recompile it, rerun crushtool to get an updated mapping, and I have found this to perfectly match what ceph does when the crush map is loaded into the cluster. What I am a bit unsure about is how to get the PG to OSD mappings when an OSD is marked out. The "reweight" field that is set from 1.0 to 0.0 when an OSD is marked out is not in the crush map, it is in the osdmap only. So clearly some modification to the crush map is needed or some post processing of the crush mappings are in order to account for OSDs marked out. I originally thought that changing the OSD weight to 0.0 in the crush map without modifying the parent bucket weights would do the trick. While this approach does remap the correct PGs (the ones that originally contain the OSD that is marked out), the new mappings are not the right ones (i.e. the new mappings do not match what live ceph does). So my question if someone can enlighten me or point me to the right direction: starting with a crush map, what do I need to do to simulate the mappings for an OSD that is marked out? I.e. what changes are needed beyond what crushtool --show-mappings would give and how to calculate those offline (without touching a live cluster)? I've tried looking at the code (OSDMap.cc) - but couldn't quite understand where the adjustments are made for OSDs marked out. Any pointers/hints/insights would be appreciated. Thanks, Andras

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Casey Bodley 24 Aug '26

24 Aug '26

thank you Peter, i've opened https://tracker.ceph.com/issues/79803 to track this regression On Mon, Aug 24, 2026 at 10:42 AM Peter Linder <ceph-users(a)ceph.io> wrote: > > Yep. > > This specifies that x-amz-content-sha256 needs to be included, but this > is the general documentation: > > https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-sign… > > > For s3, there is an exception however: > > https://docs.aws.amazon.com/AmazonS3/latest/developerguide/sig-v4-header-ba… > > So I believe this is a bug, 20.2.4 is more strict that it needs to be. > > There might be other considerations, but rgw_auth_s3.cc around like line > 800 or so could have an exception for this header. > > /Peter > > > > > On 8/24/26 4:12 PM, Omar Siam wrote: > > I think so. > > > > There is a flag mentioned in the upgrade notes that needs to be set to > > go back to the less secure protocol: > > > > "rgw_sigv4_insecure": Allow potential privilege escalation with added > > `x-amz-` headers on Sigv4. > > > > I had to set this to true for my S3 clients (velero) to work again. > > > > Am 24.08.2026 um 15:41 schrieb Peter Linder: > >> External Email: please use links and attachments from trusted sources > >> only > >> > >> Does anyone know if 20.2.4 requires x-amz-content-sha256 to be a part > >> of the signed headers now, unlike amazon that still leaves it optional? > >> > >> /Peter > >> > >> > >> On 8/19/26 6:08 PM, Patrick Donnelly wrote: > >>> Today, August 19th, 2026, the Ceph project has released Tentacle > >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > >>> components. > >>> > >>> This is the sixth backport release in the Squid series. > >>> > >>> This is the fourth backport release in the Tentacle series. > >>> > >>> Additional details will be posted on the Ceph website/blog: > >>> https://ceph.io/en/news/blog/ > >>> > >>> We strongly recommend that all Ceph operators upgrade to one of these > >>> releases as soon as possible. > >>> > >>> > >>> ## Critical Upgrade Steps > >>> > >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > >>> This is the first time Ceph has introduced a new key type for CephX > >>> credentials. Therefore, a new procedure exists for upgrading and > >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > >>> cluster, the operator should familiarize themselves with the procedure > >>> documented at: > >>> > >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > >>> > >>> > >>> These steps are for package-based deployments of Ceph. In other > >>> sections, the document also includes new information about key > >>> rotation procedures, monitor authentication controls, and emergency > >>> recovery procedures. > >>> > >>> Deployments using cephadm will automate the process except for client > >>> keys. You may notice that Cephadm spends more time than normal on the > >>> upgrade after updating all daemon images. This is due to a new process > >>> rotating the OSD and MDS keys. Please be patient. There are plans to > >>> improve this in the future. > >>> > >>> Deployments using Rook will also automate rotation of some client keys > >>> with some exclusions. Look to the Rook project’s announcement for more > >>> details. > >>> > >>> Client and kernel upgrades are recommended to support aes256k. Note > >>> that upstream Linux kernel client support began in kernel 7.0 and has > >>> been backported to CentOS Stream 9 and 10. Check with your > >>> distribution vendor for backported key support before rotating client > >>> keys used by the kernel. > >>> > >>> As part of upgrading a Ceph cluster, six new health warnings and > >>> errors will be generated. This is normal. As part of the process to > >>> upgrade/rotate entity keys, you will address the warnings and errors > >>> as you go. These are documented beginning here: > >>> > >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > >>> > >>> > >>> The CephX documentation has been brought up to date and vetted for > >>> accuracy. Those changes are live here: > >>> > >>> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > >>> > >>> > >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > >>> requests with `host` and `x-amz-` headers not included in the signed > >>> subset. Unfortunately, the REST client used in multisite was > >>> generating such improperly signed requests. > >>> > >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` > >>> option to true before you begin to upgrade. After all clusters are > >>> upgraded, set the option to ``false`` again. > >>> > >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all > >>> secrets stored in the Monitor config-key store will be forthcoming. In > >>> the meantime, operators of cephadm-managed clusters are strongly > >>> encouraged to rotate the cephadm SSH key, as an established process > >>> exists: > >>> > >>> See: > >>> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > >>> > >>> Rotating other secrets is advised but the process is not yet > >>> established. Operators should assess their cluster's potential > >>> exposure to determine if immediate Monitor config-key secret rotation > >>> is required for their environment. > >>> > >>> > >>> ## CVE Fixes > >>> > >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > >>> of AES-CBC. > >>> > >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > >>> > >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of > >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting > >>> in improper verification of a cryptographic signature. > >>> > >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > >>> > >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > >>> subscription handler. > >>> > >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > >>> > >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > >>> cryptographic signatures in RGW's SigV4 verifier. > >>> > >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > >>> > >>> ## Other Media > >>> > >>> A Cephalocon 2025 talk covers some of the details surrounding the new > >>> CephX key upgrade process. > >>> > >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > >>> > >>> > >>> ## Getting Ceph > >>> ------------ > >>> * Source at https://github.com/ceph/ceph > >>> * v19.2.6 Tarball at > >>> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > >>> * v20.2.4 Tarball at > >>> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > >>> * Containers at https://quay.io/repository/ceph/ceph > >>> * For packages, see > >>> https://docs.ceph.com/en/latest/install/get-packages/ > >>> * For upgrade with cephadm, see: > >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ > >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > >>> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Peter Linder 24 Aug '26

24 Aug '26

Yep. This specifies that x-amz-content-sha256 needs to be included, but this is the general documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-sign… For s3, there is an exception however: https://docs.aws.amazon.com/AmazonS3/latest/developerguide/sig-v4-header-ba… So I believe this is a bug, 20.2.4 is more strict that it needs to be. There might be other considerations, but rgw_auth_s3.cc around like line 800 or so could have an exception for this header. /Peter On 8/24/26 4:12 PM, Omar Siam wrote: > I think so. > > There is a flag mentioned in the upgrade notes that needs to be set to > go back to the less secure protocol: > > "rgw_sigv4_insecure": Allow potential privilege escalation with added > `x-amz-` headers on Sigv4. > > I had to set this to true for my S3 clients (velero) to work again. > > Am 24.08.2026 um 15:41 schrieb Peter Linder: >> External Email: please use links and attachments from trusted sources >> only >> >> Does anyone know if 20.2.4 requires x-amz-content-sha256 to be a part >> of the signed headers now, unlike amazon that still leaves it optional? >> >> /Peter >> >> >> On 8/19/26 6:08 PM, Patrick Donnelly wrote: >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> https://ceph.io/en/news/blog/ >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >>> >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >>> >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >>> >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >>> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >>> >>> >>> ## Getting Ceph >>> ------------ >>> * Source at https://github.com/ceph/ceph >>> * v19.2.6 Tarball at >>> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >>> * v20.2.4 Tarball at >>> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >>> * Containers at https://quay.io/repository/ceph/ceph >>> * For packages, see >>> https://docs.ceph.com/en/latest/install/get-packages/ >>> * For upgrade with cephadm, see: >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Omar Siam 24 Aug '26

24 Aug '26

I think so. There is a flag mentioned in the upgrade notes that needs to be set to go back to the less secure protocol: "rgw_sigv4_insecure": Allow potential privilege escalation with added `x-amz-` headers on Sigv4. I had to set this to true for my S3 clients (velero) to work again. Am 24.08.2026 um 15:41 schrieb Peter Linder: > External Email: please use links and attachments from trusted sources > only > > Does anyone know if 20.2.4 requires x-amz-content-sha256 to be a part > of the signed headers now, unlike amazon that still leaves it optional? > > /Peter > > > On 8/19/26 6:08 PM, Patrick Donnelly wrote: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >> >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >> >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >> >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: >> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >> >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at >> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at >> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see >> https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

1 0

Re: Ceph Tentacle 20.2.4 Noble support
by Patrick Donnelly 24 Aug '26

24 Aug '26

On Mon, Aug 24, 2026 at 2:34 AM Marek Szuba <ceph-users(a)ceph.io> wrote: > > Dear Ceph maintainers, > > Further to the thread about 20.2.3 Noble support earlier this month, I > would like to point out that with there having been Noble packages for > up to and including 20.2.0, Noble users are now stuck with Ceph versions > with known security vulnerabilities. For many operators, ourselves > included, a downgrade to Jammy is not possible even it it were to be > done on Ceph servers themselves (as I mentioned in passing in one of my > messages to this list we experienced severe I/O issues with Jammy-backed > Ceph OSDs in one of our clusters, whereas another requires Noble or > newer in order for Podman to be new enough for SMB containers). > > I would therefore STRONGLY encourage you to resume the building and > publishing of packages for Noble ASAP irrespective of whether this > particular version of Ubuntu has reached the "officially supported" > status or not. Noble support will be present in v20.2.5. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Peter Linder 24 Aug '26

24 Aug '26

Does anyone know if 20.2.4 requires x-amz-content-sha256 to be a part of the signed headers now, unlike amazon that still leaves it optional? /Peter On 8/19/26 6:08 PM, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 24 Aug '26

24 Aug '26

Hi Eugen Ah yes, of course. You are absolutely right. I missed those when I was cleaning up obsolete keys. Deleting them has removed the error. Thanks for that lightning-fast help! Regards, Chris On 24/08/2026 12:16, Eugen Block wrote: > Hi Chris, > >> "detail": [ >> { >> "message": "entity mds.c1 using insecure key type: aes" >> }, >> { >> "message": "entity mds.c2 using insecure key type: aes" >> } > > those two auth entries are most likely from before your cephadm > upgrade, assuming you adopted this cluster with cephadm. They are most > likely not in use by any daemons as you can see in your current daemon > list: > >> Deploying daemon mds.cephfs.c1.secoxv on c1 >> Deploying daemon mds.cephfs.c2.lpzncl on c2 > > Before cephadm, MDS daemons didn't have any suffixes, so those must be > your old MDS daemons from before cephadm- > > I would recommend to double check, but you can probably just delete > those two old auth entries: > > ceph auth del mds.c1 > ceph auth del mds.c2 > > Just to be safe you can export those keyrings as a backup in case you > actually do experience trouble after deleting those entries. > > Regards, > Eugen > > Zitat von Chris Palmer <ceph-users(a)ceph.io>: > >> I'm upgrading a test cluster (cephadm, 20.2.3 -> 20.2.4). The upgrade >> completed successfully, but the MDS processes (1 active, 1 standby) >> still show as using the aes service key >> (AUTH_INSECURE_SERVICE_KEY_TYPE). I have manually issued a reconfig >> and restarted each daemon but no difference. >> >> Output of "ceph -W cephadm" (with the probably irrelevant stuff >> snipped out): >> >> 2026-08-24T11:03:16.341923+0100 mgr.c3.lythrw [INF] Upgrade: Started >> with target quay.io/ceph/ceph:v20.2.4 >> 2026-08-24T11:03:17.171731+0100 mgr.c3.lythrw [INF] Upgrade: First >> pull of quay.io/ceph/ceph:v20.2.4 >> 2026-08-24T11:03:35.480918+0100 mgr.c3.lythrw [INF] Upgrade: Target >> is version 20.2.4 (tentacle) >> 2026-08-24T11:03:35.480946+0100 mgr.c3.lythrw [INF] Upgrade: Target >> container is >> quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8, >> digests >> ['quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8', >> 'quay.io/ceph/ceph@sha256:6bb1c8a42fbc0bf87938946990b65174466997bc11c31eb5a323225a779fd8f9'] >> >> 2026-08-24T11:03:35.482902+0100 mgr.c3.lythrw [INF] Upgrade: Need to >> upgrade myself (mgr.c3.lythrw) >> 2026-08-24T11:03:36.195045+0100 mgr.c3.lythrw [INF] Upgrade: Updating >> mgr.c1.hsiqod >> 2026-08-24T11:03:36.250599+0100 mgr.c3.lythrw [INF] Deploying daemon >> mgr.c1.hsiqod on c1 >> [snip] >> >> 2026-08-24T11:08:16.388893+0100 mgr.c1.hsiqod [INF] Upgrade: It >> appears safe to stop mds.cephfs.c1.secoxv >> 2026-08-24T11:08:17.180069+0100 mgr.c1.hsiqod [INF] Upgrade: Updating >> mds.cephfs.c1.secoxv >> 2026-08-24T11:08:17.217174+0100 mgr.c1.hsiqod [INF] Deploying daemon >> mds.cephfs.c1.secoxv on c1 >> 2026-08-24T11:08:24.528567+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mds.cephfs.c2.lpzncl (monmap changed)... >> 2026-08-24T11:08:24.534395+0100 mgr.c1.hsiqod [INF] Upgrade: It >> appears safe to stop mds.cephfs.c2.lpzncl >> 2026-08-24T11:08:25.420651+0100 mgr.c1.hsiqod [INF] Upgrade: Updating >> mds.cephfs.c2.lpzncl >> 2026-08-24T11:08:25.486295+0100 mgr.c1.hsiqod [INF] Deploying daemon >> mds.cephfs.c2.lpzncl on c2 >> [snip] >> >> 2026-08-24T11:11:16.759649+0100 mgr.c1.hsiqod [INF] Rotating keyring >> for mds.cephfs.c1.secoxv >> 2026-08-24T11:11:16.786643+0100 mgr.c1.hsiqod [INF] Redeploying >> mds.cephfs.c1.secoxv with new keyring >> 2026-08-24T11:11:16.787629+0100 mgr.c1.hsiqod [INF] Deploying daemon >> mds.cephfs.c1.secoxv on c1 >> 2026-08-24T11:11:26.030560+0100 mgr.c1.hsiqod [INF] Rotating keyring >> for mds.cephfs.c2.lpzncl >> 2026-08-24T11:11:26.076154+0100 mgr.c1.hsiqod [INF] Redeploying >> mds.cephfs.c2.lpzncl with new keyring >> 2026-08-24T11:11:26.076957+0100 mgr.c1.hsiqod [INF] Deploying daemon >> mds.cephfs.c2.lpzncl on c2 >> 2026-08-24T11:11:35.278324+0100 mgr.c1.hsiqod [INF] Upgrade: checking >> if all mon, mgr, OSD, mds daemons upgraded before changing service >> cipher >> 2026-08-24T11:11:35.447319+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_KEYS_ALLOWED warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:36.529431+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_KEYS_CREATABLE warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:37.580675+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_SERVICE_TICKETS warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:38.673337+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_CLIENT_KEY_TYPE warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:39.743724+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_SERVICE_KEY_TYPE warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:40.873968+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning as upgrade is >> completed or has been stopped >> 2026-08-24T11:11:41.937998+0100 mgr.c1.hsiqod [INF] Upgrade: Complete! >> 2026-08-24T11:11:44.155457+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c1 (monmap changed)... >> 2026-08-24T11:11:44.155490+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c1 (monmap changed)... >> 2026-08-24T11:11:44.156559+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mon.c1 on c1 >> 2026-08-24T11:11:45.052365+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mgr.c1.hsiqod (monmap changed)... >> 2026-08-24T11:11:45.052389+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mgr.c1.hsiqod (monmap changed)... >> 2026-08-24T11:11:45.053849+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mgr.c1.hsiqod on c1 >> 2026-08-24T11:11:45.886112+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c2 (monmap changed)... >> 2026-08-24T11:11:45.886156+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c2 (monmap changed)... >> 2026-08-24T11:11:45.887368+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mon.c2 on c2 >> 2026-08-24T11:11:46.725430+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c3 (monmap changed)... >> 2026-08-24T11:11:46.725461+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c3 (monmap changed)... >> 2026-08-24T11:11:46.726439+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mon.c3 on c3 >> >> >> At this point nothing more happens. But: >> >> root@c3:~# ceph health detail --format=json|jq . >> { >> "status": "HEALTH_ERR", >> "checks": { >> "AUTH_INSECURE_CLIENT_KEY_TYPE": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "19 auth client entities with insecure key types", >> "count": 19 >> }, >> "detail": [ >> { >> "message": "entity client.admin using insecure key type: aes" >> }, >> { >> "message": "entity client.block-physical using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-mds using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-mgr using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-osd using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-rbd using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-rbd-mirror using >> insecure key type: aes" >> }, >> { >> "message": "entity client.bootstrap-rgw using insecure key >> type: aes" >> }, >> { >> "message": "entity client.ceph-exporter.c1 using insecure >> key type: aes" >> }, >> { >> "message": "entity client.ceph-exporter.c2 using insecure >> key type: aes" >> }, >> { >> "message": "entity client.ceph-exporter.c3 using insecure >> key type: aes" >> }, >> { >> "message": "entity client.crash.c1 using insecure key type: >> aes" >> }, >> { >> "message": "entity client.crash.c2 using insecure key type: >> aes" >> }, >> { >> "message": "entity client.crash.c3 using insecure key type: >> aes" >> }, >> { >> "message": "entity client.rgw. using insecure key type: aes" >> }, >> { >> "message": "entity client.rgw.c1 using insecure key type: aes" >> }, >> { >> "message": "entity client.rgw.c2 using insecure key type: aes" >> }, >> { >> "message": "entity client.rgw.rgw.c1.kevxdx using insecure >> key type: aes" >> }, >> { >> "message": "entity client.rgw.rgw.c2.fndzzj using insecure >> key type: aes" >> } >> ], >> "muted": false >> }, >> "AUTH_INSECURE_KEYS_ALLOWED": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "Monitors are configured to allow auth using >> insecure key types", >> "count": 1 >> }, >> "detail": [ >> { >> "message": "insecure cipher aes allowed for auth" >> } >> ], >> "muted": false >> }, >> "AUTH_INSECURE_KEYS_CREATABLE": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "Monitors are configured to allow creation of >> insecure key types", >> "count": 1 >> }, >> "detail": [], >> "muted": false >> }, >> "AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "4 rotating auth service keys using insecure key >> types", >> "count": 4 >> }, >> "detail": [ >> { >> "message": "rotating service keys for mon using insecure >> key type: aes" >> }, >> { >> "message": "rotating service keys for mds using insecure >> key type: aes" >> }, >> { >> "message": "rotating service keys for osd using insecure >> key type: aes" >> }, >> { >> "message": "rotating service keys for mgr using insecure >> key type: aes" >> } >> ], >> "muted": false >> }, >> "AUTH_INSECURE_SERVICE_KEY_TYPE": { >> "severity": "HEALTH_ERR", >> "summary": { >> "message": "2 auth service entities with insecure key types", >> "count": 2 >> }, >> "detail": [ >> { >> "message": "entity mds.c1 using insecure key type: aes" >> }, >> { >> "message": "entity mds.c2 using insecure key type: aes" >> } >> ], >> "muted": false >> }, >> "OSDMAP_FLAGS": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "noout,norebalance flag(s) set", >> "count": 17 >> }, >> "detail": [], >> "muted": false >> } >> }, >> "mutes": [] >> } >> >> root@c3:~# ceph orch reconfig mds.cephfs >> Scheduled to reconfig mds.cephfs.c1.secoxv on host 'c1' >> Scheduled to reconfig mds.cephfs.c2.lpzncl on host 'c2' >> 2026-08-24T11:33:27.362951+0100 mgr.c1.hsiqod [INF] Reconfig service >> mds.cephfs >> 2026-08-24T11:33:27.561183+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mds.cephfs.c1.secoxv on c1 >> 2026-08-24T11:33:28.448509+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mds.cephfs.c2.lpzncl on c2 >> >> root@c3:~# ceph orch daemon restart mds.cephfs.c2.lpzncl >> Scheduled to restart mds.cephfs.c2.lpzncl on host 'c2' >> root@c3:~# ceph orch daemon restart mds.cephfs.c1.secoxv >> Scheduled to restart mds.cephfs.c1.secoxv on host 'c1' >> 2026-08-24T11:34:33.571897+0100 mgr.c1.hsiqod [INF] Schedule restart >> daemon mds.cephfs.c2.lpzncl >> 2026-08-24T11:41:40.075162+0100 mgr.c1.hsiqod [INF] Schedule restart >> daemon mds.cephfs.c1.secoxv >> >> But the HEALTH_ERR : AUTH_INSECURE_SERVICE_KEY_TYPE error persists. >> >> root@c3:~# ceph -s >> cluster: >> id: 90573d1e-7825-58a3-b34a-2890a7f53cea >> health: HEALTH_ERR >> 19 auth client entities with insecure key types >> Monitors are configured to allow auth using insecure key >> types >> Monitors are configured to allow creation of insecure key >> types >> 4 rotating auth service keys using insecure key types >> 2 auth service entities with insecure key types >> noout,norebalance flag(s) set >> >> services: >> mon: 3 daemons, quorum c1,c2,c3 (age 75s) [leader: c1] >> mgr: c1.hsiqod(active, since 54m), standbys: c3.lythrw >> mds: 1/1 daemons up, 1 standby >> osd: 12 osds: 12 up (since 69s), 12 in (since 7M) >> flags noout,norebalance,noautoscale >> rgw: 2 daemons active (2 hosts, 1 zones) >> >> data: >> volumes: 1/1 healthy >> pools: 14 pools, 249 pgs >> objects: 322 objects, 35 MiB >> usage: 180 GiB used, 750 GiB / 930 GiB avail >> pgs: 249 active+clean >> >> io: >> client: 85 B/s rd, 0 op/s rd, 0 op/s wr >> >> root@c3:~# ceph fs status >> cephfs - 0 clients >> ====== >> RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS >> 0 active cephfs.c1.secoxv Reqs: 0 /s 12 15 13 0 >> POOL TYPE USED AVAIL >> cephfsmeta metadata 3161k 37.9G >> cephfsdata data 14.3k 148G >> STANDBY MDS >> cephfs.c2.lpzncl >> MDS version: ceph version 20.2.4 >> (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable - >> RelWithDebInfo) >> >> What should I do from here? >> >> Thanks, Chris >> >> >> On 19/08/2026 17:07, Patrick Donnelly wrote: >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> https://ceph.io/en/news/blog/ >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >>> >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >>> >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >>> >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >>> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >>> >>> >>> ## Getting Ceph >>> ------------ >>> * Source at https://github.com/ceph/ceph >>> * v19.2.6 Tarball at >>> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >>> * v20.2.4 Tarball at >>> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >>> * Containers at https://quay.io/repository/ceph/ceph >>> * For packages, see >>> https://docs.ceph.com/en/latest/install/get-packages/ >>> * For upgrade with cephadm, see: >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 24 Aug '26

24 Aug '26

Hi Chris, > "detail": [ > { > "message": "entity mds.c1 using insecure key type: aes" > }, > { > "message": "entity mds.c2 using insecure key type: aes" > } those two auth entries are most likely from before your cephadm upgrade, assuming you adopted this cluster with cephadm. They are most likely not in use by any daemons as you can see in your current daemon list: > Deploying daemon mds.cephfs.c1.secoxv on c1 > Deploying daemon mds.cephfs.c2.lpzncl on c2 Before cephadm, MDS daemons didn't have any suffixes, so those must be your old MDS daemons from before cephadm- I would recommend to double check, but you can probably just delete those two old auth entries: ceph auth del mds.c1 ceph auth del mds.c2 Just to be safe you can export those keyrings as a backup in case you actually do experience trouble after deleting those entries. Regards, Eugen Zitat von Chris Palmer <ceph-users(a)ceph.io>: > I'm upgrading a test cluster (cephadm, 20.2.3 -> 20.2.4). The > upgrade completed successfully, but the MDS processes (1 active, 1 > standby) still show as using the aes service key > (AUTH_INSECURE_SERVICE_KEY_TYPE). I have manually issued a reconfig > and restarted each daemon but no difference. > > Output of "ceph -W cephadm" (with the probably irrelevant stuff snipped out): > > 2026-08-24T11:03:16.341923+0100 mgr.c3.lythrw [INF] Upgrade: Started > with target quay.io/ceph/ceph:v20.2.4 > 2026-08-24T11:03:17.171731+0100 mgr.c3.lythrw [INF] Upgrade: First > pull of quay.io/ceph/ceph:v20.2.4 > 2026-08-24T11:03:35.480918+0100 mgr.c3.lythrw [INF] Upgrade: Target > is version 20.2.4 (tentacle) > 2026-08-24T11:03:35.480946+0100 mgr.c3.lythrw [INF] Upgrade: Target > container is > quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8, digests ['quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8', > 'quay.io/ceph/ceph@sha256:6bb1c8a42fbc0bf87938946990b65174466997bc11c31eb5a323225a779fd8f9'] > 2026-08-24T11:03:35.482902+0100 mgr.c3.lythrw [INF] Upgrade: Need to > upgrade myself (mgr.c3.lythrw) > 2026-08-24T11:03:36.195045+0100 mgr.c3.lythrw [INF] Upgrade: > Updating mgr.c1.hsiqod > 2026-08-24T11:03:36.250599+0100 mgr.c3.lythrw [INF] Deploying daemon > mgr.c1.hsiqod on c1 > [snip] > > 2026-08-24T11:08:16.388893+0100 mgr.c1.hsiqod [INF] Upgrade: It > appears safe to stop mds.cephfs.c1.secoxv > 2026-08-24T11:08:17.180069+0100 mgr.c1.hsiqod [INF] Upgrade: > Updating mds.cephfs.c1.secoxv > 2026-08-24T11:08:17.217174+0100 mgr.c1.hsiqod [INF] Deploying daemon > mds.cephfs.c1.secoxv on c1 > 2026-08-24T11:08:24.528567+0100 mgr.c1.hsiqod [INF] Reconfiguring > mds.cephfs.c2.lpzncl (monmap changed)... > 2026-08-24T11:08:24.534395+0100 mgr.c1.hsiqod [INF] Upgrade: It > appears safe to stop mds.cephfs.c2.lpzncl > 2026-08-24T11:08:25.420651+0100 mgr.c1.hsiqod [INF] Upgrade: > Updating mds.cephfs.c2.lpzncl > 2026-08-24T11:08:25.486295+0100 mgr.c1.hsiqod [INF] Deploying daemon > mds.cephfs.c2.lpzncl on c2 > [snip] > > 2026-08-24T11:11:16.759649+0100 mgr.c1.hsiqod [INF] Rotating keyring > for mds.cephfs.c1.secoxv > 2026-08-24T11:11:16.786643+0100 mgr.c1.hsiqod [INF] Redeploying > mds.cephfs.c1.secoxv with new keyring > 2026-08-24T11:11:16.787629+0100 mgr.c1.hsiqod [INF] Deploying daemon > mds.cephfs.c1.secoxv on c1 > 2026-08-24T11:11:26.030560+0100 mgr.c1.hsiqod [INF] Rotating keyring > for mds.cephfs.c2.lpzncl > 2026-08-24T11:11:26.076154+0100 mgr.c1.hsiqod [INF] Redeploying > mds.cephfs.c2.lpzncl with new keyring > 2026-08-24T11:11:26.076957+0100 mgr.c1.hsiqod [INF] Deploying daemon > mds.cephfs.c2.lpzncl on c2 > 2026-08-24T11:11:35.278324+0100 mgr.c1.hsiqod [INF] Upgrade: > checking if all mon, mgr, OSD, mds daemons upgraded before changing > service cipher > 2026-08-24T11:11:35.447319+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_KEYS_ALLOWED warning as upgrade is completed or has > been stopped > 2026-08-24T11:11:36.529431+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_KEYS_CREATABLE warning as upgrade is completed or has > been stopped > 2026-08-24T11:11:37.580675+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_SERVICE_TICKETS warning as upgrade is completed or has > been stopped > 2026-08-24T11:11:38.673337+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_CLIENT_KEY_TYPE warning as upgrade is completed or has > been stopped > 2026-08-24T11:11:39.743724+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_SERVICE_KEY_TYPE warning as upgrade is completed or > has been stopped > 2026-08-24T11:11:40.873968+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning as upgrade is > completed or has been stopped > 2026-08-24T11:11:41.937998+0100 mgr.c1.hsiqod [INF] Upgrade: Complete! > 2026-08-24T11:11:44.155457+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c1 (monmap changed)... > 2026-08-24T11:11:44.155490+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c1 (monmap changed)... > 2026-08-24T11:11:44.156559+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mon.c1 on c1 > 2026-08-24T11:11:45.052365+0100 mgr.c1.hsiqod [INF] Reconfiguring > mgr.c1.hsiqod (monmap changed)... > 2026-08-24T11:11:45.052389+0100 mgr.c1.hsiqod [INF] Reconfiguring > mgr.c1.hsiqod (monmap changed)... > 2026-08-24T11:11:45.053849+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mgr.c1.hsiqod on c1 > 2026-08-24T11:11:45.886112+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c2 (monmap changed)... > 2026-08-24T11:11:45.886156+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c2 (monmap changed)... > 2026-08-24T11:11:45.887368+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mon.c2 on c2 > 2026-08-24T11:11:46.725430+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c3 (monmap changed)... > 2026-08-24T11:11:46.725461+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c3 (monmap changed)... > 2026-08-24T11:11:46.726439+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mon.c3 on c3 > > > At this point nothing more happens. But: > > root@c3:~# ceph health detail --format=json|jq . > { > "status": "HEALTH_ERR", > "checks": { > "AUTH_INSECURE_CLIENT_KEY_TYPE": { > "severity": "HEALTH_WARN", > "summary": { > "message": "19 auth client entities with insecure key types", > "count": 19 > }, > "detail": [ > { > "message": "entity client.admin using insecure key type: aes" > }, > { > "message": "entity client.block-physical using insecure > key type: aes" > }, > { > "message": "entity client.bootstrap-mds using insecure key > type: aes" > }, > { > "message": "entity client.bootstrap-mgr using insecure key > type: aes" > }, > { > "message": "entity client.bootstrap-osd using insecure key > type: aes" > }, > { > "message": "entity client.bootstrap-rbd using insecure key > type: aes" > }, > { > "message": "entity client.bootstrap-rbd-mirror using > insecure key type: aes" > }, > { > "message": "entity client.bootstrap-rgw using insecure key > type: aes" > }, > { > "message": "entity client.ceph-exporter.c1 using insecure > key type: aes" > }, > { > "message": "entity client.ceph-exporter.c2 using insecure > key type: aes" > }, > { > "message": "entity client.ceph-exporter.c3 using insecure > key type: aes" > }, > { > "message": "entity client.crash.c1 using insecure key type: aes" > }, > { > "message": "entity client.crash.c2 using insecure key type: aes" > }, > { > "message": "entity client.crash.c3 using insecure key type: aes" > }, > { > "message": "entity client.rgw. using insecure key type: aes" > }, > { > "message": "entity client.rgw.c1 using insecure key type: aes" > }, > { > "message": "entity client.rgw.c2 using insecure key type: aes" > }, > { > "message": "entity client.rgw.rgw.c1.kevxdx using insecure > key type: aes" > }, > { > "message": "entity client.rgw.rgw.c2.fndzzj using insecure > key type: aes" > } > ], > "muted": false > }, > "AUTH_INSECURE_KEYS_ALLOWED": { > "severity": "HEALTH_WARN", > "summary": { > "message": "Monitors are configured to allow auth using > insecure key types", > "count": 1 > }, > "detail": [ > { > "message": "insecure cipher aes allowed for auth" > } > ], > "muted": false > }, > "AUTH_INSECURE_KEYS_CREATABLE": { > "severity": "HEALTH_WARN", > "summary": { > "message": "Monitors are configured to allow creation of > insecure key types", > "count": 1 > }, > "detail": [], > "muted": false > }, > "AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE": { > "severity": "HEALTH_WARN", > "summary": { > "message": "4 rotating auth service keys using insecure key types", > "count": 4 > }, > "detail": [ > { > "message": "rotating service keys for mon using insecure > key type: aes" > }, > { > "message": "rotating service keys for mds using insecure > key type: aes" > }, > { > "message": "rotating service keys for osd using insecure > key type: aes" > }, > { > "message": "rotating service keys for mgr using insecure > key type: aes" > } > ], > "muted": false > }, > "AUTH_INSECURE_SERVICE_KEY_TYPE": { > "severity": "HEALTH_ERR", > "summary": { > "message": "2 auth service entities with insecure key types", > "count": 2 > }, > "detail": [ > { > "message": "entity mds.c1 using insecure key type: aes" > }, > { > "message": "entity mds.c2 using insecure key type: aes" > } > ], > "muted": false > }, > "OSDMAP_FLAGS": { > "severity": "HEALTH_WARN", > "summary": { > "message": "noout,norebalance flag(s) set", > "count": 17 > }, > "detail": [], > "muted": false > } > }, > "mutes": [] > } > > root@c3:~# ceph orch reconfig mds.cephfs > Scheduled to reconfig mds.cephfs.c1.secoxv on host 'c1' > Scheduled to reconfig mds.cephfs.c2.lpzncl on host 'c2' > 2026-08-24T11:33:27.362951+0100 mgr.c1.hsiqod [INF] Reconfig service > mds.cephfs > 2026-08-24T11:33:27.561183+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mds.cephfs.c1.secoxv on c1 > 2026-08-24T11:33:28.448509+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mds.cephfs.c2.lpzncl on c2 > > root@c3:~# ceph orch daemon restart mds.cephfs.c2.lpzncl > Scheduled to restart mds.cephfs.c2.lpzncl on host 'c2' > root@c3:~# ceph orch daemon restart mds.cephfs.c1.secoxv > Scheduled to restart mds.cephfs.c1.secoxv on host 'c1' > 2026-08-24T11:34:33.571897+0100 mgr.c1.hsiqod [INF] Schedule restart > daemon mds.cephfs.c2.lpzncl > 2026-08-24T11:41:40.075162+0100 mgr.c1.hsiqod [INF] Schedule restart > daemon mds.cephfs.c1.secoxv > > But the HEALTH_ERR : AUTH_INSECURE_SERVICE_KEY_TYPE error persists. > > root@c3:~# ceph -s > cluster: > id: 90573d1e-7825-58a3-b34a-2890a7f53cea > health: HEALTH_ERR > 19 auth client entities with insecure key types > Monitors are configured to allow auth using insecure key types > Monitors are configured to allow creation of insecure key types > 4 rotating auth service keys using insecure key types > 2 auth service entities with insecure key types > noout,norebalance flag(s) set > > services: > mon: 3 daemons, quorum c1,c2,c3 (age 75s) [leader: c1] > mgr: c1.hsiqod(active, since 54m), standbys: c3.lythrw > mds: 1/1 daemons up, 1 standby > osd: 12 osds: 12 up (since 69s), 12 in (since 7M) > flags noout,norebalance,noautoscale > rgw: 2 daemons active (2 hosts, 1 zones) > > data: > volumes: 1/1 healthy > pools: 14 pools, 249 pgs > objects: 322 objects, 35 MiB > usage: 180 GiB used, 750 GiB / 930 GiB avail > pgs: 249 active+clean > > io: > client: 85 B/s rd, 0 op/s rd, 0 op/s wr > > root@c3:~# ceph fs status > cephfs - 0 clients > ====== > RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS > 0 active cephfs.c1.secoxv Reqs: 0 /s 12 15 13 0 > POOL TYPE USED AVAIL > cephfsmeta metadata 3161k 37.9G > cephfsdata data 14.3k 148G > STANDBY MDS > cephfs.c2.lpzncl > MDS version: ceph version 20.2.4 > (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable - > RelWithDebInfo) > > What should I do from here? > > Thanks, Chris > > > On 19/08/2026 17:07, Patrick Donnelly wrote: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: >> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 24 Aug '26

24 Aug '26

I'm upgrading a test cluster (cephadm, 20.2.3 -> 20.2.4). The upgrade completed successfully, but the MDS processes (1 active, 1 standby) still show as using the aes service key (AUTH_INSECURE_SERVICE_KEY_TYPE). I have manually issued a reconfig and restarted each daemon but no difference. Output of "ceph -W cephadm" (with the probably irrelevant stuff snipped out): 2026-08-24T11:03:16.341923+0100 mgr.c3.lythrw [INF] Upgrade: Started with target quay.io/ceph/ceph:v20.2.4 2026-08-24T11:03:17.171731+0100 mgr.c3.lythrw [INF] Upgrade: First pull of quay.io/ceph/ceph:v20.2.4 2026-08-24T11:03:35.480918+0100 mgr.c3.lythrw [INF] Upgrade: Target is version 20.2.4 (tentacle) 2026-08-24T11:03:35.480946+0100 mgr.c3.lythrw [INF] Upgrade: Target container is quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8, digests ['quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8', 'quay.io/ceph/ceph@sha256:6bb1c8a42fbc0bf87938946990b65174466997bc11c31eb5a323225a779fd8f9'] 2026-08-24T11:03:35.482902+0100 mgr.c3.lythrw [INF] Upgrade: Need to upgrade myself (mgr.c3.lythrw) 2026-08-24T11:03:36.195045+0100 mgr.c3.lythrw [INF] Upgrade: Updating mgr.c1.hsiqod 2026-08-24T11:03:36.250599+0100 mgr.c3.lythrw [INF] Deploying daemon mgr.c1.hsiqod on c1 [snip] 2026-08-24T11:08:16.388893+0100 mgr.c1.hsiqod [INF] Upgrade: It appears safe to stop mds.cephfs.c1.secoxv 2026-08-24T11:08:17.180069+0100 mgr.c1.hsiqod [INF] Upgrade: Updating mds.cephfs.c1.secoxv 2026-08-24T11:08:17.217174+0100 mgr.c1.hsiqod [INF] Deploying daemon mds.cephfs.c1.secoxv on c1 2026-08-24T11:08:24.528567+0100 mgr.c1.hsiqod [INF] Reconfiguring mds.cephfs.c2.lpzncl (monmap changed)... 2026-08-24T11:08:24.534395+0100 mgr.c1.hsiqod [INF] Upgrade: It appears safe to stop mds.cephfs.c2.lpzncl 2026-08-24T11:08:25.420651+0100 mgr.c1.hsiqod [INF] Upgrade: Updating mds.cephfs.c2.lpzncl 2026-08-24T11:08:25.486295+0100 mgr.c1.hsiqod [INF] Deploying daemon mds.cephfs.c2.lpzncl on c2 [snip] 2026-08-24T11:11:16.759649+0100 mgr.c1.hsiqod [INF] Rotating keyring for mds.cephfs.c1.secoxv 2026-08-24T11:11:16.786643+0100 mgr.c1.hsiqod [INF] Redeploying mds.cephfs.c1.secoxv with new keyring 2026-08-24T11:11:16.787629+0100 mgr.c1.hsiqod [INF] Deploying daemon mds.cephfs.c1.secoxv on c1 2026-08-24T11:11:26.030560+0100 mgr.c1.hsiqod [INF] Rotating keyring for mds.cephfs.c2.lpzncl 2026-08-24T11:11:26.076154+0100 mgr.c1.hsiqod [INF] Redeploying mds.cephfs.c2.lpzncl with new keyring 2026-08-24T11:11:26.076957+0100 mgr.c1.hsiqod [INF] Deploying daemon mds.cephfs.c2.lpzncl on c2 2026-08-24T11:11:35.278324+0100 mgr.c1.hsiqod [INF] Upgrade: checking if all mon, mgr, OSD, mds daemons upgraded before changing service cipher 2026-08-24T11:11:35.447319+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_KEYS_ALLOWED warning as upgrade is completed or has been stopped 2026-08-24T11:11:36.529431+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_KEYS_CREATABLE warning as upgrade is completed or has been stopped 2026-08-24T11:11:37.580675+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_SERVICE_TICKETS warning as upgrade is completed or has been stopped 2026-08-24T11:11:38.673337+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_CLIENT_KEY_TYPE warning as upgrade is completed or has been stopped 2026-08-24T11:11:39.743724+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_SERVICE_KEY_TYPE warning as upgrade is completed or has been stopped 2026-08-24T11:11:40.873968+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning as upgrade is completed or has been stopped 2026-08-24T11:11:41.937998+0100 mgr.c1.hsiqod [INF] Upgrade: Complete! 2026-08-24T11:11:44.155457+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c1 (monmap changed)... 2026-08-24T11:11:44.155490+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c1 (monmap changed)... 2026-08-24T11:11:44.156559+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mon.c1 on c1 2026-08-24T11:11:45.052365+0100 mgr.c1.hsiqod [INF] Reconfiguring mgr.c1.hsiqod (monmap changed)... 2026-08-24T11:11:45.052389+0100 mgr.c1.hsiqod [INF] Reconfiguring mgr.c1.hsiqod (monmap changed)... 2026-08-24T11:11:45.053849+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mgr.c1.hsiqod on c1 2026-08-24T11:11:45.886112+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c2 (monmap changed)... 2026-08-24T11:11:45.886156+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c2 (monmap changed)... 2026-08-24T11:11:45.887368+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mon.c2 on c2 2026-08-24T11:11:46.725430+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c3 (monmap changed)... 2026-08-24T11:11:46.725461+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c3 (monmap changed)... 2026-08-24T11:11:46.726439+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mon.c3 on c3 At this point nothing more happens. But: root@c3:~# ceph health detail --format=json|jq . { "status": "HEALTH_ERR", "checks": { "AUTH_INSECURE_CLIENT_KEY_TYPE": { "severity": "HEALTH_WARN", "summary": { "message": "19 auth client entities with insecure key types", "count": 19 }, "detail": [ { "message": "entity client.admin using insecure key type: aes" }, { "message": "entity client.block-physical using insecure key type: aes" }, { "message": "entity client.bootstrap-mds using insecure key type: aes" }, { "message": "entity client.bootstrap-mgr using insecure key type: aes" }, { "message": "entity client.bootstrap-osd using insecure key type: aes" }, { "message": "entity client.bootstrap-rbd using insecure key type: aes" }, { "message": "entity client.bootstrap-rbd-mirror using insecure key type: aes" }, { "message": "entity client.bootstrap-rgw using insecure key type: aes" }, { "message": "entity client.ceph-exporter.c1 using insecure key type: aes" }, { "message": "entity client.ceph-exporter.c2 using insecure key type: aes" }, { "message": "entity client.ceph-exporter.c3 using insecure key type: aes" }, { "message": "entity client.crash.c1 using insecure key type: aes" }, { "message": "entity client.crash.c2 using insecure key type: aes" }, { "message": "entity client.crash.c3 using insecure key type: aes" }, { "message": "entity client.rgw. using insecure key type: aes" }, { "message": "entity client.rgw.c1 using insecure key type: aes" }, { "message": "entity client.rgw.c2 using insecure key type: aes" }, { "message": "entity client.rgw.rgw.c1.kevxdx using insecure key type: aes" }, { "message": "entity client.rgw.rgw.c2.fndzzj using insecure key type: aes" } ], "muted": false }, "AUTH_INSECURE_KEYS_ALLOWED": { "severity": "HEALTH_WARN", "summary": { "message": "Monitors are configured to allow auth using insecure key types", "count": 1 }, "detail": [ { "message": "insecure cipher aes allowed for auth" } ], "muted": false }, "AUTH_INSECURE_KEYS_CREATABLE": { "severity": "HEALTH_WARN", "summary": { "message": "Monitors are configured to allow creation of insecure key types", "count": 1 }, "detail": [], "muted": false }, "AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE": { "severity": "HEALTH_WARN", "summary": { "message": "4 rotating auth service keys using insecure key types", "count": 4 }, "detail": [ { "message": "rotating service keys for mon using insecure key type: aes" }, { "message": "rotating service keys for mds using insecure key type: aes" }, { "message": "rotating service keys for osd using insecure key type: aes" }, { "message": "rotating service keys for mgr using insecure key type: aes" } ], "muted": false }, "AUTH_INSECURE_SERVICE_KEY_TYPE": { "severity": "HEALTH_ERR", "summary": { "message": "2 auth service entities with insecure key types", "count": 2 }, "detail": [ { "message": "entity mds.c1 using insecure key type: aes" }, { "message": "entity mds.c2 using insecure key type: aes" } ], "muted": false }, "OSDMAP_FLAGS": { "severity": "HEALTH_WARN", "summary": { "message": "noout,norebalance flag(s) set", "count": 17 }, "detail": [], "muted": false } }, "mutes": [] } root@c3:~# ceph orch reconfig mds.cephfs Scheduled to reconfig mds.cephfs.c1.secoxv on host 'c1' Scheduled to reconfig mds.cephfs.c2.lpzncl on host 'c2' 2026-08-24T11:33:27.362951+0100 mgr.c1.hsiqod [INF] Reconfig service mds.cephfs 2026-08-24T11:33:27.561183+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mds.cephfs.c1.secoxv on c1 2026-08-24T11:33:28.448509+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mds.cephfs.c2.lpzncl on c2 root@c3:~# ceph orch daemon restart mds.cephfs.c2.lpzncl Scheduled to restart mds.cephfs.c2.lpzncl on host 'c2' root@c3:~# ceph orch daemon restart mds.cephfs.c1.secoxv Scheduled to restart mds.cephfs.c1.secoxv on host 'c1' 2026-08-24T11:34:33.571897+0100 mgr.c1.hsiqod [INF] Schedule restart daemon mds.cephfs.c2.lpzncl 2026-08-24T11:41:40.075162+0100 mgr.c1.hsiqod [INF] Schedule restart daemon mds.cephfs.c1.secoxv But the HEALTH_ERR : AUTH_INSECURE_SERVICE_KEY_TYPE error persists. root@c3:~# ceph -s cluster: id: 90573d1e-7825-58a3-b34a-2890a7f53cea health: HEALTH_ERR 19 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types 4 rotating auth service keys using insecure key types 2 auth service entities with insecure key types noout,norebalance flag(s) set services: mon: 3 daemons, quorum c1,c2,c3 (age 75s) [leader: c1] mgr: c1.hsiqod(active, since 54m), standbys: c3.lythrw mds: 1/1 daemons up, 1 standby osd: 12 osds: 12 up (since 69s), 12 in (since 7M) flags noout,norebalance,noautoscale rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 249 pgs objects: 322 objects, 35 MiB usage: 180 GiB used, 750 GiB / 930 GiB avail pgs: 249 active+clean io: client: 85 B/s rd, 0 op/s rd, 0 op/s wr root@c3:~# ceph fs status cephfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.c1.secoxv Reqs: 0 /s 12 15 13 0 POOL TYPE USED AVAIL cephfsmeta metadata 3161k 37.9G cephfsdata data 14.3k 148G STANDBY MDS cephfs.c2.lpzncl MDS version: ceph version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable - RelWithDebInfo) What should I do from here? Thanks, Chris On 19/08/2026 17:07, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Johan 24 Aug '26

24 Aug '26

Ok, on Debian 12 it was not a problem upgrading to ceph 20.2.4 using apt and https://download.ceph.com/debian-tentacle/ With Debian 13 which I have at my troublesome cluster I get several dependency-issues. ceph-common is dependent on libicu72 which doesn't have any candidates python3-cephfs (and others) is dependent on python3 < 3.12 but 3.13.5-1 will be installed etc Is this because I perhaps hadn't upgraded ceph-common and others properly before upgrading Debian from 12 to 13? /Johan Den 2026-08-24 kl. 11:06, skrev Johan: > Im not a professional computer tech but have two ceph clusters (cephadm, > Trixie, Debian 12/13). > > I upgraded the least sensitive one using the dashboard. When I upgrade > the admin client I now get > auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error > setting modifier for [client.admin] type=key val=<key>: Malformed input > [buffer:3] > auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/ > output error > > I get the same respons when using a ordinary user to connect the > filesystem. > > I found this issue describing the same error > https://github.com/rook/rook/issues/18240 > > As they, in above issue, discussed which version/image of the toolbox > was used I checked my version of ceph at the command line. I get > ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef > (stable) > I would guess this is the problem. > > How do I get to run up-to-date version on the command line? > > /Johan > > > Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/ >> index.html#upgrading-and-rotating-cephx-keys >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/ >> index.html#auth-insecure-keys-creatable >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high- >> availability-authentication >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh- >> configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx- >> keys-patrick-donnelly-ibm-yehuda-sadeh-weinraub-ubiquiti >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ >> ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ >> ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Jan Kasprzak 24 Aug '26

24 Aug '26

Hi Nico, Nico Schottelius wrote: > Robert Sander <ceph-users(a)ceph.io> writes: > > > Hi, > > > > Am 21.08.26 um 4:10 PM schrieb Nico Schottelius: > > > >> a) All VPS/VMs need to be restarted > >> To be able to rotate keys, aside from the almost impossible flow to > >> actually doing the rotation (more to that below), this will affect > >> every consumer of VMs. The usual setup is that all hypervisors are > >> using the same key. > > > > VM live migration should be able to do the trick here. When the new > > compute node already has the new key and the librbd version to support > > it. > > That's a very good point. However this might also need work (or > hacks) in the hypervisor sphere, to support using different keys in the > 2 qemu processes. By default in opennebula, the ceph key is saved in > a libvirtd secret. So in the best case, the new process will read it > from the local secret store. > > In the not so great case, opennebula (and other hypervisors) might infer > the key from the previously deployed VM, instead of reading it from the > local secret store. (*) > > (*) I'm not sure about the state of other hypervisors, but in the > opennebula case keys are actually embededd in the XML VM definition (sic!) that is > saved in the database (sic!!) and copied on creation of the VM (!!!) and > will never every be changed. I think the problem is not the secret/key itself, Ceph username is the problem, as it seems to be global in OpenNebula. The problem you assume does not exist in OpenNebula - it correctly refers to the libvirt secret UUID, not the Ceph user key itself. You can put another key under the same libvirt secret UUID if you want: # virsh dumpxml one-... [...] <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <auth username='libvirt'> <secret type='ceph' uuid='9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf'/> </auth> <source protocol='rbd' name='one/one-1234-123456-0' index='2'> <host name='mon1' port='6789'/> <host name='mon2' port='6789'/> <host name='mon3' port='6789'/> </source> Create a new Ceph user with # ceph auth get-or-create client.libvirt2026 \ mon 'profile rbd' \ osd 'profile rbd pool=one' \ mgr 'profile rbd pool=one' \ -o libvirt2026.keyring Import the key under the same UUID on an idle/disabled node: # cat client.libvirt2026.secret.xml <secret ephemeral='no' private='no'> <uuid>9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf</uuid> <usage type='ceph'> <name>client.libvirt2026 secret</name> </usage> </secret> # virsh secret-undefine 9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf # virsh secret-define client.libvirt2026.secret.xml # KEY=`ceph auth get-key client.libvirt2026` # virsh secret-set-value --secret 9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf --base64 $KEY But it does not work, as I cannot persuade OpenNebula to use a different Ceph user instead of "libvirt" on that node (as opposed to the whole cluster, which would probably break running VMs). In the running VM template there are the following embedded values for a Ceph disk: "DISK": { [...] "CEPH_HOST": "mon1 mon2 mon3", "CEPH_SECRET": "9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf", "CEPH_TRASH": "yes", "CEPH_USER": "libvirt", [...] FWIW, I also created this post on OpenNebula forum: https://forum.opennebula.io/t/ceph-key-rollover/14547 -Yenya > > A likely working hack is to modify the key inside the XML inside the > database and then trigger live migration with a small time window of > uncertainty that if the VM crashes on the original host before the live > migration is triggered, it will likely not restart. > > -- > Sustainable and modern Infrastructures by ungleich.ch > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Johan 24 Aug '26

24 Aug '26

Im not a professional computer tech but have two ceph clusters (cephadm, Trixie, Debian 12/13). I upgraded the least sensitive one using the dashboard. When I upgrade the admin client I now get auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error setting modifier for [client.admin] type=key val=<key>: Malformed input [buffer:3] auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error I get the same respons when using a ordinary user to connect the filesystem. I found this issue describing the same error https://github.com/rook/rook/issues/18240 As they, in above issue, discussed which version/image of the toolbox was used I checked my version of ceph at the command line. I get ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) I would guess this is the problem. How do I get to run up-to-date version on the command line? /Johan Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0

Ceph Tentacle 20.2.4 Noble support
by Marek Szuba 24 Aug '26

24 Aug '26

Dear Ceph maintainers, Further to the thread about 20.2.3 Noble support earlier this month, I would like to point out that with there having been Noble packages for up to and including 20.2.0, Noble users are now stuck with Ceph versions with known security vulnerabilities. For many operators, ourselves included, a downgrade to Jammy is not possible even it it were to be done on Ceph servers themselves (as I mentioned in passing in one of my messages to this list we experienced severe I/O issues with Jammy-backed Ceph OSDs in one of our clusters, whereas another requires Noble or newer in order for Podman to be new enough for SMB containers). I would therefore STRONGLY encourage you to resume the building and publishing of packages for Noble ASAP irrespective of whether this particular version of Ubuntu has reached the "officially supported" status or not. Thank you for your consideration. -- MS

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Alexander Patrakov 22 Aug '26

22 Aug '26

On Fri, Aug 21, 2026 at 7:28 PM Stephan Hohn <ceph-users(a)ceph.io> wrote: > podman run --privileged --entrypoint /tmp/cryptsetup_action.sh \ > -v /tmp/cephadm-osd-14-rotate-helper.sh:/tmp/cryptsetup_action.sh <image> Hey, isn't this (the use of a predictably named file in /tmp, that the adversary can pre-create, e.g., as a symlink to something else) a vulnerability by itself? -- Alexander Patrakov

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by William Xiong 21 Aug '26

21 Aug '26

Hi Patrick, Regarding the fix for CVE-2025-30156 We've discovered that the bluestore label still contains the old key after following instructions at https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… Old key shows up as per: ceph-bluestore-tool show-label --dev "$(readlink -f /var/lib/ceph/osd/ceph-X/block)" | grep osd_key If we reboot the osd servers, the new key is overwritten with the old key in /var/lib/ceph/osd/ceph-${OSD_ID}/keyring causing osd auth failure > osd service doesnt start Can simulate this without reboot via `umount /var/lib/ceph/osd/ceph-${OSD_ID}/` and reactivate via `ceph-volume lvm activate --all` We are running a non cephadm cluster. Need to update the label as well after rotating the key: ceph-bluestore-tool set-label-key --dev "$(readlink -f /var/lib/ceph/osd/ceph-${OSD_ID}/block)" -k osd_key -v $NEW_KEY Kind Regards, William Xiong | Infrastructure Engineer - Storage Research Computing Services The University of Melbourne, Victoria 3010 Australia E: william.xiong(a)unimelb.edu.au<mailto:william.xiong@unimelb.edu.au> ________________________________ From: Patrick Donnelly <ceph-users(a)ceph.io> Sent: Thursday, 20 August 2026 2:08 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released Today, August 19th, 2026, the Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components. This is the sixth backport release in the Squid series. This is the fourth backport release in the Tentacle series. Additional details will be posted on the Ceph website/blog: https://ceph.io/en/news/blog We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible. ## Critical Upgrade Steps The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at: https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… These steps are for package-based deployments of Ceph. In other sections, the document also includes new information about key rotation procedures, monitor authentication controls, and emergency recovery procedures. Deployments using cephadm will automate the process except for client keys. You may notice that Cephadm spends more time than normal on the upgrade after updating all daemon images. This is due to a new process rotating the OSD and MDS keys. Please be patient. There are plans to improve this in the future. Deployments using Rook will also automate rotation of some client keys with some exclusions. Look to the Rook project’s announcement for more details. Client and kernel upgrades are recommended to support aes256k. Note that upstream Linux kernel client support began in kernel 7.0 and has been backported to CentOS Stream 9 and 10. Check with your distribution vendor for backported key support before rotating client keys used by the kernel. As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go. These are documented beginning here: https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… The CephX documentation has been brought up to date and vetted for accuracy. Those changes are live here: https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 requests with `host` and `x-amz-` headers not included in the signed subset. Unfortunately, the REST client used in multisite was generating such improperly signed requests. If you are running multisite, you must set the ``rgw_sigv4_insecure`` option to true before you begin to upgrade. After all clusters are upgraded, set the option to ``false`` again. Regarding the fix for CVE-2026-50152, formal guidance on rotating all secrets stored in the Monitor config-key store will be forthcoming. In the meantime, operators of cephadm-managed clusters are strongly encouraged to rotate the cephadm SSH key, as an established process exists: See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration Rotating other secrets is advised but the process is not yet established. Operators should assess their cluster's potential exposure to determine if immediate Monitor config-key secret rotation is required for their environment. ## CVE Fixes * CVE-2025-30156 is an authentication bypass in CephX caused by misuse of AES-CBC. See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 * CVE-2026-39944 shares the unauthenticated-encryption root cause of CVE-2025-30156, but applies it to RGW's STS session tokens resulting in improper verification of a cryptographic signature. See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor subscription handler. See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 cryptographic signatures in RGW's SigV4 verifier. See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 ## Other Media A Cephalocon 2025 talk covers some of the details surrounding the new CephX key upgrade process. https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… ## Getting Ceph ------------ * Source at https://github.com/ceph/ceph * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages * For upgrade with cephadm, see: https://docs.ceph.com/en/latest/cephadm/upgrade * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Robert Sander 21 Aug '26

21 Aug '26

Hi Patrick, Am 21.08.26 um 4:27 PM schrieb Patrick Donnelly: > Again, rotating the client keys is not necessary to address the main > vulnerabilities associated with the CVE. You may mute the > AUTH_INSECURE_CLIENT_KEY_TYPE warning on an extended basis or > permanently if it cannot be resolved for the forseeable future. Thanks for pointing this out so clearly. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Nico Schottelius 21 Aug '26

21 Aug '26

Ciao Robert, Robert Sander <ceph-users(a)ceph.io> writes: > Hi, > > Am 21.08.26 um 4:10 PM schrieb Nico Schottelius: > >> a) All VPS/VMs need to be restarted >> To be able to rotate keys, aside from the almost impossible flow to >> actually doing the rotation (more to that below), this will affect >> every consumer of VMs. The usual setup is that all hypervisors are >> using the same key. > > VM live migration should be able to do the trick here. When the new > compute node already has the new key and the librbd version to support > it. That's a very good point. However this might also need work (or hacks) in the hypervisor sphere, to support using different keys in the 2 qemu processes. By default in opennebula, the ceph key is saved in a libvirtd secret. So in the best case, the new process will read it from the local secret store. In the not so great case, opennebula (and other hypervisors) might infer the key from the previously deployed VM, instead of reading it from the local secret store. (*) >> I'm not saying anything cannot be done, but the way how this >> vulnerabilty is solved will cause major headache around the industry. > > IMHO we will see a muted health warning for the years to come. Agreed. BR, Nico (*) I'm not sure about the state of other hypervisors, but in the opennebula case keys are actually embededd in the XML VM definition (sic!) that is saved in the database (sic!!) and copied on creation of the VM (!!!) and will never every be changed. A likely working hack is to modify the key inside the XML inside the database and then trigger live migration with a small time window of uncertainty that if the VM crashes on the original host before the live migration is triggered, it will likely not restart. -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 21 Aug '26

21 Aug '26

On Fri, Aug 21, 2026 at 10:08 AM Nico Schottelius <ceph-users(a)ceph.io> wrote: > > > Good morning list, > > some of you already mentioned the issue in relation to proxmox, but > virtually all VPS/VM hostings are affected that are running qemu/kvm. > > The situation is even worse, if I see it correctly: > > a) All VPS/VMs need to be restarted > To be able to rotate keys, aside from the almost impossible flow to > actually doing the rotation (more to that below), this will affect > every consumer of VMs. The usual setup is that all hypervisors are > using the same key. > > b) Key support in qemu/kvm > From my understanding, qemu/kvm + librbd does at the moment not have > support for the new key type. Thus there will be major waiting > periods until the packages have been updated and eventually make it > back into the distros. If they are being done at all, depending on > your setup there might be old qemu versions which cannot easily be > upgraded. > > c) Key rotation > Assuming that the support actually has landed in qemu/kvm (and that > can still take months), one will need to rotate potentially > update/edit generade libvirtd xml that contains the reference to the > ceph key or the key itself. > > I'm not saying anything cannot be done, but the way how this > vulnerabilty is solved will cause major headache around the industry. Again, rotating the client keys is not necessary to address the main vulnerabilities associated with the CVE. You may mute the AUTH_INSECURE_CLIENT_KEY_TYPE warning on an extended basis or permanently if it cannot be resolved for the forseeable future. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 2

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Jan Kasprzak 21 Aug '26

21 Aug '26

Hello Patrick, regarding the step 4 of the documentation: I upgraded another cluster, and now I see the issue: > 4. Confirm the AUTH_INSECURE_SERVICE_KEY_TYPE is cleared. > > ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_SERVICE_KEY_TYPE") | not' > > output gives false. > > If it outputs true, there is another daemon that needs to be upgraded. > Check the output of ceph health detail. This is incorrect. When all the daemons have their keys rotated, then "ceph health detail" does _not_ contain AUTH_INSECURE_SERVICE_KEY_TYPE error, so the above "... | not" returns true, not false. So either remove the "| not" part from the docs, or replace false <--> true in the next two sentences. Regards, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Dmitry Melekhov 21 Aug '26

21 Aug '26

21.08.2026 18:06, Nico Schottelius пишет: > Good morning list, > > some of you already mentioned the issue in relation to proxmox, but > virtually all VPS/VM hostings are affected that are running qemu/kvm. > > By the way I don't see that CVE is critical in case of hypervisor is running on the same hosts and subnet is isolated by firewall. May be it is better to have an option to not change keys during upgrade but leave everything as it is until it is suitable time to implement more secure keys?

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Robert Sander 21 Aug '26

21 Aug '26

Hi, Am 21.08.26 um 4:10 PM schrieb Nico Schottelius: > a) All VPS/VMs need to be restarted > To be able to rotate keys, aside from the almost impossible flow to > actually doing the rotation (more to that below), this will affect > every consumer of VMs. The usual setup is that all hypervisors are > using the same key. VM live migration should be able to do the trick here. When the new compute node already has the new key and the librbd version to support it. > I'm not saying anything cannot be done, but the way how this > vulnerabilty is solved will cause major headache around the industry. IMHO we will see a muted health warning for the years to come. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 21 Aug '26

21 Aug '26

On Fri, Aug 21, 2026 at 7:40 AM Burkhard Linke <ceph-users(a)ceph.io> wrote: > > Hi, > > On 8/20/26 21:35, Patrick Donnelly wrote: > >> A challenge to that approach is that Proxmox PVE defines external RBD > >> pools at the datacenter level using a single user/key pair. This then > >> becomes the storage pool on each individual node in the proxmox > >> cluster. I believe ceph-csi K8 storage class also defines this once > >> cluster wide. > > I understand but the only way to change things is to spread the word. > > If the key is that challenging to rotate, then I would suggest leaving > > it as aes and push for changes in the product or deployment > > technology. Rook developers are already aware this should be reworked. > > So you expect every user running Proxmox PVE, K8s with ceph csi or > Openstack (which is usually more than just a handful of hosts) to figure > out how to adopt their deployment? Which is not possible at this time > for the mentioned cases. I'm not sure what you mean. I think the docs are clear that it may not be possible to rotate client keys to the stronger key type due to various technical hurdles. That is okay. The CVE is mitigated by updating the core Ceph daemons. > Most users will probably welcome a more detailed documentation on how to > handle key rotation _and_ especially how to manage the client side in > this scenario. As was already mentioned before, most of us cannot simply > shutdown all virtual machines in Proxmox or Openstack for a key rotation.... https://docs.ceph.com/en/latest/security/CVE-2025-30156/ "Client/kernel upgrades are recommended to support the new key type but not required to resolve the most serious aspects of the security vulnerability." and https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… "If you cannot rotate a particular client key yet, you may prefer to mute the health warning until you can complete upgrading all of the client keys. We expect this to be typical situation for some clusters." How would you like the docs improved? -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Handling key rotation for proxmox , openstack and ceph csi clients
by Burkhard Linke 21 Aug '26

21 Aug '26

Hi, since the latest ceph releases introduce a new key type to fix a security vulnerability, the release notes recommend to rotate the client keys and switch to the new key algorithm. This might not be possible for client that use a shared key for multiple hosts, e.g. proxmox, openstack and ceph-csi in k8s clusters. The recommended way to handle this is using host specific keys, but this it currently not an option for the mentioned clients. I would like to collect the various methods for handling the client side of the key rotation in the thread as a reference for other users. We did not upgrade yet, so the following steps are not tested yet. But we did something similar in the past in in different scenario, so it should work [tm]. So: use at own risk - Proxmox and Openstack Both usually use qemu, and qemu has support for performing a data live migration. In the case of proxmox I would propose to define a new rbd pool, using the same setup (crush etc.) as the original data pool. Then define a new storage backend in proxmox using the new pool and a ceph client with the new key algorithm. After testing the pool you should be able to migrate existing disks to the new pool. The migration option is available in the hardware overview of each virtual machine; select the disk, and the "Disk action" menu will allow to move the storage to a different backend. As mentioned above this is a live migration, so the running instance will not be affected (except for slightly higher I/O latencies). Since proxmox offers an API and CLI utilities, you should also be able to automate this process if you are running a large number of instances. Openstack supports live data migration for cinder volumes. You need to define a second volume type, again with a new pool and a new ceph client. The pool has to be configured on all controllers , which should be performed by your deployment tool. The "openstack volume set" command can be used to trigger the migration; I'm not sure whether it supports an option for waiting until the migration is done. In the worst case you'll have to poll the volume state. The might be some restriction, e.g. snapshot presence, which might prevent a volume from being migrated. Automation should also be possible as script or command line. We did a similar migration in the past, migrating from one ceph cluster to another; we had to update some stuff in the cinder database afterwards since they were still referring to the old cluster. You might want to test this first... Advantage: - live migration, no downtime necessary for most instances - automation possible Disadvantage: - physical data migration to new pool aka data is being copied - ceph-csi RBD and cephfs via csi in k8s clusters do not use qemu or other layers of indirection; rbds are mapped directly to the host and mounted, cephfs volume are mounted. CSI supports cloning and volume migration, so similar approaches might be possible. I haven't done this in the past, so I cannot comment on this or recommend any approach. Feedback from experiences users is highly welcome. Best regards, Burkhard Linke

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Nico Schottelius 21 Aug '26

21 Aug '26

Good morning list, some of you already mentioned the issue in relation to proxmox, but virtually all VPS/VM hostings are affected that are running qemu/kvm. The situation is even worse, if I see it correctly: a) All VPS/VMs need to be restarted To be able to rotate keys, aside from the almost impossible flow to actually doing the rotation (more to that below), this will affect every consumer of VMs. The usual setup is that all hypervisors are using the same key. b) Key support in qemu/kvm From my understanding, qemu/kvm + librbd does at the moment not have support for the new key type. Thus there will be major waiting periods until the packages have been updated and eventually make it back into the distros. If they are being done at all, depending on your setup there might be old qemu versions which cannot easily be upgraded. c) Key rotation Assuming that the support actually has landed in qemu/kvm (and that can still take months), one will need to rotate potentially update/edit generade libvirtd xml that contains the reference to the ceph key or the key itself. I'm not saying anything cannot be done, but the way how this vulnerabilty is solved will cause major headache around the industry. BR, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Dan van der Ster 21 Aug '26

21 Aug '26

Heads up: large clusters might crash during the first steps of the upgrade to v19.2.6. See https://tracker.ceph.com/issues/77610#note-9 On a cluster with ~1400 OSDs, upgrading from 18.2.8 to 19.2.6, the squid mgr's crash when reading the osdmap .mgr pool read_balance_score = inf. Workaround is to increase the number of pgs on the .mgr pool to 8 or 16. ( Patrick is this why you merged this a few hours ago? https://github.com/ceph/ceph/pull/60063 ) Cheers, Dan [1] 2026-08-21T01:37:37.329+0000 7a754d6b0640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'devicehealth' while running on mgr.xxxx.zgsszn: Expecting value: line 1 column 2403 (char 2402) 2026-08-21T01:37:37.329+0000 7a754d6b0640 -1 devicehealth.serve: 2026-08-21T01:37:37.329+0000 7a754d6b0640 -1 Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 408, in serve self._do_serve() File "/usr/share/ceph/mgr/mgr_module.py", line 551, in check return func(self, *args, **kwargs) File "/usr/share/ceph/mgr/devicehealth/module.py", line 364, in _do_serve if self.db_ready() and self.enable_monitoring: File "/usr/share/ceph/mgr/mgr_module.py", line 1314, in db_ready return self.db is not None File "/usr/share/ceph/mgr/mgr_module.py", line 1326, in db self._db = self.open_db() File "/usr/share/ceph/mgr/mgr_module.py", line 1293, in open_db if not self.pool_exists(self.MGR_POOL_NAME): File "/usr/share/ceph/mgr/mgr_module.py", line 1120, in pool_exists pools = [p['pool_name'] for p in self.get('osd_map')['pools']] File "/usr/share/ceph/mgr/mgr_module.py", line 1444, in get obj = json.loads(obj) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 2403 (char 2402) On Wed, Aug 19, 2026 at 9:05 AM Patrick Donnelly <ceph-users(a)ceph.io> wrote: > > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Burkhard Linke 21 Aug '26

21 Aug '26

Hi, On 8/20/26 21:35, Patrick Donnelly wrote: >> A challenge to that approach is that Proxmox PVE defines external RBD >> pools at the datacenter level using a single user/key pair. This then >> becomes the storage pool on each individual node in the proxmox >> cluster. I believe ceph-csi K8 storage class also defines this once >> cluster wide. > I understand but the only way to change things is to spread the word. > If the key is that challenging to rotate, then I would suggest leaving > it as aes and push for changes in the product or deployment > technology. Rook developers are already aware this should be reworked. So you expect every user running Proxmox PVE, K8s with ceph csi or Openstack (which is usually more than just a handful of hosts) to figure out how to adopt their deployment? Which is not possible at this time for the mentioned cases. Most users will probably welcome a more detailed documentation on how to handle key rotation _and_ especially how to manage the client side in this scenario. As was already mentioned before, most of us cannot simply shutdown all virtual machines in Proxmox or Openstack for a key rotation.... Best regards, Burkhard Linke

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Jan Kasprzak 21 Aug '26

21 Aug '26

Hi William and all, I ran into the same, thanks for the suggested fix. FWIW, the readlink calls in your code are supefluous, one can simply use the symlink itself (after all, this is the purpose of symlinks :-): OSD_ID=123 KEY=`ceph auth get-key osd.$OSD_ID` ssh root@... "ceph-bluestore-tool set-label-key --dev /var/lib/ceph/osd/ceph-$OSD_ID/block -k osd_key -v $KEY" -Yenya William Xiong wrote: > Hi Patrick, > > Regarding the fix for CVE-2025-30156 > > We've discovered that the bluestore label still contains the old key after following instructions at > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > Old key shows up as per: > ceph-bluestore-tool show-label --dev "$(readlink -f /var/lib/ceph/osd/ceph-X/block)" | grep osd_key > > > If we reboot the osd servers, the new key is overwritten with the old key in /var/lib/ceph/osd/ceph-${OSD_ID}/keyring causing osd auth failure > osd service doesnt start > Can simulate this without reboot via `umount /var/lib/ceph/osd/ceph-${OSD_ID}/` and reactivate via `ceph-volume lvm activate --all` > We are running a non cephadm cluster. > > > Need to update the label as well after rotating the key: > ceph-bluestore-tool set-label-key --dev "$(readlink -f /var/lib/ceph/osd/ceph-${OSD_ID}/block)" -k osd_key -v $NEW_KEY > > Kind Regards, > > > William Xiong | Infrastructure Engineer - Storage > > Research Computing Services > > The University of Melbourne, Victoria 3010 Australia > > E: william.xiong(a)unimelb.edu.au<mailto:william.xiong@unimelb.edu.au> > > ________________________________ > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Thursday, 20 August 2026 2:08 AM > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released > > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Stephan Hohn 21 Aug '26

21 Aug '26

I ran into a problem upgrading encrypted OSDs with cephadm on a hardened node where /tmp is mounted with noexec. The rotate helper script will fail on such setups. podman run --privileged --entrypoint /tmp/cryptsetup_action.sh \ -v /tmp/cephadm-osd-14-rotate-helper.sh:/tmp/cryptsetup_action.sh <image> → ERROR (catatonit:2): failed to exec pid1: Permission denied → ERROR: Got error rotating osd keyring while using cryptsetup tool → cephadm exited with an error code: 1 A temporary remount of /tmp which allowed exec solved it for me. BR Stephan Hohn IT Technical Leader stephan(a)gridscale.io gridscale GmbH Oskar-Jäger-Str. 173 50825 Köln www.gridscale.io HRB 97235 Amtsgericht Köln Vertreten durch: Marc Dittmann, Yaniv Fdida Am Mi., 19. Aug. 2026 um 18:08 Uhr schrieb Patrick Donnelly < ceph-users(a)ceph.io>: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: > https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 21 Aug '26

21 Aug '26

Hi Jan, On Thu, Aug 20, 2026 at 10:54 AM Jan Kasprzak <ceph-users(a)ceph.io> wrote: > > Hi all, > > Patrick Donnelly wrote: > > Today, August 19th, 2026, the Ceph project has released Tentacle > > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > > components. > > I have just upgraded one of my cluster from 20.2.3 to 20.2.4. > The documentation seems to be clear (thanks!) and worked for me > with the exception of step 4: > > > 4. Confirm the AUTH_INSECURE_SERVICE_KEY_TYPE is cleared. > > > > ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_SERVICE_KEY_TYPE") | not' > > > > output gives false. > > > > If it outputs true, there is another daemon that needs to be upgraded. > > Check the output of ceph health detail. > > I might have done something wrong, but if I remember it correctly, > at this point I still got 4 AUTH_INSECURE_SERVICE_KEY_TYPE keys, > and had to do step 5 "ceph mon set auth_service_cipher aes256k" first > and then step 6 "ceph auth wipe-rotating-service-keys". Only then > the AUTH_INSECURE_SERVICE_KEY_TYPE disappeared from ceph health detail > (together with AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE). AUTH_INSECURE_SERVICE_KEY_TYPE indicates that the service, e.g. osd.0, is using an aes key to authenticate with the Monitors. That can only be addressed by rotating that service's key. The other health error is AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE which is addressed by the auth_service_cipher monitor setting (required) and wiping the rotating key database (optional). > One question, though: is it possible to have a client with two keys, > aes and aes256? One of my cluster is mainly used for RBD with client.libvirt > user, and I have hundreds of running VMs, which I cannot restart at once. > So I would like to _add_ an aes256 key to the client.libvirt user, > and incrementally restart the qemu processes which use this secret. No, you cannot have two keys. I think a lesson (for everyone) here is that each node should have its own client key. It became prevalent to use the same key everywhere for simplicity but it makes deployments and updates more challenging. > Or what is the recommended way of gradually replacing a client key? Create new keys for each node. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Maximilian Stinsky-Damke 21 Aug '26

21 Aug '26

Hi Adam, I tried following your guidance regarding rotating an OSD lockbox key via ceph auth rotate and then redeploying the OSD via ceph orch redeploy in a test cluster. It seems that cephadm does not update the lockbox keyring during a redeploy, which results in the OSDs failing to start with the new keyring because they cannot unlock their dm-crypt volumes. To restore my test cluster, I had to manually update the lockbox.keyring for each OSD with the new key. Is this a bug in cephadm, or am I missing something here? Best regards, Max ________________________________ From: Adam King <ceph-users(a)ceph.io> Sent: 20 August 2026 15:46 To: Pedro Macedo <pmacedo(a)pmacedo.com> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released Warning: This email is from an unusual correspondent. Warning: Make sure this is someone you trust. That command got removed for the time being as it used an old way of doing the key rotation that wasn't built to work with the changes for the cve and seemed to be broken for a number of daemon types. Since the name would make people want to use it it was decided to just temporarily disables it so we can fix it and reintroduce it in a new version as an easier way to rotate the keys. For any keys tied to a cephadm deployed daemon that aren't automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using nfs exports managed through the nfs module the export's keyrings also need to be rotated. `ceph auth rotate <export-keyring-entity> --key-type aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. Between rotating the keyrings for the nfs cluster/daemons and rotating the keyring for the individual export(s) those export(s) are unavailable iirc. On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > > On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: > > Hi, > > > > I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The > following cephadm managed client keys were not rotated automatically: > > > > client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) > > client.ceph-exporter. > > client.crash. > > client.osd-lockbox.. > > client.rgw. > > > > The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* > this command does not exist. > > Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing > there, but present on 20.2.3. > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Fsquid%2F… > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph%2Fbl… > > > > What is the correct way to rotate these client keys via cephadm as part > of the cve fix? > > > > Thanks for any guidance. > > Best Regards > > Max > > > > ________________________________ > > From: Patrick Donnelly <ceph-users(a)ceph.io> > > Sent: 19 August 2026 18:08 > > To: ceph-users <ceph-users(a)ceph.io> > > Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 > released > > > > Warning: This email is from an unusual correspondent. > > Warning: Make sure this is someone you trust. > > > > Today, August 19th, 2026, the Ceph project has released Tentacle > > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > > components. > > > > This is the sixth backport release in the Squid series. > > > > This is the fourth backport release in the Tentacle series. > > > > Additional details will be posted on the Ceph website/blog: > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… > > > > We strongly recommend that all Ceph operators upgrade to one of these > > releases as soon as possible. > > > > > > ## Critical Upgrade Steps > > > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > > This is the first time Ceph has introduced a new key type for CephX > > credentials. Therefore, a new procedure exists for upgrading and > > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > > cluster, the operator should familiarize themselves with the procedure > > documented at: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > These steps are for package-based deployments of Ceph. In other > > sections, the document also includes new information about key > > rotation procedures, monitor authentication controls, and emergency > > recovery procedures. > > > > Deployments using cephadm will automate the process except for client > > keys. You may notice that Cephadm spends more time than normal on the > > upgrade after updating all daemon images. This is due to a new process > > rotating the OSD and MDS keys. Please be patient. There are plans to > > improve this in the future. > > > > Deployments using Rook will also automate rotation of some client keys > > with some exclusions. Look to the Rook project’s announcement for more > > details. > > > > Client and kernel upgrades are recommended to support aes256k. Note > > that upstream Linux kernel client support began in kernel 7.0 and has > > been backported to CentOS Stream 9 and 10. Check with your > > distribution vendor for backported key support before rotating client > > keys used by the kernel. > > > > As part of upgrading a Ceph cluster, six new health warnings and > > errors will be generated. This is normal. As part of the process to > > upgrade/rotate entity keys, you will address the warnings and errors > > as you go. These are documented beginning here: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > The CephX documentation has been brought up to date and vetted for > > accuracy. Those changes are live here: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > > requests with `host` and `x-amz-` headers not included in the signed > > subset. Unfortunately, the REST client used in multisite was > > generating such improperly signed requests. > > > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > > option to true before you begin to upgrade. After all clusters are > > upgraded, set the option to ``false`` again. > > > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > > secrets stored in the Monitor config-key store will be forthcoming. In > > the meantime, operators of cephadm-managed clusters are strongly > > encouraged to rotate the cephadm SSH key, as an established process > > exists: > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > Rotating other secrets is advised but the process is not yet > > established. Operators should assess their cluster's potential > > exposure to determine if immediate Monitor config-key secret rotation > > is required for their environment. > > > > > > ## CVE Fixes > > > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > > of AES-CBC. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > > in improper verification of a cryptographic signature. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > > subscription handler. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > > cryptographic signatures in RGW's SigV4 verifier. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > ## Other Media > > > > A Cephalocon 2025 talk covers some of the details surrounding the new > > CephX key upgrade process. > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… > > > > ## Getting Ceph > > ------------ > > * Source at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… > > * v19.2.6 Tarball at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > > * v20.2.4 Tarball at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > > * Containers at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… > > * For packages, see > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * For upgrade with cephadm, see: > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > > > -- > > Patrick Donnelly, Ph.D. > > He / Him / His > > Red Hat Partner Engineer > > IBM, Inc. > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > This message has been checked by Libraesva ESG and is found to be clean. > > Follow this link to report as spam/bad: > > https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad > > Follow this link to blocklist sender: > > https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- This message has been checked by Libraesva ESG and is found to be clean. Follow this link to report as spam/bad: https://mx10.wiit.cloud/action/4hQl8t5ktzzTkbn/report-as-bad Follow this link to blocklist sender: https://mx10.wiit.cloud/action/4hQl8t5ktzzTkbn/blocklist

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 20 Aug '26

20 Aug '26

On Thu, Aug 20, 2026 at 3:51 PM Salatiel Filho <ceph-users(a)ceph.io> wrote: > entity client.bootstrap-mds using insecure key type: aes > entity client.bootstrap-mgr using insecure key type: aes > entity client.bootstrap-osd using insecure key type: aes > entity client.bootstrap-rbd using insecure key type: aes > entity client.bootstrap-rbd-mirror using insecure key type: aes > entity client.bootstrap-rgw using insecure key type: aes Bootstrap keys should be safe to delete. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Question on so-called tenant level roles in tentacle
by Manuel Rios - EDH 20 Aug '26

20 Aug '26

I don’t know how this patch become online without an auto-rotation... -----Mensaje original----- De: Casey Bodley <ceph-users(a)ceph.io> Enviado el: Thursday, August 20, 2026 7:15 PM Para: Christopher Durham <caduceus42(a)aol.com> CC: ceph-users(a)ceph.io Asunto: [ceph-users] Re: Question on so-called tenant level roles in tentacle On Wed, Aug 19, 2026 at 5:20 PM Christopher Durham <ceph-users(a)ceph.io> wrote: > > > The release notes for ceph tentacle, at: > https://docs.ceph.com/en/latest/releases/tentacle/ > state the following > "RGW: The User Account feature introduced in Squid provides first-class support for IAM APIs and policy. Our preliminary STS support was based on tenants, and exposed some IAM APIs to admins only. This tenant-level IAM functionality is now deprecated in favor of accounts. While we’ll continue to support the tenant feature itself for namespace isolation, the following features will be removed no sooner than the V release: Tenant-level IAM APIs including CreateRole, PutRolePolicy and PutUserPolicy, Use of tenant names instead of accounts in IAM policy documents, Interpretation of IAM policy without cross-account policy evaluation, S3 API support for cross-tenant names such as Bucket=’tenant:bucketname’ STS Lite and sts:GetSessionToken." > > Does the 'V' release mean the Vampire release? Does this statement mean that those API calls will be available in all tentacle releases and umbrella, but maybe not vampire? that's right, Chris > I need to have the CreateRole,PutRolePolicy et al available in tentacle and do a planned transition to IAM users/roles. Any documentation on how to do this? I have STS roles for various groups of users to gain temporary access to solme number of buckets and I want to be sure my upgrade to tentacle will be ok. (I am on 19.2.5 squid). We are not using tenants but buckets owned by various s3 users and roles to give temporary access to buckets by other users. We have been doing this since either nautilus or octopus and it has worked well for us. (We do not use the gui for role management but have our own python scripts to add users/buckets to roles). https://docs.ceph.com/en/squid/radosgw/account/#migrate-an-existing-user-in… shows how to migrate existing users into an account, but not how to migrate existing roles. you'll need to create new roles in accounts and delete the old ones because the initial sts implementation treated the tenant like an account, the most direct mapping would be to create one account per tenant (in your case just the global/unnamed tenant) and migrate all of that tenant's users into it. you could then recreate the roles in that account, with role policy that names the same buckets and user principals. however, all buckets would be owned by the account and no users would have any access by default. so you'd have to use iam policy to control all permissions, not just to facilitate sharing between users/buckets if there are distinct groups of ownership/sharing, it would make more sense to create separate accounts for each. users within each account could be granted direct access to that account's buckets, and account roles could be used to share access to some of those buckets with other trusted accounts/users. you'd also have the option to give the account root user's credentials to someone in that group so they could use the iam apis to manage the sharing of their own resources. it's this self-service aspect of accounts that most differentiates it from the tenant-level sts support, which was an admin-only feature if you do decide to put all of the users in a single account, you might consider using iam groups rather than roles. you could use group policy to grant access to specific buckets in the account, then manage group membership with iam:AddUserToGroup/RemoveUserFromGroup. this would avoid the need for role trust policy to name all of the individual users, and those users wouldn't need to go through AssumeRole for that shared access > -Chris > > > > > > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Question on so-called tenant level roles in tentacle
by Casey Bodley 20 Aug '26

20 Aug '26

On Wed, Aug 19, 2026 at 5:20 PM Christopher Durham <ceph-users(a)ceph.io> wrote: > > > The release notes for ceph tentacle, at: > https://docs.ceph.com/en/latest/releases/tentacle/ > state the following > "RGW: The User Account feature introduced in Squid provides first-class support for IAM APIs and policy. Our preliminary STS support was based on tenants, and exposed some IAM APIs to admins only. This tenant-level IAM functionality is now deprecated in favor of accounts. While we’ll continue to support the tenant feature itself for namespace isolation, the following features will be removed no sooner than the V release: Tenant-level IAM APIs including CreateRole, PutRolePolicy and PutUserPolicy, Use of tenant names instead of accounts in IAM policy documents, Interpretation of IAM policy without cross-account policy evaluation, S3 API support for cross-tenant names such as Bucket=’tenant:bucketname’ STS Lite and sts:GetSessionToken." > > Does the 'V' release mean the Vampire release? Does this statement mean that those API calls will be available in all tentacle releases and umbrella, but maybe not vampire? that's right, Chris > I need to have the CreateRole,PutRolePolicy et al available in tentacle and do a planned transition to IAM users/roles. Any documentation on how to do this? I have STS roles for various groups of users to gain temporary access to solme number of buckets and I want to be sure my upgrade to tentacle will be ok. (I am on 19.2.5 squid). We are not using tenants but buckets owned by various s3 users and roles to give temporary access to buckets by other users. We have been doing this since either nautilus or octopus and it has worked well for us. (We do not use the gui for role management but have our own python scripts to add users/buckets to roles). https://docs.ceph.com/en/squid/radosgw/account/#migrate-an-existing-user-in… shows how to migrate existing users into an account, but not how to migrate existing roles. you'll need to create new roles in accounts and delete the old ones because the initial sts implementation treated the tenant like an account, the most direct mapping would be to create one account per tenant (in your case just the global/unnamed tenant) and migrate all of that tenant's users into it. you could then recreate the roles in that account, with role policy that names the same buckets and user principals. however, all buckets would be owned by the account and no users would have any access by default. so you'd have to use iam policy to control all permissions, not just to facilitate sharing between users/buckets if there are distinct groups of ownership/sharing, it would make more sense to create separate accounts for each. users within each account could be granted direct access to that account's buckets, and account roles could be used to share access to some of those buckets with other trusted accounts/users. you'd also have the option to give the account root user's credentials to someone in that group so they could use the iam apis to manage the sharing of their own resources. it's this self-service aspect of accounts that most differentiates it from the tenant-level sts support, which was an admin-only feature if you do decide to put all of the users in a single account, you might consider using iam groups rather than roles. you could use group policy to grant access to specific buckets in the account, then manage group membership with iam:AddUserToGroup/RemoveUserFromGroup. this would avoid the need for role trust policy to name all of the individual users, and those users wouldn't need to go through AssumeRole for that shared access > -Chris > > > > > > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Salatiel Filho 20 Aug '26

20 Aug '26

Hi, I also miss in the docs the steps for those clients that are not daemons? ( cephadm managed ) entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes Atenciosamente/Kind regards, Salatiel On Thu, Aug 20, 2026 at 11:00 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > > Gotcha; in that case, sounds like the documentation and/or the > CVE-related docs should be updated to reflect these changes and steps > related to non- mgr/mon/osd/mds daemons. > > On 2026-08-20 3:43 PM, Adam King wrote: > > That command got removed for the time being as it used an old way of doing > > the key rotation that wasn't built to work with the changes for the cve and > > seemed to be broken for a number of daemon types. Since the name would make > > people want to use it it was decided to just temporarily disables it so we > > can fix it and reintroduce it in a new version as an easier way to rotate > > the keys. For any keys tied to a cephadm deployed daemon that aren't > > automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings > > during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type > > aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using > > nfs exports managed through the nfs module the export's keyrings also need > > to be rotated. `ceph auth rotate <export-keyring-entity> --key-type > > aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > > > export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. > > Between rotating the keyrings for the nfs cluster/daemons and rotating the > > keyring for the individual export(s) those export(s) are unavailable iirc. > > > > On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > > > >> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: > >>> Hi, > >>> > >>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The > >> following cephadm managed client keys were not rotated automatically: > >>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) > >>> client.ceph-exporter. > >>> client.crash. > >>> client.osd-lockbox.. > >>> client.rgw. > >>> > >>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* > >> this command does not exist. > >> > >> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing > >> there, but present on 20.2.3. > >> > >> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… > >> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… > >>> What is the correct way to rotate these client keys via cephadm as part > >> of the cve fix? > >>> Thanks for any guidance. > >>> Best Regards > >>> Max > >>> > >>> ________________________________ > >>> From: Patrick Donnelly <ceph-users(a)ceph.io> > >>> Sent: 19 August 2026 18:08 > >>> To: ceph-users <ceph-users(a)ceph.io> > >>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 > >> released > >>> Warning: This email is from an unusual correspondent. > >>> Warning: Make sure this is someone you trust. > >>> > >>> Today, August 19th, 2026, the Ceph project has released Tentacle > >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > >>> components. > >>> > >>> This is the sixth backport release in the Squid series. > >>> > >>> This is the fourth backport release in the Tentacle series. > >>> > >>> Additional details will be posted on the Ceph website/blog: > >>> > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… > >>> We strongly recommend that all Ceph operators upgrade to one of these > >>> releases as soon as possible. > >>> > >>> > >>> ## Critical Upgrade Steps > >>> > >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > >>> This is the first time Ceph has introduced a new key type for CephX > >>> credentials. Therefore, a new procedure exists for upgrading and > >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > >>> cluster, the operator should familiarize themselves with the procedure > >>> documented at: > >>> > >>> > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> These steps are for package-based deployments of Ceph. In other > >>> sections, the document also includes new information about key > >>> rotation procedures, monitor authentication controls, and emergency > >>> recovery procedures. > >>> > >>> Deployments using cephadm will automate the process except for client > >>> keys. You may notice that Cephadm spends more time than normal on the > >>> upgrade after updating all daemon images. This is due to a new process > >>> rotating the OSD and MDS keys. Please be patient. There are plans to > >>> improve this in the future. > >>> > >>> Deployments using Rook will also automate rotation of some client keys > >>> with some exclusions. Look to the Rook project’s announcement for more > >>> details. > >>> > >>> Client and kernel upgrades are recommended to support aes256k. Note > >>> that upstream Linux kernel client support began in kernel 7.0 and has > >>> been backported to CentOS Stream 9 and 10. Check with your > >>> distribution vendor for backported key support before rotating client > >>> keys used by the kernel. > >>> > >>> As part of upgrading a Ceph cluster, six new health warnings and > >>> errors will be generated. This is normal. As part of the process to > >>> upgrade/rotate entity keys, you will address the warnings and errors > >>> as you go. These are documented beginning here: > >>> > >>> > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> The CephX documentation has been brought up to date and vetted for > >>> accuracy. Those changes are live here: > >>> > >>> > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > >>> requests with `host` and `x-amz-` headers not included in the signed > >>> subset. Unfortunately, the REST client used in multisite was > >>> generating such improperly signed requests. > >>> > >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` > >>> option to true before you begin to upgrade. After all clusters are > >>> upgraded, set the option to ``false`` again. > >>> > >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all > >>> secrets stored in the Monitor config-key store will be forthcoming. In > >>> the meantime, operators of cephadm-managed clusters are strongly > >>> encouraged to rotate the cephadm SSH key, as an established process > >>> exists: > >>> > >>> See: > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> Rotating other secrets is advised but the process is not yet > >>> established. Operators should assess their cluster's potential > >>> exposure to determine if immediate Monitor config-key secret rotation > >>> is required for their environment. > >>> > >>> > >>> ## CVE Fixes > >>> > >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > >>> of AES-CBC. > >>> > >>> See: > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of > >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting > >>> in improper verification of a cryptographic signature. > >>> > >>> See: > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > >>> subscription handler. > >>> > >>> See: > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > >>> cryptographic signatures in RGW's SigV4 verifier. > >>> > >>> See: > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> ## Other Media > >>> > >>> A Cephalocon 2025 talk covers some of the details surrounding the new > >>> CephX key upgrade process. > >>> > >>> > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… > >>> ## Getting Ceph > >>> ------------ > >>> * Source at > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… > >>> * v19.2.6 Tarball at > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > >>> * v20.2.4 Tarball at > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > >>> * Containers at > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… > >>> * For packages, see > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> * For upgrade with cephadm, see: > >>> > >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > >>> > >>> -- > >>> Patrick Donnelly, Ph.D. > >>> He / Him / His > >>> Red Hat Partner Engineer > >>> IBM, Inc. > >>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> > >>> -- > >>> This message has been checked by Libraesva ESG and is found to be clean. > >>> Follow this link to report as spam/bad: > >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad > >>> Follow this link to blocklist sender: > >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist > >>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 20 Aug '26

20 Aug '26

Hi Gary, On Thu, Aug 20, 2026 at 2:31 PM Gary Molenkamp <ceph-users(a)ceph.io> wrote: > > Hi Patrick > >> One question, though: is it possible to have a client with two keys, > >> aes and aes256? One of my cluster is mainly used for RBD with client.libvirt > >> user, and I have hundreds of running VMs, which I cannot restart at once. > >> So I would like to _add_ an aes256 key to the client.libvirt user, > >> and incrementally restart the qemu processes which use this secret. > > No, you cannot have two keys. I think a lesson (for everyone) here is > > that each node should have its own client key. It became prevalent to > > use the same key everywhere for simplicity but it makes deployments > > and updates more challenging. > > > >> Or what is the recommended way of gradually replacing a client key? > > Create new keys for each node. > > > > A challenge to that approach is that Proxmox PVE defines external RBD > pools at the datacenter level using a single user/key pair. This then > becomes the storage pool on each individual node in the proxmox > cluster. I believe ceph-csi K8 storage class also defines this once > cluster wide. I understand but the only way to change things is to spread the word. If the key is that challenging to rotate, then I would suggest leaving it as aes and push for changes in the product or deployment technology. Rook developers are already aware this should be reworked. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Gary Molenkamp 20 Aug '26

20 Aug '26

Hi Patrick >> One question, though: is it possible to have a client with two keys, >> aes and aes256? One of my cluster is mainly used for RBD with client.libvirt >> user, and I have hundreds of running VMs, which I cannot restart at once. >> So I would like to _add_ an aes256 key to the client.libvirt user, >> and incrementally restart the qemu processes which use this secret. > No, you cannot have two keys. I think a lesson (for everyone) here is > that each node should have its own client key. It became prevalent to > use the same key everywhere for simplicity but it makes deployments > and updates more challenging. > >> Or what is the recommended way of gradually replacing a client key? > Create new keys for each node. > A challenge to that approach is that Proxmox PVE defines external RBD pools at the datacenter level using a single user/key pair. This then becomes the storage pool on each individual node in the proxmox cluster. I believe ceph-csi K8 storage class also defines this once cluster wide. Gary

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Jan Kasprzak 20 Aug '26

20 Aug '26

Hi all, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. I have just upgraded one of my cluster from 20.2.3 to 20.2.4. The documentation seems to be clear (thanks!) and worked for me with the exception of step 4: > 4. Confirm the AUTH_INSECURE_SERVICE_KEY_TYPE is cleared. > > ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_SERVICE_KEY_TYPE") | not' > > output gives false. > > If it outputs true, there is another daemon that needs to be upgraded. > Check the output of ceph health detail. I might have done something wrong, but if I remember it correctly, at this point I still got 4 AUTH_INSECURE_SERVICE_KEY_TYPE keys, and had to do step 5 "ceph mon set auth_service_cipher aes256k" first and then step 6 "ceph auth wipe-rotating-service-keys". Only then the AUTH_INSECURE_SERVICE_KEY_TYPE disappeared from ceph health detail (together with AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE). One question, though: is it possible to have a client with two keys, aes and aes256? One of my cluster is mainly used for RBD with client.libvirt user, and I have hundreds of running VMs, which I cannot restart at once. So I would like to _add_ an aes256 key to the client.libvirt user, and incrementally restart the qemu processes which use this secret. Or what is the recommended way of gradually replacing a client key? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 20 Aug '26

20 Aug '26

Can anyone confirm whether OSDs that were adopted into cephadm but remain unmanaged will be automatically rotated? (Currently on 20.2.3). On 20/08/2026 14:45, Adam King wrote: > That command got removed for the time being as it used an old way of doing > the key rotation that wasn't built to work with the changes for the cve and > seemed to be broken for a number of daemon types. Since the name would make > people want to use it it was decided to just temporarily disables it so we > can fix it and reintroduce it in a new version as an easier way to rotate > the keys. For any keys tied to a cephadm deployed daemon that aren't > automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings > during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type > aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using > nfs exports managed through the nfs module the export's keyrings also need > to be rotated. `ceph auth rotate <export-keyring-entity> --key-type > aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > > export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. > Between rotating the keyrings for the nfs cluster/daemons and rotating the > keyring for the individual export(s) those export(s) are unavailable iirc. > > On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > >> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: >>> Hi, >>> >>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The >> following cephadm managed client keys were not rotated automatically: >>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) >>> client.ceph-exporter. >>> client.crash. >>> client.osd-lockbox.. >>> client.rgw. >>> >>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* >> this command does not exist. >> >> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing >> there, but present on 20.2.3. >> >> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… >> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… >>> What is the correct way to rotate these client keys via cephadm as part >> of the cve fix? >>> Thanks for any guidance. >>> Best Regards >>> Max >>> >>> ________________________________ >>> From: Patrick Donnelly <ceph-users(a)ceph.io> >>> Sent: 19 August 2026 18:08 >>> To: ceph-users <ceph-users(a)ceph.io> >>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 >> released >>> Warning: This email is from an unusual correspondent. >>> Warning: Make sure this is someone you trust. >>> >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… >>> ## Getting Ceph >>> ------------ >>> * Source at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… >>> * v19.2.6 Tarball at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>> * v20.2.4 Tarball at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>> * Containers at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… >>> * For packages, see >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * For upgrade with cephadm, see: >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >>> -- >>> Patrick Donnelly, Ph.D. >>> He / Him / His >>> Red Hat Partner Engineer >>> IBM, Inc. >>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> -- >>> This message has been checked by Libraesva ESG and is found to be clean. >>> Follow this link to report as spam/bad: >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad >>> Follow this link to blocklist sender: >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Pedro Macedo 20 Aug '26

20 Aug '26

Gotcha; in that case, sounds like the documentation and/or the CVE-related docs should be updated to reflect these changes and steps related to non- mgr/mon/osd/mds daemons. On 2026-08-20 3:43 PM, Adam King wrote: > That command got removed for the time being as it used an old way of doing > the key rotation that wasn't built to work with the changes for the cve and > seemed to be broken for a number of daemon types. Since the name would make > people want to use it it was decided to just temporarily disables it so we > can fix it and reintroduce it in a new version as an easier way to rotate > the keys. For any keys tied to a cephadm deployed daemon that aren't > automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings > during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type > aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using > nfs exports managed through the nfs module the export's keyrings also need > to be rotated. `ceph auth rotate <export-keyring-entity> --key-type > aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > > export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. > Between rotating the keyrings for the nfs cluster/daemons and rotating the > keyring for the individual export(s) those export(s) are unavailable iirc. > > On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > >> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: >>> Hi, >>> >>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The >> following cephadm managed client keys were not rotated automatically: >>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) >>> client.ceph-exporter. >>> client.crash. >>> client.osd-lockbox.. >>> client.rgw. >>> >>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* >> this command does not exist. >> >> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing >> there, but present on 20.2.3. >> >> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… >> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… >>> What is the correct way to rotate these client keys via cephadm as part >> of the cve fix? >>> Thanks for any guidance. >>> Best Regards >>> Max >>> >>> ________________________________ >>> From: Patrick Donnelly <ceph-users(a)ceph.io> >>> Sent: 19 August 2026 18:08 >>> To: ceph-users <ceph-users(a)ceph.io> >>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 >> released >>> Warning: This email is from an unusual correspondent. >>> Warning: Make sure this is someone you trust. >>> >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… >>> ## Getting Ceph >>> ------------ >>> * Source at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… >>> * v19.2.6 Tarball at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>> * v20.2.4 Tarball at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>> * Containers at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… >>> * For packages, see >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * For upgrade with cephadm, see: >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >>> -- >>> Patrick Donnelly, Ph.D. >>> He / Him / His >>> Red Hat Partner Engineer >>> IBM, Inc. >>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> -- >>> This message has been checked by Libraesva ESG and is found to be clean. >>> Follow this link to report as spam/bad: >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad >>> Follow this link to blocklist sender: >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Adam King 20 Aug '26

20 Aug '26

That command got removed for the time being as it used an old way of doing the key rotation that wasn't built to work with the changes for the cve and seemed to be broken for a number of daemon types. Since the name would make people want to use it it was decided to just temporarily disables it so we can fix it and reintroduce it in a new version as an easier way to rotate the keys. For any keys tied to a cephadm deployed daemon that aren't automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using nfs exports managed through the nfs module the export's keyrings also need to be rotated. `ceph auth rotate <export-keyring-entity> --key-type aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. Between rotating the keyrings for the nfs cluster/daemons and rotating the keyring for the individual export(s) those export(s) are unavailable iirc. On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > > On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: > > Hi, > > > > I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The > following cephadm managed client keys were not rotated automatically: > > > > client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) > > client.ceph-exporter. > > client.crash. > > client.osd-lockbox.. > > client.rgw. > > > > The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* > this command does not exist. > > Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing > there, but present on 20.2.3. > > > > https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… > > > https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… > > > > What is the correct way to rotate these client keys via cephadm as part > of the cve fix? > > > > Thanks for any guidance. > > Best Regards > > Max > > > > ________________________________ > > From: Patrick Donnelly <ceph-users(a)ceph.io> > > Sent: 19 August 2026 18:08 > > To: ceph-users <ceph-users(a)ceph.io> > > Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 > released > > > > Warning: This email is from an unusual correspondent. > > Warning: Make sure this is someone you trust. > > > > Today, August 19th, 2026, the Ceph project has released Tentacle > > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > > components. > > > > This is the sixth backport release in the Squid series. > > > > This is the fourth backport release in the Tentacle series. > > > > Additional details will be posted on the Ceph website/blog: > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… > > > > We strongly recommend that all Ceph operators upgrade to one of these > > releases as soon as possible. > > > > > > ## Critical Upgrade Steps > > > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > > This is the first time Ceph has introduced a new key type for CephX > > credentials. Therefore, a new procedure exists for upgrading and > > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > > cluster, the operator should familiarize themselves with the procedure > > documented at: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > These steps are for package-based deployments of Ceph. In other > > sections, the document also includes new information about key > > rotation procedures, monitor authentication controls, and emergency > > recovery procedures. > > > > Deployments using cephadm will automate the process except for client > > keys. You may notice that Cephadm spends more time than normal on the > > upgrade after updating all daemon images. This is due to a new process > > rotating the OSD and MDS keys. Please be patient. There are plans to > > improve this in the future. > > > > Deployments using Rook will also automate rotation of some client keys > > with some exclusions. Look to the Rook project’s announcement for more > > details. > > > > Client and kernel upgrades are recommended to support aes256k. Note > > that upstream Linux kernel client support began in kernel 7.0 and has > > been backported to CentOS Stream 9 and 10. Check with your > > distribution vendor for backported key support before rotating client > > keys used by the kernel. > > > > As part of upgrading a Ceph cluster, six new health warnings and > > errors will be generated. This is normal. As part of the process to > > upgrade/rotate entity keys, you will address the warnings and errors > > as you go. These are documented beginning here: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > The CephX documentation has been brought up to date and vetted for > > accuracy. Those changes are live here: > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > > requests with `host` and `x-amz-` headers not included in the signed > > subset. Unfortunately, the REST client used in multisite was > > generating such improperly signed requests. > > > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > > option to true before you begin to upgrade. After all clusters are > > upgraded, set the option to ``false`` again. > > > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > > secrets stored in the Monitor config-key store will be forthcoming. In > > the meantime, operators of cephadm-managed clusters are strongly > > encouraged to rotate the cephadm SSH key, as an established process > > exists: > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > Rotating other secrets is advised but the process is not yet > > established. Operators should assess their cluster's potential > > exposure to determine if immediate Monitor config-key secret rotation > > is required for their environment. > > > > > > ## CVE Fixes > > > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > > of AES-CBC. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > > in improper verification of a cryptographic signature. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > > subscription handler. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > > cryptographic signatures in RGW's SigV4 verifier. > > > > See: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > > > ## Other Media > > > > A Cephalocon 2025 talk covers some of the details surrounding the new > > CephX key upgrade process. > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… > > > > ## Getting Ceph > > ------------ > > * Source at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… > > * v19.2.6 Tarball at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > > * v20.2.4 Tarball at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > > * Containers at > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… > > * For packages, see > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * For upgrade with cephadm, see: > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > > > -- > > Patrick Donnelly, Ph.D. > > He / Him / His > > Red Hat Partner Engineer > > IBM, Inc. > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > This message has been checked by Libraesva ESG and is found to be clean. > > Follow this link to report as spam/bad: > > https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad > > Follow this link to blocklist sender: > > https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 20 Aug '26

20 Aug '26

On Thu, Aug 20, 2026 at 4:33 AM Iztok Gregori <ceph-users(a)ceph.io> wrote: > Silly question maybe, but I want to be sure, for RBD clients (Proxmox > QEMU virtual machines in my case) the new key type needs updated "client > libraries" to be "understood", right? If you're using the rbd kernel driver, the kernel must be updated. If you also rely on userspace libraries or mount helpers, those need updated too. The main focus should be on upgrading the Ceph service daemons and rotating keys. Once that's complete, the main vulnerability can be considered closed. Upgrading client keys is still encouraged where possible. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Alex Gorbachev 20 Aug '26

20 Aug '26

We have advised Proxmox of this issue, and you will need to wait until the appropriate versions of Ceph are distributed in an upcoming PVE update. -- Best regards, Alex Gorbachev ISS/Storcium On Thu, Aug 20, 2026 at 4:33 AM Iztok Gregori <ceph-users(a)ceph.io> wrote: > Hi! > > Silly question maybe, but I want to be sure, for RBD clients (Proxmox > QEMU virtual machines in my case) the new key type needs updated "client > libraries" to be "understood", right? > > Cheers > Iztok > > On 19/08/26 18:07, Patrick Donnelly wrote: > > Today, August 19th, 2026, the Ceph project has released Tentacle > > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > > components. > > > > This is the sixth backport release in the Squid series. > > > > This is the fourth backport release in the Tentacle series. > > > > Additional details will be posted on the Ceph website/blog: > > https://ceph.io/en/news/blog/ > > > > We strongly recommend that all Ceph operators upgrade to one of these > > releases as soon as possible. > > > > > > ## Critical Upgrade Steps > > > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > > This is the first time Ceph has introduced a new key type for CephX > > credentials. Therefore, a new procedure exists for upgrading and > > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > > cluster, the operator should familiarize themselves with the procedure > > documented at: > > > > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > > > These steps are for package-based deployments of Ceph. In other > > sections, the document also includes new information about key > > rotation procedures, monitor authentication controls, and emergency > > recovery procedures. > > > > Deployments using cephadm will automate the process except for client > > keys. You may notice that Cephadm spends more time than normal on the > > upgrade after updating all daemon images. This is due to a new process > > rotating the OSD and MDS keys. Please be patient. There are plans to > > improve this in the future. > > > > Deployments using Rook will also automate rotation of some client keys > > with some exclusions. Look to the Rook project’s announcement for more > > details. > > > > Client and kernel upgrades are recommended to support aes256k. Note > > that upstream Linux kernel client support began in kernel 7.0 and has > > been backported to CentOS Stream 9 and 10. Check with your > > distribution vendor for backported key support before rotating client > > keys used by the kernel. > > > > As part of upgrading a Ceph cluster, six new health warnings and > > errors will be generated. This is normal. As part of the process to > > upgrade/rotate entity keys, you will address the warnings and errors > > as you go. These are documented beginning here: > > > > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > > > The CephX documentation has been brought up to date and vetted for > > accuracy. Those changes are live here: > > > > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > > requests with `host` and `x-amz-` headers not included in the signed > > subset. Unfortunately, the REST client used in multisite was > > generating such improperly signed requests. > > > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > > option to true before you begin to upgrade. After all clusters are > > upgraded, set the option to ``false`` again. > > > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > > secrets stored in the Monitor config-key store will be forthcoming. In > > the meantime, operators of cephadm-managed clusters are strongly > > encouraged to rotate the cephadm SSH key, as an established process > > exists: > > > > See: > https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > > > Rotating other secrets is advised but the process is not yet > > established. Operators should assess their cluster's potential > > exposure to determine if immediate Monitor config-key secret rotation > > is required for their environment. > > > > > > ## CVE Fixes > > > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > > of AES-CBC. > > > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > > in improper verification of a cryptographic signature. > > > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > > subscription handler. > > > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > > cryptographic signatures in RGW's SigV4 verifier. > > > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > > > ## Other Media > > > > A Cephalocon 2025 talk covers some of the details surrounding the new > > CephX key upgrade process. > > > > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > > > ## Getting Ceph > > ------------ > > * Source at https://github.com/ceph/ceph > > * v19.2.6 Tarball at > https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > > * v20.2.4 Tarball at > https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > > * Containers at https://quay.io/repository/ceph/ceph > > * For packages, see > https://docs.ceph.com/en/latest/install/get-packages/ > > * For upgrade with cephadm, see: > > https://docs.ceph.com/en/latest/cephadm/upgrade/ > > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Maximilian Stinsky-Damke 20 Aug '26

20 Aug '26

I linked the wrong documentation regarding the ceph orch daemon rotate-key command, here is the correct link for the command that does not work in 19.2.x: https://docs.ceph.com/en/squid/cephadm/operations/#rotating-a-daemon-s-auth… Best Regards ________________________________ From: Maximilian Stinsky-Damke <ceph-users(a)ceph.io> Sent: 20 August 2026 13:00 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released Warning: This email is from an unusual correspondent. Warning: Make sure this is someone you trust. Hi, I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The following cephadm managed client keys were not rotated automatically: client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) client.ceph-exporter. client.crash. client.osd-lockbox.. client.rgw. The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* this command does not exist. https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Fsquid%2F… https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph%2Fbl… What is the correct way to rotate these client keys via cephadm as part of the cve fix? Thanks for any guidance. Best Regards Max ________________________________ From: Patrick Donnelly <ceph-users(a)ceph.io> Sent: 19 August 2026 18:08 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released Warning: This email is from an unusual correspondent. Warning: Make sure this is someone you trust. Today, August 19th, 2026, the Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components. This is the sixth backport release in the Squid series. This is the fourth backport release in the Tentacle series. Additional details will be posted on the Ceph website/blog: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible. ## Critical Upgrade Steps The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… These steps are for package-based deployments of Ceph. In other sections, the document also includes new information about key rotation procedures, monitor authentication controls, and emergency recovery procedures. Deployments using cephadm will automate the process except for client keys. You may notice that Cephadm spends more time than normal on the upgrade after updating all daemon images. This is due to a new process rotating the OSD and MDS keys. Please be patient. There are plans to improve this in the future. Deployments using Rook will also automate rotation of some client keys with some exclusions. Look to the Rook project’s announcement for more details. Client and kernel upgrades are recommended to support aes256k. Note that upstream Linux kernel client support began in kernel 7.0 and has been backported to CentOS Stream 9 and 10. Check with your distribution vendor for backported key support before rotating client keys used by the kernel. As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go. These are documented beginning here: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… The CephX documentation has been brought up to date and vetted for accuracy. Those changes are live here: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 requests with `host` and `x-amz-` headers not included in the signed subset. Unfortunately, the REST client used in multisite was generating such improperly signed requests. If you are running multisite, you must set the ``rgw_sigv4_insecure`` option to true before you begin to upgrade. After all clusters are upgraded, set the option to ``false`` again. Regarding the fix for CVE-2026-50152, formal guidance on rotating all secrets stored in the Monitor config-key store will be forthcoming. In the meantime, operators of cephadm-managed clusters are strongly encouraged to rotate the cephadm SSH key, as an established process exists: See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… Rotating other secrets is advised but the process is not yet established. Operators should assess their cluster's potential exposure to determine if immediate Monitor config-key secret rotation is required for their environment. ## CVE Fixes * CVE-2025-30156 is an authentication bypass in CephX caused by misuse of AES-CBC. See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * CVE-2026-39944 shares the unauthenticated-encryption root cause of CVE-2025-30156, but applies it to RGW's STS session tokens resulting in improper verification of a cryptographic signature. See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor subscription handler. See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 cryptographic signatures in RGW's SigV4 verifier. See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… ## Other Media A Cephalocon 2025 talk covers some of the details surrounding the new CephX key upgrade process. https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… ## Getting Ceph ------------ * Source at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… * v19.2.6 Tarball at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… * v20.2.4 Tarball at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… * Containers at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… * For packages, see https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * For upgrade with cephadm, see: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- This message has been checked by Libraesva ESG and is found to be clean. Follow this link to report as spam/bad: https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad Follow this link to blocklist sender: https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- This message has been checked by Libraesva ESG and is found to be clean. Follow this link to report as spam/bad: https://mx10.wiit.cloud/action/4hQgT36zpCzTkgH/report-as-bad Follow this link to blocklist sender: https://mx10.wiit.cloud/action/4hQgT36zpCzTkgH/blocklist

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Pedro Macedo 20 Aug '26

20 Aug '26

On 2026-08-20 10:32 AM, Iztok Gregori wrote: > Hi! > > Silly question maybe, but I want to be sure, for RBD clients (Proxmox > QEMU virtual machines in my case) the new key type needs updated > "client libraries" to be "understood", right? Seems to be the case, according to the presentation that talked about the cephx rotation when it was added ( title "Rotating Heads and CephX Keys - Patrick Donnelly, IBM & Yehuda Sadeh Weinraub, Ubiquiti" on youtube; not including link in case it gets blocked. Relevant bit is at 19:50). Quote from the slide: "Old userspace clients will crash if you attempt to use a new key type!". And while I haven't tested libvirt against 20.2.4 yet, the new key type breaks cephfs mounts on Debian stable (which has ceph-common @ 18.2.7) with an error about parsing the keyring file. -- Pedro Macedo > > Cheers > Iztok > > On 19/08/26 18:07, Patrick Donnelly wrote: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >> >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >> >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >> >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: >> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >> >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at >> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at >> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see >> https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Pedro Macedo 20 Aug '26

20 Aug '26

On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: > Hi, > > I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The following cephadm managed client keys were not rotated automatically: > > client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) > client.ceph-exporter. > client.crash. > client.osd-lockbox.. > client.rgw. > > The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* this command does not exist. Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing there, but present on 20.2.3. > https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… > https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… > > What is the correct way to rotate these client keys via cephadm as part of the cve fix? > > Thanks for any guidance. > Best Regards > Max > > ________________________________ > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: 19 August 2026 18:08 > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released > > Warning: This email is from an unusual correspondent. > Warning: Make sure this is someone you trust. > > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… > > ## Getting Ceph > ------------ > * Source at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… > * v19.2.6 Tarball at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > * v20.2.4 Tarball at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… > * Containers at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… > * For packages, see https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > * For upgrade with cephadm, see: > https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > This message has been checked by Libraesva ESG and is found to be clean. > Follow this link to report as spam/bad: > https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad > Follow this link to blocklist sender: > https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Maximilian Stinsky-Damke 20 Aug '26

20 Aug '26

Hi, I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The following cephadm managed client keys were not rotated automatically: client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) client.ceph-exporter. client.crash. client.osd-lockbox.. client.rgw. The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* this command does not exist. https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… What is the correct way to rotate these client keys via cephadm as part of the cve fix? Thanks for any guidance. Best Regards Max ________________________________ From: Patrick Donnelly <ceph-users(a)ceph.io> Sent: 19 August 2026 18:08 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released Warning: This email is from an unusual correspondent. Warning: Make sure this is someone you trust. Today, August 19th, 2026, the Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components. This is the sixth backport release in the Squid series. This is the fourth backport release in the Tentacle series. Additional details will be posted on the Ceph website/blog: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible. ## Critical Upgrade Steps The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… These steps are for package-based deployments of Ceph. In other sections, the document also includes new information about key rotation procedures, monitor authentication controls, and emergency recovery procedures. Deployments using cephadm will automate the process except for client keys. You may notice that Cephadm spends more time than normal on the upgrade after updating all daemon images. This is due to a new process rotating the OSD and MDS keys. Please be patient. There are plans to improve this in the future. Deployments using Rook will also automate rotation of some client keys with some exclusions. Look to the Rook project’s announcement for more details. Client and kernel upgrades are recommended to support aes256k. Note that upstream Linux kernel client support began in kernel 7.0 and has been backported to CentOS Stream 9 and 10. Check with your distribution vendor for backported key support before rotating client keys used by the kernel. As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go. These are documented beginning here: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… The CephX documentation has been brought up to date and vetted for accuracy. Those changes are live here: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 requests with `host` and `x-amz-` headers not included in the signed subset. Unfortunately, the REST client used in multisite was generating such improperly signed requests. If you are running multisite, you must set the ``rgw_sigv4_insecure`` option to true before you begin to upgrade. After all clusters are upgraded, set the option to ``false`` again. Regarding the fix for CVE-2026-50152, formal guidance on rotating all secrets stored in the Monitor config-key store will be forthcoming. In the meantime, operators of cephadm-managed clusters are strongly encouraged to rotate the cephadm SSH key, as an established process exists: See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… Rotating other secrets is advised but the process is not yet established. Operators should assess their cluster's potential exposure to determine if immediate Monitor config-key secret rotation is required for their environment. ## CVE Fixes * CVE-2025-30156 is an authentication bypass in CephX caused by misuse of AES-CBC. See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * CVE-2026-39944 shares the unauthenticated-encryption root cause of CVE-2025-30156, but applies it to RGW's STS session tokens resulting in improper verification of a cryptographic signature. See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor subscription handler. See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 cryptographic signatures in RGW's SigV4 verifier. See: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… ## Other Media A Cephalocon 2025 talk covers some of the details surrounding the new CephX key upgrade process. https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… ## Getting Ceph ------------ * Source at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… * v19.2.6 Tarball at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… * v20.2.4 Tarball at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… * Containers at https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… * For packages, see https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * For upgrade with cephadm, see: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- This message has been checked by Libraesva ESG and is found to be clean. Follow this link to report as spam/bad: https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad Follow this link to blocklist sender: https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Wannes Smet 20 Aug '26

20 Aug '26

@Gregori: I have opened a ticket at our PVE support partner. I want to know too. ________________________________ From: Iztok Gregori <ceph-users(a)ceph.io> Sent: Thursday, August 20, 2026 10:33 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released Hi! Silly question maybe, but I want to be sure, for RBD clients (Proxmox QEMU virtual machines in my case) the new key type needs updated "client libraries" to be "understood", right? Cheers Iztok On 19/08/26 18: 07, Patrick Donnelly wrote: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi! Silly question maybe, but I want to be sure, for RBD clients (Proxmox QEMU virtual machines in my case) the new key type needs updated "client libraries" to be "understood", right? Cheers Iztok On 19/08/26 18:07, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://urldefense.com/v3/__https://ceph.io/en/news/blog/__;!!FtrhtPsWDhZ6t… > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/configura… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/operation… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://urldefense.com/v3/__https://docs.ceph.com/en/latest/architecture/in… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://urldefense.com/v3/__https://docs.ceph.com/en/latest/cephadm/host-ma… > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://urldefense.com/v3/__https://docs.ceph.com/en/latest/security/CVE-20… > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://urldefense.com/v3/__https://docs.ceph.com/en/latest/security/CVE-20… > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://urldefense.com/v3/__https://docs.ceph.com/en/latest/security/CVE-20… > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://urldefense.com/v3/__https://docs.ceph.com/en/latest/security/CVE-20… > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://urldefense.com/v3/__https://cephalocon2025.sched.com/event/27f2O/ro… > > ## Getting Ceph > ------------ > * Source at https://urldefense.com/v3/__https://github.com/ceph/ceph__;!!FtrhtPsWDhZ6tw… > * v19.2.6 Tarball at https://urldefense.com/v3/__https://download.ceph.com/tarballs/ceph-19.2.6.… > * v20.2.4 Tarball at https://urldefense.com/v3/__https://download.ceph.com/tarballs/ceph-20.2.4.… > * Containers at https://urldefense.com/v3/__https://quay.io/repository/ceph/ceph__;!!FtrhtP… > * For packages, see https://urldefense.com/v3/__https://docs.ceph.com/en/latest/install/get-pac… > * For upgrade with cephadm, see: > https://urldefense.com/v3/__https://docs.ceph.com/en/latest/cephadm/upgrade… > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Iztok Gregori 20 Aug '26

20 Aug '26

Hi! Silly question maybe, but I want to be sure, for RBD clients (Proxmox QEMU virtual machines in my case) the new key type needs updated "client libraries" to be "understood", right? Cheers Iztok On 19/08/26 18:07, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Dmitry Melekhov 20 Aug '26

20 Aug '26

19.08.2026 20:04, Patrick Donnelly пишет: > the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > Hello! Is there step by step guide for cephadm installation? I do not want to miss something. Thank you!

1 0

Question on so-called tenant level roles in tentacle
by Christopher Durham 19 Aug '26

19 Aug '26

The release notes for ceph tentacle, at: https://docs.ceph.com/en/latest/releases/tentacle/ state the following "RGW: The User Account feature introduced in Squid provides first-class support for IAM APIs and policy. Our preliminary STS support was based on tenants, and exposed some IAM APIs to admins only. This tenant-level IAM functionality is now deprecated in favor of accounts. While we’ll continue to support the tenant feature itself for namespace isolation, the following features will be removed no sooner than the V release: Tenant-level IAM APIs including CreateRole, PutRolePolicy and PutUserPolicy, Use of tenant names instead of accounts in IAM policy documents, Interpretation of IAM policy without cross-account policy evaluation, S3 API support for cross-tenant names such as Bucket=’tenant:bucketname’ STS Lite and sts:GetSessionToken." Does the 'V' release mean the Vampire release? Does this statement mean that those API calls will be available in all tentacle releases and umbrella, but maybe not vampire? I need to have the CreateRole,PutRolePolicy et al available in tentacle and do a planned transition to IAM users/roles. Any documentation on how to do this? I have STS roles for various groups of users to gain temporary access to solme number of buckets and I want to be sure my upgrade to tentacle will be ok. (I am on 19.2.5 squid). We are not using tenants but buckets owned by various s3 users and roles to give temporary access to buckets by other users. We have been doing this since either nautilus or octopus and it has worked well for us. (We do not use the gui for role management but have our own python scripts to add users/buckets to roles). -Chris

1 0

Re: Affects of high pg count on OSD peering
by Elias Carter 19 Aug '26

19 Aug '26

> the IO should not be blocked for the entire pool but only for those repeering PGs Yes, the issue is that when running 600 PGs per OSD a high fraction of PGs re-peer simultaneously. > Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement. Thank you for the confirmation. We run Ceph within kubernetes (rook) and my plan is to write an operator which will handle gradually reweighting OSDs as they are added/removed from the cluster. Rook allows setting the initial OSD weight, so my plan is to set the initial OSD weights to 0 and then have the operator gradually reweight the OSDs to 1. > One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds). Thank you; I will try that out. We have a mix of 100G and 400G networks, so I think the increased ping frequency will be ok. I will try it out on our staging clusters first and report back if there are any interesting findings. Elias On Tue, Aug 18, 2026 at 2:21 AM Eugen Block <ceph-users(a)ceph.io> wrote: > Hi, > > the IO should not be blocked for the entire pool but only for those > repeering PGs. Warnings during that phase are expected. > Gradually reweighting instead of setting an OSD "out" directly is a > common practice among ceph users, there should be plenty of threads on > this list about that. The downside is of course a lot more manual > intervention and possibly (some) redundant data movement. > > One knob to turn could be to decrease osd_heartbeat_interval (number > of seconds between peer pings) from 6 seconds to a lower value (1 > second is the minimum, but I probably would start with 5 or 4 > seconds). That could (or probably would) increase network traffic > significantly, so it's a trade-off. If you have a test cluster with > similar behavior, test it there first. > > Regards, > Eugen > > Zitat von Elias Carter via ceph-users <ceph-users(a)ceph.io>: > > > We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and > > upwards). > > > > Something that I have noticed is that CRUSH changes (marking an OSD as > out, > > or a new OSD joining the cluster) can cause many hundreds of PGs to remap > > and peer. During this time we get PG_AVAILABILITY health check warnings > as > > the OSDs need to peer with their new PGs before they can serve IOs. RADOS > > tail latency suffers because IOs sent to a peering PG are blocked until > > peering completes. > > > > I believe that the more PGs we have, the more PGs need to peer, and the > > longer IOs are blocked for the pool. > > > > Does anyone have suggestions on how to deal with PG peering blocking IOs? > > > > I was thinking about gradually re-weighting OSDs instead of marking them > > out or in, but that can result in multiple redundant data moves. > > > > Elias > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 4

[CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Patrick Donnelly 19 Aug '26

19 Aug '26

Today, August 19th, 2026, the Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components. This is the sixth backport release in the Squid series. This is the fourth backport release in the Tentacle series. Additional details will be posted on the Ceph website/blog: https://ceph.io/en/news/blog/ We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible. ## Critical Upgrade Steps The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at: https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… These steps are for package-based deployments of Ceph. In other sections, the document also includes new information about key rotation procedures, monitor authentication controls, and emergency recovery procedures. Deployments using cephadm will automate the process except for client keys. You may notice that Cephadm spends more time than normal on the upgrade after updating all daemon images. This is due to a new process rotating the OSD and MDS keys. Please be patient. There are plans to improve this in the future. Deployments using Rook will also automate rotation of some client keys with some exclusions. Look to the Rook project’s announcement for more details. Client and kernel upgrades are recommended to support aes256k. Note that upstream Linux kernel client support began in kernel 7.0 and has been backported to CentOS Stream 9 and 10. Check with your distribution vendor for backported key support before rotating client keys used by the kernel. As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go. These are documented beginning here: https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… The CephX documentation has been brought up to date and vetted for accuracy. Those changes are live here: https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 requests with `host` and `x-amz-` headers not included in the signed subset. Unfortunately, the REST client used in multisite was generating such improperly signed requests. If you are running multisite, you must set the ``rgw_sigv4_insecure`` option to true before you begin to upgrade. After all clusters are upgraded, set the option to ``false`` again. Regarding the fix for CVE-2026-50152, formal guidance on rotating all secrets stored in the Monitor config-key store will be forthcoming. In the meantime, operators of cephadm-managed clusters are strongly encouraged to rotate the cephadm SSH key, as an established process exists: See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration Rotating other secrets is advised but the process is not yet established. Operators should assess their cluster's potential exposure to determine if immediate Monitor config-key secret rotation is required for their environment. ## CVE Fixes * CVE-2025-30156 is an authentication bypass in CephX caused by misuse of AES-CBC. See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 * CVE-2026-39944 shares the unauthenticated-encryption root cause of CVE-2025-30156, but applies it to RGW's STS session tokens resulting in improper verification of a cryptographic signature. See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor subscription handler. See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 cryptographic signatures in RGW's SigV4 verifier. See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 ## Other Media A Cephalocon 2025 talk covers some of the details surrounding the new CephX key upgrade process. https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… ## Getting Ceph ------------ * Source at https://github.com/ceph/ceph * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * For upgrade with cephadm, see: https://docs.ceph.com/en/latest/cephadm/upgrade/ * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

cephx key rotation - blue green rollout
by mudman 19 Aug '26

19 Aug '26

Hi, I'm noticing that ceph does not have dual validity for keys. Is that correct? Because of this, I am thinking to use a blue green process to accomplish cephx key rotation. The caps of the existing client can be cloned with something like the below snippet, watch out for typos... And then clients can do rollout with new client+credentials with reduced impact. AUTH_OUT=$(ceph auth get "$SRC" 2>&1) || { echo “ERROR: 'ceph auth get $SRC' failed. Does the client exist?" >&2 exit 1 } CAP_ARGS=() while IFS= read -r line; do # Match: caps <daemon> = "<perms>" if [[ "$line" =~ ^[[:space:]]*caps[[:space:]]+([a-z]+)[[: space:]]*=[[:space:]]*\"(.*)\"[[:space:]]*$ ]]; then daemon="${BASH_REMATCH[1]}" perms="${BASH_REMATCH[2]}" CAP_ARGS+=("$daemon" “$perms” ) fi done <<< "“$AUTH_OUT" if [[ ${#CAP_ARGS[@]} -eq 0 ]]; then echo “ERROR: No caps found for $SRC. Refusing to create a client with no capabilities." >&2 exit 1 fi echo "Caps to apply to $NEW:" for (( i=0; i<${#CAP_ARGS[@]}; i+=2 )); do printf " %-6s = \"%s\"\n" "${CAP_ARGS[i]}" "${CAP_ARGS[i+1]}" done echo "" echo "Creating $NEW ..." ceph auth get-or-create "$NEW" "${CAP_ARGS[@]}" -o "$OUTFILE"

1 0

Re: Bucket listing suddenly 404
by Eugen Block 19 Aug '26

19 Aug '26

I feel like rgw_dns_name popped up more often recently, not sure if here on this list or on slack. I don't have anything to back that feeling up, though. But it's great that you figured it out. Zitat von Thomas Schneider <ceph-users(a)ceph.io>: > Ok it's completely solved. > I too late realized that suddenly when calling the s3 gateway > instead of seeing the default xml I saw our domain as Bucket. > I _somehow_ after nearly 10 years of running the rgw had to set > rgw_dns_name (again?) > I'm too tired right now to analyze it any further. > Hope this helps _someone_ in the future. > > On 2026-08-17 19:18, Thomas Schneider wrote: >> We "solved" the sync_owner() problem by deleting all Buckets owned >> by the user by changing the placement >>> # rubgw is radosgw-admin with realm/zone/zonegroup set >>> $rubgw metadata get "bucket.instance:$bucket:$meta" > $bucket-old.json >>> cp $bucket-old.json $bucket-new.json >>> sed -i 's/"placement_rule": "[^"]*"/"placement_rule": >>> "default-placement"/g' $bucket-new.json >>> $rubgw metadata put "bucket.instance:$bucket:$meta" >>> --infile=$bucket-new.json >> >> and then deleting the buckets, but the 404 problem when trying to >> list/access buckets still exist :/ >> >> I'm all out of ideas right now. >> >> On 2026-08-17 16:57, Thomas Schneider wrote: >>> Hey you all, >>> >>> so today our certificates got renewed and we suddenly have the >>> problem that every ls returns a 404 >>>> # rclone ls s3-nichtmuk: >>>> 2026/08/17 16:49:58 NOTICE: Failed to ls: operation error S3: >>>> ListBuckets, https response error StatusCode: 404, RequestID: >>>> tx00000d5116646fe615090-006a831f96-885923856-rub-z, HostID: >>>> 885923856-rub-z-rub-zg, api error NoSuchBucket: UnknownError >>> >>> We are not sure what caused it but one likely culprit is a test >>> with a placement rule we created and removed a while ago. >>> We created a test placement and later removed it, but still got a >>> user that is in that placement and because ceph still things that >>> the user exists we get an error on the rgw startup >>> >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: deferred set uid:gid >>>> to 167:167 (ceph:ceph) >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: ceph version 20.2.0 >>>> (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - >>>> RelWithDebInfo), process radosgw, pid 8 >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: framework: beast >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: framework conf key: >>>> port, val: 8001 >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: init_numa not setting >>>> numa affinity >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Realm: >>>> rub-realm (4d725f78-42cc-4ec1-8a0e-feff5800bdfb) >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: ZoneGroup: >>>> rub-zg (4170a834-cbe5-4cee-b4f9-83a47941c5a3) >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Zone: >>>> rub-z (99d273b2-e74c-432f-94de-1a73886419e8) >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: using >>>> period configuration: bcf5d281-9eb3-4a13-a011-7cc94c040cdf:19 >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: all 8 watchers are >>>> set, enabling cache >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: >>>> garbage collection: start >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw object expirer >>>> Worker thread: object expiration: start >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: >>>> garbage collection: stop >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: >>>> watch_reload::DEDUP_WATCH_OBJ exists >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: >>>> dedup_bg::watch_reload(): Started watching >>>> DEDUP_WATCH_OBJ::watch_handle=94372625010048 >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework: beast >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: >>>> ssl_certificate, val: config://rgw/cert/$realm/$zone.crt >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: >>>> ssl_private_key, val: config://rgw/cert/$realm/$zone.key >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: starting handler: beast >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on >>>> 0.0.0.0:8001 >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on >>>> [::]:8001 >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: set uid:gid to >>>> 167:167 (ceph:ceph) >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: mgrc >>>> service_daemon_register rgw.885923856 metadata >>>> {arch=x86_64,ceph_release=tentacle,ceph_version=ceph version >>>> 20.2.0 (69f84cc2651aa259a15bc >>>> 192ddaabd3baba07489) tentacle (stable - >>>> RelWithDebInfo),ceph_version_short=20.2.0,container_hostname=my-hostname,container_image=quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b >>>> 6dac688bcc77369b06009b5830fa8d86,cpu=Intel(R) Xeon(R) Silver 4114 >>>> CPU @ 2.20GHz,distro=centos,distro_description=CentOS Stream >>>> 9,distro_version=9,frontend_config#0=beast >>>> port=8001,frontend_type#0=beast,hostname=rub-01-osd.c >>>> eph-2.it-services.ruhr-uni-bochum.de,id=rub-rgw.rub-01-osd.ckfkhi,kernel_description=#1 SMP Thu May 18 10:27:05 EDT >>>> 2023,kernel_version=4.18.0-477.13.1.el8_8.x86_64,mem_swap_kb=16777212,mem_total_kb=131086224,num_handles=1, >>>> os=Linux,pid=8,realm_id=4d725f78-42cc-4ec1-8a0e-feff5800bdfb,realm_name=rub-realm,zone_id=99d273b2-e74c-432f-94de-1a73886419e8,zone_name=rub-z,zonegroup_id=4170a834-cbe5-4cee-b4f9-83a47941c5a3,zonegroup_name=rub-zg} >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw period pusher: No >>>> zones to update >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>>> ERROR: could not sync bucket stats: ret=-5 >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>>> ERROR: failed user stats sync, ret=-5 >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>>> ERROR: sync_owner() failed, owner=rub-fowi-katzen ret=-5 >>>> Aug 17 16:43:06 my-hostname radosgw[49228]: rgw object expirer >>>> Worker thread: object expiration: stop >>>> Aug 17 16:43:22 my-hostname radosgw[49228]: ====== starting new >>>> request req=0x7fd2a608b310 >>>> request_id=tx0000000924606841dd39d-006a831e0a-885923856-rub-z ===== >>> >>> >>> (set rgw_debug to 5/5 for this) >>>> Aug 17 16:49:54 my-hostname radosgw[49228]: beast: >>>> 0x7fd2a608b310: 127.0.0.1 - rub-bucket [17/Aug/2026:14:49:54.826 >>>> +0000] "GET /sfb1491-f7?max-keys=1000&prefix= HTTP/1.1" 404 255 - >>>> "rclone/v1.74.3" - latency=0.001000081s >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: ====== starting new >>>> request req=0x7fd2a608b310 >>>> request_id=tx00000d5116646fe615090-006a831f96-885923856-rub-z ===== >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.000000000s getting op 0 >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.000000000s s3:list_bucket verifying >>>> requester >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.001000082s s3:list_bucket normalizing >>>> buckets and tenants >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.001000082s s3:list_bucket init permissions >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.001000082s s3:list_bucket op status=0 >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.001000082s s3:list_bucket http status=404 >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: ====== req done >>>> req=0x7fd2a608b310 op=list_bucket bucket=our-s3-host status=0 >>>> http_status=404 latency=0. >>> >>> >>> We are not sure how to remove the user or debug the problem right now >>>> # rubgw user rm --uid=rub-fowi-katzen --purge-data >>>> could not remove user: unable to remove user, unable to delete user data >>>> 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: >>>> cls_bucket_list_unordered: error in rgw_rados_operate (bucket >>>> list op), r=-2 >>>> 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: >>>> list_objects_unordered cls_bucket_list_unordered returned -2 for >>>> :FOO[99d273b2-e74c-432f-94de-1a73886419e8.791780109.1]) >>> >>> Any help or hints would be greatly appreciated >>> >>> Kind regards, >>> Thomas > > -- > Thomas Schneider > IT.SERVICES > Wissenschaftliche Informationsversorgung > Ruhr-Universität Bochum | 44780 Bochum > Telefon: +49 234 32 23939 > http://www.it-services.rub.de/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0