- ceph-users - lists.ceph.io

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Peter Linder 24 Aug '26

24 Aug '26

Yep. This specifies that x-amz-content-sha256 needs to be included, but this is the general documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-sign… For s3, there is an exception however: https://docs.aws.amazon.com/AmazonS3/latest/developerguide/sig-v4-header-ba… So I believe this is a bug, 20.2.4 is more strict that it needs to be. There might be other considerations, but rgw_auth_s3.cc around like line 800 or so could have an exception for this header. /Peter On 8/24/26 4:12 PM, Omar Siam wrote: > I think so. > > There is a flag mentioned in the upgrade notes that needs to be set to > go back to the less secure protocol: > > "rgw_sigv4_insecure": Allow potential privilege escalation with added > `x-amz-` headers on Sigv4. > > I had to set this to true for my S3 clients (velero) to work again. > > Am 24.08.2026 um 15:41 schrieb Peter Linder: >> External Email: please use links and attachments from trusted sources >> only >> >> Does anyone know if 20.2.4 requires x-amz-content-sha256 to be a part >> of the signed headers now, unlike amazon that still leaves it optional? >> >> /Peter >> >> >> On 8/19/26 6:08 PM, Patrick Donnelly wrote: >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> https://ceph.io/en/news/blog/ >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >>> >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >>> >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >>> >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >>> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >>> >>> >>> ## Getting Ceph >>> ------------ >>> * Source at https://github.com/ceph/ceph >>> * v19.2.6 Tarball at >>> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >>> * v20.2.4 Tarball at >>> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >>> * Containers at https://quay.io/repository/ceph/ceph >>> * For packages, see >>> https://docs.ceph.com/en/latest/install/get-packages/ >>> * For upgrade with cephadm, see: >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Omar Siam 24 Aug '26

24 Aug '26

I think so. There is a flag mentioned in the upgrade notes that needs to be set to go back to the less secure protocol: "rgw_sigv4_insecure": Allow potential privilege escalation with added `x-amz-` headers on Sigv4. I had to set this to true for my S3 clients (velero) to work again. Am 24.08.2026 um 15:41 schrieb Peter Linder: > External Email: please use links and attachments from trusted sources > only > > Does anyone know if 20.2.4 requires x-amz-content-sha256 to be a part > of the signed headers now, unlike amazon that still leaves it optional? > > /Peter > > > On 8/19/26 6:08 PM, Patrick Donnelly wrote: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >> >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >> >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >> >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: >> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >> >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at >> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at >> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see >> https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

1 0

Re: Ceph Tentacle 20.2.4 Noble support
by Patrick Donnelly 24 Aug '26

24 Aug '26

On Mon, Aug 24, 2026 at 2:34 AM Marek Szuba <ceph-users(a)ceph.io> wrote: > > Dear Ceph maintainers, > > Further to the thread about 20.2.3 Noble support earlier this month, I > would like to point out that with there having been Noble packages for > up to and including 20.2.0, Noble users are now stuck with Ceph versions > with known security vulnerabilities. For many operators, ourselves > included, a downgrade to Jammy is not possible even it it were to be > done on Ceph servers themselves (as I mentioned in passing in one of my > messages to this list we experienced severe I/O issues with Jammy-backed > Ceph OSDs in one of our clusters, whereas another requires Noble or > newer in order for Podman to be new enough for SMB containers). > > I would therefore STRONGLY encourage you to resume the building and > publishing of packages for Noble ASAP irrespective of whether this > particular version of Ubuntu has reached the "officially supported" > status or not. Noble support will be present in v20.2.5. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Peter Linder 24 Aug '26

24 Aug '26

Does anyone know if 20.2.4 requires x-amz-content-sha256 to be a part of the signed headers now, unlike amazon that still leaves it optional? /Peter On 8/19/26 6:08 PM, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 24 Aug '26

24 Aug '26

Hi Eugen Ah yes, of course. You are absolutely right. I missed those when I was cleaning up obsolete keys. Deleting them has removed the error. Thanks for that lightning-fast help! Regards, Chris On 24/08/2026 12:16, Eugen Block wrote: > Hi Chris, > >> "detail": [ >> { >> "message": "entity mds.c1 using insecure key type: aes" >> }, >> { >> "message": "entity mds.c2 using insecure key type: aes" >> } > > those two auth entries are most likely from before your cephadm > upgrade, assuming you adopted this cluster with cephadm. They are most > likely not in use by any daemons as you can see in your current daemon > list: > >> Deploying daemon mds.cephfs.c1.secoxv on c1 >> Deploying daemon mds.cephfs.c2.lpzncl on c2 > > Before cephadm, MDS daemons didn't have any suffixes, so those must be > your old MDS daemons from before cephadm- > > I would recommend to double check, but you can probably just delete > those two old auth entries: > > ceph auth del mds.c1 > ceph auth del mds.c2 > > Just to be safe you can export those keyrings as a backup in case you > actually do experience trouble after deleting those entries. > > Regards, > Eugen > > Zitat von Chris Palmer <ceph-users(a)ceph.io>: > >> I'm upgrading a test cluster (cephadm, 20.2.3 -> 20.2.4). The upgrade >> completed successfully, but the MDS processes (1 active, 1 standby) >> still show as using the aes service key >> (AUTH_INSECURE_SERVICE_KEY_TYPE). I have manually issued a reconfig >> and restarted each daemon but no difference. >> >> Output of "ceph -W cephadm" (with the probably irrelevant stuff >> snipped out): >> >> 2026-08-24T11:03:16.341923+0100 mgr.c3.lythrw [INF] Upgrade: Started >> with target quay.io/ceph/ceph:v20.2.4 >> 2026-08-24T11:03:17.171731+0100 mgr.c3.lythrw [INF] Upgrade: First >> pull of quay.io/ceph/ceph:v20.2.4 >> 2026-08-24T11:03:35.480918+0100 mgr.c3.lythrw [INF] Upgrade: Target >> is version 20.2.4 (tentacle) >> 2026-08-24T11:03:35.480946+0100 mgr.c3.lythrw [INF] Upgrade: Target >> container is >> quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8, >> digests >> ['quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8', >> 'quay.io/ceph/ceph@sha256:6bb1c8a42fbc0bf87938946990b65174466997bc11c31eb5a323225a779fd8f9'] >> >> 2026-08-24T11:03:35.482902+0100 mgr.c3.lythrw [INF] Upgrade: Need to >> upgrade myself (mgr.c3.lythrw) >> 2026-08-24T11:03:36.195045+0100 mgr.c3.lythrw [INF] Upgrade: Updating >> mgr.c1.hsiqod >> 2026-08-24T11:03:36.250599+0100 mgr.c3.lythrw [INF] Deploying daemon >> mgr.c1.hsiqod on c1 >> [snip] >> >> 2026-08-24T11:08:16.388893+0100 mgr.c1.hsiqod [INF] Upgrade: It >> appears safe to stop mds.cephfs.c1.secoxv >> 2026-08-24T11:08:17.180069+0100 mgr.c1.hsiqod [INF] Upgrade: Updating >> mds.cephfs.c1.secoxv >> 2026-08-24T11:08:17.217174+0100 mgr.c1.hsiqod [INF] Deploying daemon >> mds.cephfs.c1.secoxv on c1 >> 2026-08-24T11:08:24.528567+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mds.cephfs.c2.lpzncl (monmap changed)... >> 2026-08-24T11:08:24.534395+0100 mgr.c1.hsiqod [INF] Upgrade: It >> appears safe to stop mds.cephfs.c2.lpzncl >> 2026-08-24T11:08:25.420651+0100 mgr.c1.hsiqod [INF] Upgrade: Updating >> mds.cephfs.c2.lpzncl >> 2026-08-24T11:08:25.486295+0100 mgr.c1.hsiqod [INF] Deploying daemon >> mds.cephfs.c2.lpzncl on c2 >> [snip] >> >> 2026-08-24T11:11:16.759649+0100 mgr.c1.hsiqod [INF] Rotating keyring >> for mds.cephfs.c1.secoxv >> 2026-08-24T11:11:16.786643+0100 mgr.c1.hsiqod [INF] Redeploying >> mds.cephfs.c1.secoxv with new keyring >> 2026-08-24T11:11:16.787629+0100 mgr.c1.hsiqod [INF] Deploying daemon >> mds.cephfs.c1.secoxv on c1 >> 2026-08-24T11:11:26.030560+0100 mgr.c1.hsiqod [INF] Rotating keyring >> for mds.cephfs.c2.lpzncl >> 2026-08-24T11:11:26.076154+0100 mgr.c1.hsiqod [INF] Redeploying >> mds.cephfs.c2.lpzncl with new keyring >> 2026-08-24T11:11:26.076957+0100 mgr.c1.hsiqod [INF] Deploying daemon >> mds.cephfs.c2.lpzncl on c2 >> 2026-08-24T11:11:35.278324+0100 mgr.c1.hsiqod [INF] Upgrade: checking >> if all mon, mgr, OSD, mds daemons upgraded before changing service >> cipher >> 2026-08-24T11:11:35.447319+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_KEYS_ALLOWED warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:36.529431+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_KEYS_CREATABLE warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:37.580675+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_SERVICE_TICKETS warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:38.673337+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_CLIENT_KEY_TYPE warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:39.743724+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_SERVICE_KEY_TYPE warning as upgrade is completed or has >> been stopped >> 2026-08-24T11:11:40.873968+0100 mgr.c1.hsiqod [INF] Unmuting >> AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning as upgrade is >> completed or has been stopped >> 2026-08-24T11:11:41.937998+0100 mgr.c1.hsiqod [INF] Upgrade: Complete! >> 2026-08-24T11:11:44.155457+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c1 (monmap changed)... >> 2026-08-24T11:11:44.155490+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c1 (monmap changed)... >> 2026-08-24T11:11:44.156559+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mon.c1 on c1 >> 2026-08-24T11:11:45.052365+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mgr.c1.hsiqod (monmap changed)... >> 2026-08-24T11:11:45.052389+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mgr.c1.hsiqod (monmap changed)... >> 2026-08-24T11:11:45.053849+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mgr.c1.hsiqod on c1 >> 2026-08-24T11:11:45.886112+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c2 (monmap changed)... >> 2026-08-24T11:11:45.886156+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c2 (monmap changed)... >> 2026-08-24T11:11:45.887368+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mon.c2 on c2 >> 2026-08-24T11:11:46.725430+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c3 (monmap changed)... >> 2026-08-24T11:11:46.725461+0100 mgr.c1.hsiqod [INF] Reconfiguring >> mon.c3 (monmap changed)... >> 2026-08-24T11:11:46.726439+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mon.c3 on c3 >> >> >> At this point nothing more happens. But: >> >> root@c3:~# ceph health detail --format=json|jq . >> { >> "status": "HEALTH_ERR", >> "checks": { >> "AUTH_INSECURE_CLIENT_KEY_TYPE": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "19 auth client entities with insecure key types", >> "count": 19 >> }, >> "detail": [ >> { >> "message": "entity client.admin using insecure key type: aes" >> }, >> { >> "message": "entity client.block-physical using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-mds using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-mgr using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-osd using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-rbd using insecure key >> type: aes" >> }, >> { >> "message": "entity client.bootstrap-rbd-mirror using >> insecure key type: aes" >> }, >> { >> "message": "entity client.bootstrap-rgw using insecure key >> type: aes" >> }, >> { >> "message": "entity client.ceph-exporter.c1 using insecure >> key type: aes" >> }, >> { >> "message": "entity client.ceph-exporter.c2 using insecure >> key type: aes" >> }, >> { >> "message": "entity client.ceph-exporter.c3 using insecure >> key type: aes" >> }, >> { >> "message": "entity client.crash.c1 using insecure key type: >> aes" >> }, >> { >> "message": "entity client.crash.c2 using insecure key type: >> aes" >> }, >> { >> "message": "entity client.crash.c3 using insecure key type: >> aes" >> }, >> { >> "message": "entity client.rgw. using insecure key type: aes" >> }, >> { >> "message": "entity client.rgw.c1 using insecure key type: aes" >> }, >> { >> "message": "entity client.rgw.c2 using insecure key type: aes" >> }, >> { >> "message": "entity client.rgw.rgw.c1.kevxdx using insecure >> key type: aes" >> }, >> { >> "message": "entity client.rgw.rgw.c2.fndzzj using insecure >> key type: aes" >> } >> ], >> "muted": false >> }, >> "AUTH_INSECURE_KEYS_ALLOWED": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "Monitors are configured to allow auth using >> insecure key types", >> "count": 1 >> }, >> "detail": [ >> { >> "message": "insecure cipher aes allowed for auth" >> } >> ], >> "muted": false >> }, >> "AUTH_INSECURE_KEYS_CREATABLE": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "Monitors are configured to allow creation of >> insecure key types", >> "count": 1 >> }, >> "detail": [], >> "muted": false >> }, >> "AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "4 rotating auth service keys using insecure key >> types", >> "count": 4 >> }, >> "detail": [ >> { >> "message": "rotating service keys for mon using insecure >> key type: aes" >> }, >> { >> "message": "rotating service keys for mds using insecure >> key type: aes" >> }, >> { >> "message": "rotating service keys for osd using insecure >> key type: aes" >> }, >> { >> "message": "rotating service keys for mgr using insecure >> key type: aes" >> } >> ], >> "muted": false >> }, >> "AUTH_INSECURE_SERVICE_KEY_TYPE": { >> "severity": "HEALTH_ERR", >> "summary": { >> "message": "2 auth service entities with insecure key types", >> "count": 2 >> }, >> "detail": [ >> { >> "message": "entity mds.c1 using insecure key type: aes" >> }, >> { >> "message": "entity mds.c2 using insecure key type: aes" >> } >> ], >> "muted": false >> }, >> "OSDMAP_FLAGS": { >> "severity": "HEALTH_WARN", >> "summary": { >> "message": "noout,norebalance flag(s) set", >> "count": 17 >> }, >> "detail": [], >> "muted": false >> } >> }, >> "mutes": [] >> } >> >> root@c3:~# ceph orch reconfig mds.cephfs >> Scheduled to reconfig mds.cephfs.c1.secoxv on host 'c1' >> Scheduled to reconfig mds.cephfs.c2.lpzncl on host 'c2' >> 2026-08-24T11:33:27.362951+0100 mgr.c1.hsiqod [INF] Reconfig service >> mds.cephfs >> 2026-08-24T11:33:27.561183+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mds.cephfs.c1.secoxv on c1 >> 2026-08-24T11:33:28.448509+0100 mgr.c1.hsiqod [INF] Reconfiguring >> daemon mds.cephfs.c2.lpzncl on c2 >> >> root@c3:~# ceph orch daemon restart mds.cephfs.c2.lpzncl >> Scheduled to restart mds.cephfs.c2.lpzncl on host 'c2' >> root@c3:~# ceph orch daemon restart mds.cephfs.c1.secoxv >> Scheduled to restart mds.cephfs.c1.secoxv on host 'c1' >> 2026-08-24T11:34:33.571897+0100 mgr.c1.hsiqod [INF] Schedule restart >> daemon mds.cephfs.c2.lpzncl >> 2026-08-24T11:41:40.075162+0100 mgr.c1.hsiqod [INF] Schedule restart >> daemon mds.cephfs.c1.secoxv >> >> But the HEALTH_ERR : AUTH_INSECURE_SERVICE_KEY_TYPE error persists. >> >> root@c3:~# ceph -s >> cluster: >> id: 90573d1e-7825-58a3-b34a-2890a7f53cea >> health: HEALTH_ERR >> 19 auth client entities with insecure key types >> Monitors are configured to allow auth using insecure key >> types >> Monitors are configured to allow creation of insecure key >> types >> 4 rotating auth service keys using insecure key types >> 2 auth service entities with insecure key types >> noout,norebalance flag(s) set >> >> services: >> mon: 3 daemons, quorum c1,c2,c3 (age 75s) [leader: c1] >> mgr: c1.hsiqod(active, since 54m), standbys: c3.lythrw >> mds: 1/1 daemons up, 1 standby >> osd: 12 osds: 12 up (since 69s), 12 in (since 7M) >> flags noout,norebalance,noautoscale >> rgw: 2 daemons active (2 hosts, 1 zones) >> >> data: >> volumes: 1/1 healthy >> pools: 14 pools, 249 pgs >> objects: 322 objects, 35 MiB >> usage: 180 GiB used, 750 GiB / 930 GiB avail >> pgs: 249 active+clean >> >> io: >> client: 85 B/s rd, 0 op/s rd, 0 op/s wr >> >> root@c3:~# ceph fs status >> cephfs - 0 clients >> ====== >> RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS >> 0 active cephfs.c1.secoxv Reqs: 0 /s 12 15 13 0 >> POOL TYPE USED AVAIL >> cephfsmeta metadata 3161k 37.9G >> cephfsdata data 14.3k 148G >> STANDBY MDS >> cephfs.c2.lpzncl >> MDS version: ceph version 20.2.4 >> (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable - >> RelWithDebInfo) >> >> What should I do from here? >> >> Thanks, Chris >> >> >> On 19/08/2026 17:07, Patrick Donnelly wrote: >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> https://ceph.io/en/news/blog/ >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >>> >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >>> >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >>> >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >>> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >>> >>> >>> ## Getting Ceph >>> ------------ >>> * Source at https://github.com/ceph/ceph >>> * v19.2.6 Tarball at >>> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >>> * v20.2.4 Tarball at >>> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >>> * Containers at https://quay.io/repository/ceph/ceph >>> * For packages, see >>> https://docs.ceph.com/en/latest/install/get-packages/ >>> * For upgrade with cephadm, see: >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 24 Aug '26

24 Aug '26

Hi Chris, > "detail": [ > { > "message": "entity mds.c1 using insecure key type: aes" > }, > { > "message": "entity mds.c2 using insecure key type: aes" > } those two auth entries are most likely from before your cephadm upgrade, assuming you adopted this cluster with cephadm. They are most likely not in use by any daemons as you can see in your current daemon list: > Deploying daemon mds.cephfs.c1.secoxv on c1 > Deploying daemon mds.cephfs.c2.lpzncl on c2 Before cephadm, MDS daemons didn't have any suffixes, so those must be your old MDS daemons from before cephadm- I would recommend to double check, but you can probably just delete those two old auth entries: ceph auth del mds.c1 ceph auth del mds.c2 Just to be safe you can export those keyrings as a backup in case you actually do experience trouble after deleting those entries. Regards, Eugen Zitat von Chris Palmer <ceph-users(a)ceph.io>: > I'm upgrading a test cluster (cephadm, 20.2.3 -> 20.2.4). The > upgrade completed successfully, but the MDS processes (1 active, 1 > standby) still show as using the aes service key > (AUTH_INSECURE_SERVICE_KEY_TYPE). I have manually issued a reconfig > and restarted each daemon but no difference. > > Output of "ceph -W cephadm" (with the probably irrelevant stuff snipped out): > > 2026-08-24T11:03:16.341923+0100 mgr.c3.lythrw [INF] Upgrade: Started > with target quay.io/ceph/ceph:v20.2.4 > 2026-08-24T11:03:17.171731+0100 mgr.c3.lythrw [INF] Upgrade: First > pull of quay.io/ceph/ceph:v20.2.4 > 2026-08-24T11:03:35.480918+0100 mgr.c3.lythrw [INF] Upgrade: Target > is version 20.2.4 (tentacle) > 2026-08-24T11:03:35.480946+0100 mgr.c3.lythrw [INF] Upgrade: Target > container is > quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8, digests ['quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8', > 'quay.io/ceph/ceph@sha256:6bb1c8a42fbc0bf87938946990b65174466997bc11c31eb5a323225a779fd8f9'] > 2026-08-24T11:03:35.482902+0100 mgr.c3.lythrw [INF] Upgrade: Need to > upgrade myself (mgr.c3.lythrw) > 2026-08-24T11:03:36.195045+0100 mgr.c3.lythrw [INF] Upgrade: > Updating mgr.c1.hsiqod > 2026-08-24T11:03:36.250599+0100 mgr.c3.lythrw [INF] Deploying daemon > mgr.c1.hsiqod on c1 > [snip] > > 2026-08-24T11:08:16.388893+0100 mgr.c1.hsiqod [INF] Upgrade: It > appears safe to stop mds.cephfs.c1.secoxv > 2026-08-24T11:08:17.180069+0100 mgr.c1.hsiqod [INF] Upgrade: > Updating mds.cephfs.c1.secoxv > 2026-08-24T11:08:17.217174+0100 mgr.c1.hsiqod [INF] Deploying daemon > mds.cephfs.c1.secoxv on c1 > 2026-08-24T11:08:24.528567+0100 mgr.c1.hsiqod [INF] Reconfiguring > mds.cephfs.c2.lpzncl (monmap changed)... > 2026-08-24T11:08:24.534395+0100 mgr.c1.hsiqod [INF] Upgrade: It > appears safe to stop mds.cephfs.c2.lpzncl > 2026-08-24T11:08:25.420651+0100 mgr.c1.hsiqod [INF] Upgrade: > Updating mds.cephfs.c2.lpzncl > 2026-08-24T11:08:25.486295+0100 mgr.c1.hsiqod [INF] Deploying daemon > mds.cephfs.c2.lpzncl on c2 > [snip] > > 2026-08-24T11:11:16.759649+0100 mgr.c1.hsiqod [INF] Rotating keyring > for mds.cephfs.c1.secoxv > 2026-08-24T11:11:16.786643+0100 mgr.c1.hsiqod [INF] Redeploying > mds.cephfs.c1.secoxv with new keyring > 2026-08-24T11:11:16.787629+0100 mgr.c1.hsiqod [INF] Deploying daemon > mds.cephfs.c1.secoxv on c1 > 2026-08-24T11:11:26.030560+0100 mgr.c1.hsiqod [INF] Rotating keyring > for mds.cephfs.c2.lpzncl > 2026-08-24T11:11:26.076154+0100 mgr.c1.hsiqod [INF] Redeploying > mds.cephfs.c2.lpzncl with new keyring > 2026-08-24T11:11:26.076957+0100 mgr.c1.hsiqod [INF] Deploying daemon > mds.cephfs.c2.lpzncl on c2 > 2026-08-24T11:11:35.278324+0100 mgr.c1.hsiqod [INF] Upgrade: > checking if all mon, mgr, OSD, mds daemons upgraded before changing > service cipher > 2026-08-24T11:11:35.447319+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_KEYS_ALLOWED warning as upgrade is completed or has > been stopped > 2026-08-24T11:11:36.529431+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_KEYS_CREATABLE warning as upgrade is completed or has > been stopped > 2026-08-24T11:11:37.580675+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_SERVICE_TICKETS warning as upgrade is completed or has > been stopped > 2026-08-24T11:11:38.673337+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_CLIENT_KEY_TYPE warning as upgrade is completed or has > been stopped > 2026-08-24T11:11:39.743724+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_SERVICE_KEY_TYPE warning as upgrade is completed or > has been stopped > 2026-08-24T11:11:40.873968+0100 mgr.c1.hsiqod [INF] Unmuting > AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning as upgrade is > completed or has been stopped > 2026-08-24T11:11:41.937998+0100 mgr.c1.hsiqod [INF] Upgrade: Complete! > 2026-08-24T11:11:44.155457+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c1 (monmap changed)... > 2026-08-24T11:11:44.155490+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c1 (monmap changed)... > 2026-08-24T11:11:44.156559+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mon.c1 on c1 > 2026-08-24T11:11:45.052365+0100 mgr.c1.hsiqod [INF] Reconfiguring > mgr.c1.hsiqod (monmap changed)... > 2026-08-24T11:11:45.052389+0100 mgr.c1.hsiqod [INF] Reconfiguring > mgr.c1.hsiqod (monmap changed)... > 2026-08-24T11:11:45.053849+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mgr.c1.hsiqod on c1 > 2026-08-24T11:11:45.886112+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c2 (monmap changed)... > 2026-08-24T11:11:45.886156+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c2 (monmap changed)... > 2026-08-24T11:11:45.887368+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mon.c2 on c2 > 2026-08-24T11:11:46.725430+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c3 (monmap changed)... > 2026-08-24T11:11:46.725461+0100 mgr.c1.hsiqod [INF] Reconfiguring > mon.c3 (monmap changed)... > 2026-08-24T11:11:46.726439+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mon.c3 on c3 > > > At this point nothing more happens. But: > > root@c3:~# ceph health detail --format=json|jq . > { > "status": "HEALTH_ERR", > "checks": { > "AUTH_INSECURE_CLIENT_KEY_TYPE": { > "severity": "HEALTH_WARN", > "summary": { > "message": "19 auth client entities with insecure key types", > "count": 19 > }, > "detail": [ > { > "message": "entity client.admin using insecure key type: aes" > }, > { > "message": "entity client.block-physical using insecure > key type: aes" > }, > { > "message": "entity client.bootstrap-mds using insecure key > type: aes" > }, > { > "message": "entity client.bootstrap-mgr using insecure key > type: aes" > }, > { > "message": "entity client.bootstrap-osd using insecure key > type: aes" > }, > { > "message": "entity client.bootstrap-rbd using insecure key > type: aes" > }, > { > "message": "entity client.bootstrap-rbd-mirror using > insecure key type: aes" > }, > { > "message": "entity client.bootstrap-rgw using insecure key > type: aes" > }, > { > "message": "entity client.ceph-exporter.c1 using insecure > key type: aes" > }, > { > "message": "entity client.ceph-exporter.c2 using insecure > key type: aes" > }, > { > "message": "entity client.ceph-exporter.c3 using insecure > key type: aes" > }, > { > "message": "entity client.crash.c1 using insecure key type: aes" > }, > { > "message": "entity client.crash.c2 using insecure key type: aes" > }, > { > "message": "entity client.crash.c3 using insecure key type: aes" > }, > { > "message": "entity client.rgw. using insecure key type: aes" > }, > { > "message": "entity client.rgw.c1 using insecure key type: aes" > }, > { > "message": "entity client.rgw.c2 using insecure key type: aes" > }, > { > "message": "entity client.rgw.rgw.c1.kevxdx using insecure > key type: aes" > }, > { > "message": "entity client.rgw.rgw.c2.fndzzj using insecure > key type: aes" > } > ], > "muted": false > }, > "AUTH_INSECURE_KEYS_ALLOWED": { > "severity": "HEALTH_WARN", > "summary": { > "message": "Monitors are configured to allow auth using > insecure key types", > "count": 1 > }, > "detail": [ > { > "message": "insecure cipher aes allowed for auth" > } > ], > "muted": false > }, > "AUTH_INSECURE_KEYS_CREATABLE": { > "severity": "HEALTH_WARN", > "summary": { > "message": "Monitors are configured to allow creation of > insecure key types", > "count": 1 > }, > "detail": [], > "muted": false > }, > "AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE": { > "severity": "HEALTH_WARN", > "summary": { > "message": "4 rotating auth service keys using insecure key types", > "count": 4 > }, > "detail": [ > { > "message": "rotating service keys for mon using insecure > key type: aes" > }, > { > "message": "rotating service keys for mds using insecure > key type: aes" > }, > { > "message": "rotating service keys for osd using insecure > key type: aes" > }, > { > "message": "rotating service keys for mgr using insecure > key type: aes" > } > ], > "muted": false > }, > "AUTH_INSECURE_SERVICE_KEY_TYPE": { > "severity": "HEALTH_ERR", > "summary": { > "message": "2 auth service entities with insecure key types", > "count": 2 > }, > "detail": [ > { > "message": "entity mds.c1 using insecure key type: aes" > }, > { > "message": "entity mds.c2 using insecure key type: aes" > } > ], > "muted": false > }, > "OSDMAP_FLAGS": { > "severity": "HEALTH_WARN", > "summary": { > "message": "noout,norebalance flag(s) set", > "count": 17 > }, > "detail": [], > "muted": false > } > }, > "mutes": [] > } > > root@c3:~# ceph orch reconfig mds.cephfs > Scheduled to reconfig mds.cephfs.c1.secoxv on host 'c1' > Scheduled to reconfig mds.cephfs.c2.lpzncl on host 'c2' > 2026-08-24T11:33:27.362951+0100 mgr.c1.hsiqod [INF] Reconfig service > mds.cephfs > 2026-08-24T11:33:27.561183+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mds.cephfs.c1.secoxv on c1 > 2026-08-24T11:33:28.448509+0100 mgr.c1.hsiqod [INF] Reconfiguring > daemon mds.cephfs.c2.lpzncl on c2 > > root@c3:~# ceph orch daemon restart mds.cephfs.c2.lpzncl > Scheduled to restart mds.cephfs.c2.lpzncl on host 'c2' > root@c3:~# ceph orch daemon restart mds.cephfs.c1.secoxv > Scheduled to restart mds.cephfs.c1.secoxv on host 'c1' > 2026-08-24T11:34:33.571897+0100 mgr.c1.hsiqod [INF] Schedule restart > daemon mds.cephfs.c2.lpzncl > 2026-08-24T11:41:40.075162+0100 mgr.c1.hsiqod [INF] Schedule restart > daemon mds.cephfs.c1.secoxv > > But the HEALTH_ERR : AUTH_INSECURE_SERVICE_KEY_TYPE error persists. > > root@c3:~# ceph -s > cluster: > id: 90573d1e-7825-58a3-b34a-2890a7f53cea > health: HEALTH_ERR > 19 auth client entities with insecure key types > Monitors are configured to allow auth using insecure key types > Monitors are configured to allow creation of insecure key types > 4 rotating auth service keys using insecure key types > 2 auth service entities with insecure key types > noout,norebalance flag(s) set > > services: > mon: 3 daemons, quorum c1,c2,c3 (age 75s) [leader: c1] > mgr: c1.hsiqod(active, since 54m), standbys: c3.lythrw > mds: 1/1 daemons up, 1 standby > osd: 12 osds: 12 up (since 69s), 12 in (since 7M) > flags noout,norebalance,noautoscale > rgw: 2 daemons active (2 hosts, 1 zones) > > data: > volumes: 1/1 healthy > pools: 14 pools, 249 pgs > objects: 322 objects, 35 MiB > usage: 180 GiB used, 750 GiB / 930 GiB avail > pgs: 249 active+clean > > io: > client: 85 B/s rd, 0 op/s rd, 0 op/s wr > > root@c3:~# ceph fs status > cephfs - 0 clients > ====== > RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS > 0 active cephfs.c1.secoxv Reqs: 0 /s 12 15 13 0 > POOL TYPE USED AVAIL > cephfsmeta metadata 3161k 37.9G > cephfsdata data 14.3k 148G > STANDBY MDS > cephfs.c2.lpzncl > MDS version: ceph version 20.2.4 > (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable - > RelWithDebInfo) > > What should I do from here? > > Thanks, Chris > > > On 19/08/2026 17:07, Patrick Donnelly wrote: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: >> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 24 Aug '26

24 Aug '26

I'm upgrading a test cluster (cephadm, 20.2.3 -> 20.2.4). The upgrade completed successfully, but the MDS processes (1 active, 1 standby) still show as using the aes service key (AUTH_INSECURE_SERVICE_KEY_TYPE). I have manually issued a reconfig and restarted each daemon but no difference. Output of "ceph -W cephadm" (with the probably irrelevant stuff snipped out): 2026-08-24T11:03:16.341923+0100 mgr.c3.lythrw [INF] Upgrade: Started with target quay.io/ceph/ceph:v20.2.4 2026-08-24T11:03:17.171731+0100 mgr.c3.lythrw [INF] Upgrade: First pull of quay.io/ceph/ceph:v20.2.4 2026-08-24T11:03:35.480918+0100 mgr.c3.lythrw [INF] Upgrade: Target is version 20.2.4 (tentacle) 2026-08-24T11:03:35.480946+0100 mgr.c3.lythrw [INF] Upgrade: Target container is quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8, digests ['quay.io/ceph/ceph@sha256:09ee90f6f3e0c7b9954f71d214ee05e9bbaaaea3716b1dd619603283b829f8b8', 'quay.io/ceph/ceph@sha256:6bb1c8a42fbc0bf87938946990b65174466997bc11c31eb5a323225a779fd8f9'] 2026-08-24T11:03:35.482902+0100 mgr.c3.lythrw [INF] Upgrade: Need to upgrade myself (mgr.c3.lythrw) 2026-08-24T11:03:36.195045+0100 mgr.c3.lythrw [INF] Upgrade: Updating mgr.c1.hsiqod 2026-08-24T11:03:36.250599+0100 mgr.c3.lythrw [INF] Deploying daemon mgr.c1.hsiqod on c1 [snip] 2026-08-24T11:08:16.388893+0100 mgr.c1.hsiqod [INF] Upgrade: It appears safe to stop mds.cephfs.c1.secoxv 2026-08-24T11:08:17.180069+0100 mgr.c1.hsiqod [INF] Upgrade: Updating mds.cephfs.c1.secoxv 2026-08-24T11:08:17.217174+0100 mgr.c1.hsiqod [INF] Deploying daemon mds.cephfs.c1.secoxv on c1 2026-08-24T11:08:24.528567+0100 mgr.c1.hsiqod [INF] Reconfiguring mds.cephfs.c2.lpzncl (monmap changed)... 2026-08-24T11:08:24.534395+0100 mgr.c1.hsiqod [INF] Upgrade: It appears safe to stop mds.cephfs.c2.lpzncl 2026-08-24T11:08:25.420651+0100 mgr.c1.hsiqod [INF] Upgrade: Updating mds.cephfs.c2.lpzncl 2026-08-24T11:08:25.486295+0100 mgr.c1.hsiqod [INF] Deploying daemon mds.cephfs.c2.lpzncl on c2 [snip] 2026-08-24T11:11:16.759649+0100 mgr.c1.hsiqod [INF] Rotating keyring for mds.cephfs.c1.secoxv 2026-08-24T11:11:16.786643+0100 mgr.c1.hsiqod [INF] Redeploying mds.cephfs.c1.secoxv with new keyring 2026-08-24T11:11:16.787629+0100 mgr.c1.hsiqod [INF] Deploying daemon mds.cephfs.c1.secoxv on c1 2026-08-24T11:11:26.030560+0100 mgr.c1.hsiqod [INF] Rotating keyring for mds.cephfs.c2.lpzncl 2026-08-24T11:11:26.076154+0100 mgr.c1.hsiqod [INF] Redeploying mds.cephfs.c2.lpzncl with new keyring 2026-08-24T11:11:26.076957+0100 mgr.c1.hsiqod [INF] Deploying daemon mds.cephfs.c2.lpzncl on c2 2026-08-24T11:11:35.278324+0100 mgr.c1.hsiqod [INF] Upgrade: checking if all mon, mgr, OSD, mds daemons upgraded before changing service cipher 2026-08-24T11:11:35.447319+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_KEYS_ALLOWED warning as upgrade is completed or has been stopped 2026-08-24T11:11:36.529431+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_KEYS_CREATABLE warning as upgrade is completed or has been stopped 2026-08-24T11:11:37.580675+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_SERVICE_TICKETS warning as upgrade is completed or has been stopped 2026-08-24T11:11:38.673337+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_CLIENT_KEY_TYPE warning as upgrade is completed or has been stopped 2026-08-24T11:11:39.743724+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_SERVICE_KEY_TYPE warning as upgrade is completed or has been stopped 2026-08-24T11:11:40.873968+0100 mgr.c1.hsiqod [INF] Unmuting AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning as upgrade is completed or has been stopped 2026-08-24T11:11:41.937998+0100 mgr.c1.hsiqod [INF] Upgrade: Complete! 2026-08-24T11:11:44.155457+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c1 (monmap changed)... 2026-08-24T11:11:44.155490+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c1 (monmap changed)... 2026-08-24T11:11:44.156559+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mon.c1 on c1 2026-08-24T11:11:45.052365+0100 mgr.c1.hsiqod [INF] Reconfiguring mgr.c1.hsiqod (monmap changed)... 2026-08-24T11:11:45.052389+0100 mgr.c1.hsiqod [INF] Reconfiguring mgr.c1.hsiqod (monmap changed)... 2026-08-24T11:11:45.053849+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mgr.c1.hsiqod on c1 2026-08-24T11:11:45.886112+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c2 (monmap changed)... 2026-08-24T11:11:45.886156+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c2 (monmap changed)... 2026-08-24T11:11:45.887368+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mon.c2 on c2 2026-08-24T11:11:46.725430+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c3 (monmap changed)... 2026-08-24T11:11:46.725461+0100 mgr.c1.hsiqod [INF] Reconfiguring mon.c3 (monmap changed)... 2026-08-24T11:11:46.726439+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mon.c3 on c3 At this point nothing more happens. But: root@c3:~# ceph health detail --format=json|jq . { "status": "HEALTH_ERR", "checks": { "AUTH_INSECURE_CLIENT_KEY_TYPE": { "severity": "HEALTH_WARN", "summary": { "message": "19 auth client entities with insecure key types", "count": 19 }, "detail": [ { "message": "entity client.admin using insecure key type: aes" }, { "message": "entity client.block-physical using insecure key type: aes" }, { "message": "entity client.bootstrap-mds using insecure key type: aes" }, { "message": "entity client.bootstrap-mgr using insecure key type: aes" }, { "message": "entity client.bootstrap-osd using insecure key type: aes" }, { "message": "entity client.bootstrap-rbd using insecure key type: aes" }, { "message": "entity client.bootstrap-rbd-mirror using insecure key type: aes" }, { "message": "entity client.bootstrap-rgw using insecure key type: aes" }, { "message": "entity client.ceph-exporter.c1 using insecure key type: aes" }, { "message": "entity client.ceph-exporter.c2 using insecure key type: aes" }, { "message": "entity client.ceph-exporter.c3 using insecure key type: aes" }, { "message": "entity client.crash.c1 using insecure key type: aes" }, { "message": "entity client.crash.c2 using insecure key type: aes" }, { "message": "entity client.crash.c3 using insecure key type: aes" }, { "message": "entity client.rgw. using insecure key type: aes" }, { "message": "entity client.rgw.c1 using insecure key type: aes" }, { "message": "entity client.rgw.c2 using insecure key type: aes" }, { "message": "entity client.rgw.rgw.c1.kevxdx using insecure key type: aes" }, { "message": "entity client.rgw.rgw.c2.fndzzj using insecure key type: aes" } ], "muted": false }, "AUTH_INSECURE_KEYS_ALLOWED": { "severity": "HEALTH_WARN", "summary": { "message": "Monitors are configured to allow auth using insecure key types", "count": 1 }, "detail": [ { "message": "insecure cipher aes allowed for auth" } ], "muted": false }, "AUTH_INSECURE_KEYS_CREATABLE": { "severity": "HEALTH_WARN", "summary": { "message": "Monitors are configured to allow creation of insecure key types", "count": 1 }, "detail": [], "muted": false }, "AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE": { "severity": "HEALTH_WARN", "summary": { "message": "4 rotating auth service keys using insecure key types", "count": 4 }, "detail": [ { "message": "rotating service keys for mon using insecure key type: aes" }, { "message": "rotating service keys for mds using insecure key type: aes" }, { "message": "rotating service keys for osd using insecure key type: aes" }, { "message": "rotating service keys for mgr using insecure key type: aes" } ], "muted": false }, "AUTH_INSECURE_SERVICE_KEY_TYPE": { "severity": "HEALTH_ERR", "summary": { "message": "2 auth service entities with insecure key types", "count": 2 }, "detail": [ { "message": "entity mds.c1 using insecure key type: aes" }, { "message": "entity mds.c2 using insecure key type: aes" } ], "muted": false }, "OSDMAP_FLAGS": { "severity": "HEALTH_WARN", "summary": { "message": "noout,norebalance flag(s) set", "count": 17 }, "detail": [], "muted": false } }, "mutes": [] } root@c3:~# ceph orch reconfig mds.cephfs Scheduled to reconfig mds.cephfs.c1.secoxv on host 'c1' Scheduled to reconfig mds.cephfs.c2.lpzncl on host 'c2' 2026-08-24T11:33:27.362951+0100 mgr.c1.hsiqod [INF] Reconfig service mds.cephfs 2026-08-24T11:33:27.561183+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mds.cephfs.c1.secoxv on c1 2026-08-24T11:33:28.448509+0100 mgr.c1.hsiqod [INF] Reconfiguring daemon mds.cephfs.c2.lpzncl on c2 root@c3:~# ceph orch daemon restart mds.cephfs.c2.lpzncl Scheduled to restart mds.cephfs.c2.lpzncl on host 'c2' root@c3:~# ceph orch daemon restart mds.cephfs.c1.secoxv Scheduled to restart mds.cephfs.c1.secoxv on host 'c1' 2026-08-24T11:34:33.571897+0100 mgr.c1.hsiqod [INF] Schedule restart daemon mds.cephfs.c2.lpzncl 2026-08-24T11:41:40.075162+0100 mgr.c1.hsiqod [INF] Schedule restart daemon mds.cephfs.c1.secoxv But the HEALTH_ERR : AUTH_INSECURE_SERVICE_KEY_TYPE error persists. root@c3:~# ceph -s cluster: id: 90573d1e-7825-58a3-b34a-2890a7f53cea health: HEALTH_ERR 19 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types 4 rotating auth service keys using insecure key types 2 auth service entities with insecure key types noout,norebalance flag(s) set services: mon: 3 daemons, quorum c1,c2,c3 (age 75s) [leader: c1] mgr: c1.hsiqod(active, since 54m), standbys: c3.lythrw mds: 1/1 daemons up, 1 standby osd: 12 osds: 12 up (since 69s), 12 in (since 7M) flags noout,norebalance,noautoscale rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 249 pgs objects: 322 objects, 35 MiB usage: 180 GiB used, 750 GiB / 930 GiB avail pgs: 249 active+clean io: client: 85 B/s rd, 0 op/s rd, 0 op/s wr root@c3:~# ceph fs status cephfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.c1.secoxv Reqs: 0 /s 12 15 13 0 POOL TYPE USED AVAIL cephfsmeta metadata 3161k 37.9G cephfsdata data 14.3k 148G STANDBY MDS cephfs.c2.lpzncl MDS version: ceph version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable - RelWithDebInfo) What should I do from here? Thanks, Chris On 19/08/2026 17:07, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Johan 24 Aug '26

24 Aug '26

Ok, on Debian 12 it was not a problem upgrading to ceph 20.2.4 using apt and https://download.ceph.com/debian-tentacle/ With Debian 13 which I have at my troublesome cluster I get several dependency-issues. ceph-common is dependent on libicu72 which doesn't have any candidates python3-cephfs (and others) is dependent on python3 < 3.12 but 3.13.5-1 will be installed etc Is this because I perhaps hadn't upgraded ceph-common and others properly before upgrading Debian from 12 to 13? /Johan Den 2026-08-24 kl. 11:06, skrev Johan: > Im not a professional computer tech but have two ceph clusters (cephadm, > Trixie, Debian 12/13). > > I upgraded the least sensitive one using the dashboard. When I upgrade > the admin client I now get > auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error > setting modifier for [client.admin] type=key val=<key>: Malformed input > [buffer:3] > auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/ > output error > > I get the same respons when using a ordinary user to connect the > filesystem. > > I found this issue describing the same error > https://github.com/rook/rook/issues/18240 > > As they, in above issue, discussed which version/image of the toolbox > was used I checked my version of ceph at the command line. I get > ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef > (stable) > I would guess this is the problem. > > How do I get to run up-to-date version on the command line? > > /Johan > > > Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/ >> index.html#upgrading-and-rotating-cephx-keys >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/ >> index.html#auth-insecure-keys-creatable >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high- >> availability-authentication >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh- >> configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx- >> keys-patrick-donnelly-ibm-yehuda-sadeh-weinraub-ubiquiti >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ >> ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ >> ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Jan Kasprzak 24 Aug '26

24 Aug '26

Hi Nico, Nico Schottelius wrote: > Robert Sander <ceph-users(a)ceph.io> writes: > > > Hi, > > > > Am 21.08.26 um 4:10 PM schrieb Nico Schottelius: > > > >> a) All VPS/VMs need to be restarted > >> To be able to rotate keys, aside from the almost impossible flow to > >> actually doing the rotation (more to that below), this will affect > >> every consumer of VMs. The usual setup is that all hypervisors are > >> using the same key. > > > > VM live migration should be able to do the trick here. When the new > > compute node already has the new key and the librbd version to support > > it. > > That's a very good point. However this might also need work (or > hacks) in the hypervisor sphere, to support using different keys in the > 2 qemu processes. By default in opennebula, the ceph key is saved in > a libvirtd secret. So in the best case, the new process will read it > from the local secret store. > > In the not so great case, opennebula (and other hypervisors) might infer > the key from the previously deployed VM, instead of reading it from the > local secret store. (*) > > (*) I'm not sure about the state of other hypervisors, but in the > opennebula case keys are actually embededd in the XML VM definition (sic!) that is > saved in the database (sic!!) and copied on creation of the VM (!!!) and > will never every be changed. I think the problem is not the secret/key itself, Ceph username is the problem, as it seems to be global in OpenNebula. The problem you assume does not exist in OpenNebula - it correctly refers to the libvirt secret UUID, not the Ceph user key itself. You can put another key under the same libvirt secret UUID if you want: # virsh dumpxml one-... [...] <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <auth username='libvirt'> <secret type='ceph' uuid='9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf'/> </auth> <source protocol='rbd' name='one/one-1234-123456-0' index='2'> <host name='mon1' port='6789'/> <host name='mon2' port='6789'/> <host name='mon3' port='6789'/> </source> Create a new Ceph user with # ceph auth get-or-create client.libvirt2026 \ mon 'profile rbd' \ osd 'profile rbd pool=one' \ mgr 'profile rbd pool=one' \ -o libvirt2026.keyring Import the key under the same UUID on an idle/disabled node: # cat client.libvirt2026.secret.xml <secret ephemeral='no' private='no'> <uuid>9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf</uuid> <usage type='ceph'> <name>client.libvirt2026 secret</name> </usage> </secret> # virsh secret-undefine 9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf # virsh secret-define client.libvirt2026.secret.xml # KEY=`ceph auth get-key client.libvirt2026` # virsh secret-set-value --secret 9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf --base64 $KEY But it does not work, as I cannot persuade OpenNebula to use a different Ceph user instead of "libvirt" on that node (as opposed to the whole cluster, which would probably break running VMs). In the running VM template there are the following embedded values for a Ceph disk: "DISK": { [...] "CEPH_HOST": "mon1 mon2 mon3", "CEPH_SECRET": "9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXf", "CEPH_TRASH": "yes", "CEPH_USER": "libvirt", [...] FWIW, I also created this post on OpenNebula forum: https://forum.opennebula.io/t/ceph-key-rollover/14547 -Yenya > > A likely working hack is to modify the key inside the XML inside the > database and then trigger live migration with a small time window of > uncertainty that if the VM crashes on the original host before the live > migration is triggered, it will likely not restart. > > -- > Sustainable and modern Infrastructures by ungleich.ch > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Johan 24 Aug '26

24 Aug '26

Im not a professional computer tech but have two ceph clusters (cephadm, Trixie, Debian 12/13). I upgraded the least sensitive one using the dashboard. When I upgrade the admin client I now get auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error setting modifier for [client.admin] type=key val=<key>: Malformed input [buffer:3] auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error I get the same respons when using a ordinary user to connect the filesystem. I found this issue describing the same error https://github.com/rook/rook/issues/18240 As they, in above issue, discussed which version/image of the toolbox was used I checked my version of ceph at the command line. I get ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) I would guess this is the problem. How do I get to run up-to-date version on the command line? /Johan Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0