- ceph-users - lists.ceph.io

Re: Issue with encrypted OSDs when upgrading from version 19.2.4 to 19.2.6 using cephadm
by Eugen Block 04 Sep '26

04 Sep '26

I did a lot of tests with a virtual test cluster in the past two weeks with encrypted OSDs and I haven't encountered that yet. I would check for ceph-osd and ceph-volume package on the affected hosts. Or for remainders from previous cephadm installations, anything not properly cleaned up maybe? What do the OSDs log when they try to start up? It sounds like some stale systemd units or something, hard to tell. Have you rebooted the hosts in question? On one of my lab clusters I noticed issues with systemd when I re-bootstrap a cluster without cephadm multiple times. I have a bash script that works just fine on a freshly rebooted system, but after wiping everything and starting over, at some point the script doesn't work anymore, especially when it comes to create OSDs. But this might be unrelated, just mentioning it. Zitat von Eugeniusz Pokora <ceph-users(a)ceph.io>: > When upgrading Ceph cluster from version 19.2.4 to 19.2.6 using > cephadm, an issue occurred with encrypted OSDs. > > 2026-09-02 10:53:28,494 7f24e08141c0 DEBUG Determined image: > 'quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089' > 2026-09-02 10:53:28,501 7f24e08141c0 INFO Reconfig daemon osd.36 ... > 2026-09-02 10:53:29,885 7f24e08141c0 INFO Stopping osd.36 to update > osd_key bluestore label > 2026-09-02 10:53:31,670 7f24e08141c0 INFO Opening osd.36 with crypsetup > 2026-09-02 10:53:33,955 7f24e08141c0 INFO Non-zero exit code 5 from > /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host > --entrypoint /tmp/cryptsetup_action.sh --privileged --group-add=disk > --init -e > CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 -e NODE_NAME=k08r06cph03 -v /dev:/dev -v /tmp/cephadm-osd-36-rotate-helper.sh:/tmp/cryptsetup_action.sh > quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 > 2026-09-02 10:53:33,956 7f24e08141c0 INFO /usr/bin/podman: stderr > Cannot use device > /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or > mounted). > 2026-09-02 10:53:33,956 7f24e08141c0 ERROR ERROR: Got error rotating > osd keyring while using cryptsetup tool > Out: > Err:Cannot use device > /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or > mounted). > > The issue did not occur on all hosts in the cluster (fortunately, it > was a test env), but it affected all OSDs on affected host. After > error occurred OSDs remained in a down state and had to be up > manually. > After several attempts, the update was successfully completed, but > we had to rotate the OSD keyrings ourselves. > A few hours latter when monmap was changed and cephadm scheduled a > reconfig this problem happened again, on 2 out of 4 nodes of the > cluster all osds goes down due to cephadm reconfig and this same > error: "Cannot use device... which is in use (already mapped or > mounted)". > > Has anyone encountered similar problems? > > Regards > Eugeniusz P

1 0

Re: Upgrade 19.2.6 to 20.2.4 changes auth_cipher config
by Patrick Donnelly 04 Sep '26

04 Sep '26

On Fri, Sep 4, 2026 at 3:06 AM Eugen Block <ceph-users(a)ceph.io> wrote: > > Oops, I hit send a bit too early. > > I wanted to ask if this is by design or if it's worth a bug report. It looks like a bug. Please make a ticket. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Maintenance Notification: Mailing List Migration 9/4
by David Galloway 04 Sep '26

04 Sep '26

Hello, Just a quick heads up that the mailing lists will be migrating back into the Sepia lab on 9/4 at 10am Eastern time. Aside from no longer needing to pay AWS for the service, additional benefits we’ll see from this migration include: * Upgrade from mailman 3.3.1 to 3.3.10. This version includes a task runner that will automatically clean up stuck list subscriptions. This was the main cause for the Web UI hanging and going down over the past few years. * Postgres DB migration from 9.6 to highly available Crunchy DB based on Postgres 16 * Lab-wide SMTP relay * Modern anti-spam stack (postscreen, rspamd) * In-house DKIM signing (mail-tester.com rates us a 8.9/10 score) * Mailing alias (vs. an entire list) support * Proper reverse DNS (helps our spam score) The migration itself shouldn’t take more than 30 minutes. It’s already been tested and everything that can be staged ahead of time is already ready. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway(a)ibm.com IBM

1 3

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 04 Sep '26

04 Sep '26

I wonder if the upgrade actually finished then. What's the output of 'ceph orch upgrade status'? Mask sensitive data if necessary. Is it paused? If it is, is there a reason for the pause in the output? You could try resuming it: ceph orch upgrade resume You could redeploy a single daemon and see if that works: ceph orch daemon redeploy mgr.XXX quay.io/ceph/ceph:19.2.6 Or restart the upgrade again in case it was incomplete. Note that this will again rotate all service keyrings (no client keyrings will be touched). Regards, Eugen Zitat von Stig Telfer <ceph-users(a)ceph.io>: > I hit the same issue in the docs. > > Following the upgrade from 19.2.5 to 19.2.6, I still see MDS, OSD > and MGR services failing the health check > AUTH_INSECURE_SERVICE_KEY_TYPE. It looks like Cephadm hit an > unexpected issue and never completed the job of key rotation. > > Is there a way to trigger Cephadm to attempt this again? There's a > lot of daemons to manually process otherwise. > > Thanks, > Stig > >> On 20 Aug 2026, at 14:47, Adam King <ceph-users(a)ceph.io> wrote: >> >> That command got removed for the time being as it used an old way of doing >> the key rotation that wasn't built to work with the changes for the cve and >> seemed to be broken for a number of daemon types. Since the name would make >> people want to use it it was decided to just temporarily disables it so we >> can fix it and reintroduce it in a new version as an easier way to rotate >> the keys. For any keys tied to a cephadm deployed daemon that aren't >> automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings >> during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type >> aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using >> nfs exports managed through the nfs module the export's keyrings also need >> to be rotated. `ceph auth rotate <export-keyring-entity> --key-type >> aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > >> export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. >> Between rotating the keyrings for the nfs cluster/daemons and rotating the >> keyring for the individual export(s) those export(s) are unavailable iirc. >> >> On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: >> >>> >>> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: >>>> Hi, >>>> >>>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The >>> following cephadm managed client keys were not rotated automatically: >>>> >>>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) >>>> client.ceph-exporter. >>>> client.crash. >>>> client.osd-lockbox.. >>>> client.rgw. >>>> >>>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* >>> this command does not exist. >>> >>> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing >>> there, but present on 20.2.3. >>> >>>> >>> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… >>>> >>> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… >>>> >>>> What is the correct way to rotate these client keys via cephadm as part >>> of the cve fix? >>>> >>>> Thanks for any guidance. >>>> Best Regards >>>> Max >>>> >>>> ________________________________ >>>> From: Patrick Donnelly <ceph-users(a)ceph.io> >>>> Sent: 19 August 2026 18:08 >>>> To: ceph-users <ceph-users(a)ceph.io> >>>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 >>> released >>>> >>>> Warning: This email is from an unusual correspondent. >>>> Warning: Make sure this is someone you trust. >>>> >>>> Today, August 19th, 2026, the Ceph project has released Tentacle >>>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>>> components. >>>> >>>> This is the sixth backport release in the Squid series. >>>> >>>> This is the fourth backport release in the Tentacle series. >>>> >>>> Additional details will be posted on the Ceph website/blog: >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… >>>> >>>> We strongly recommend that all Ceph operators upgrade to one of these >>>> releases as soon as possible. >>>> >>>> >>>> ## Critical Upgrade Steps >>>> >>>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>>> This is the first time Ceph has introduced a new key type for CephX >>>> credentials. Therefore, a new procedure exists for upgrading and >>>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>>> cluster, the operator should familiarize themselves with the procedure >>>> documented at: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> These steps are for package-based deployments of Ceph. In other >>>> sections, the document also includes new information about key >>>> rotation procedures, monitor authentication controls, and emergency >>>> recovery procedures. >>>> >>>> Deployments using cephadm will automate the process except for client >>>> keys. You may notice that Cephadm spends more time than normal on the >>>> upgrade after updating all daemon images. This is due to a new process >>>> rotating the OSD and MDS keys. Please be patient. There are plans to >>>> improve this in the future. >>>> >>>> Deployments using Rook will also automate rotation of some client keys >>>> with some exclusions. Look to the Rook project’s announcement for more >>>> details. >>>> >>>> Client and kernel upgrades are recommended to support aes256k. Note >>>> that upstream Linux kernel client support began in kernel 7.0 and has >>>> been backported to CentOS Stream 9 and 10. Check with your >>>> distribution vendor for backported key support before rotating client >>>> keys used by the kernel. >>>> >>>> As part of upgrading a Ceph cluster, six new health warnings and >>>> errors will be generated. This is normal. As part of the process to >>>> upgrade/rotate entity keys, you will address the warnings and errors >>>> as you go. These are documented beginning here: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> The CephX documentation has been brought up to date and vetted for >>>> accuracy. Those changes are live here: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>>> requests with `host` and `x-amz-` headers not included in the signed >>>> subset. Unfortunately, the REST client used in multisite was >>>> generating such improperly signed requests. >>>> >>>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>>> option to true before you begin to upgrade. After all clusters are >>>> upgraded, set the option to ``false`` again. >>>> >>>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>>> secrets stored in the Monitor config-key store will be forthcoming. In >>>> the meantime, operators of cephadm-managed clusters are strongly >>>> encouraged to rotate the cephadm SSH key, as an established process >>>> exists: >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> Rotating other secrets is advised but the process is not yet >>>> established. Operators should assess their cluster's potential >>>> exposure to determine if immediate Monitor config-key secret rotation >>>> is required for their environment. >>>> >>>> >>>> ## CVE Fixes >>>> >>>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>>> of AES-CBC. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>>> in improper verification of a cryptographic signature. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>>> subscription handler. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>>> cryptographic signatures in RGW's SigV4 verifier. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> ## Other Media >>>> >>>> A Cephalocon 2025 talk covers some of the details surrounding the new >>>> CephX key upgrade process. >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… >>>> >>>> ## Getting Ceph >>>> ------------ >>>> * Source at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… >>>> * v19.2.6 Tarball at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>>> * v20.2.4 Tarball at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>>> * Containers at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… >>>> * For packages, see >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * For upgrade with cephadm, see: >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>>> >>>> -- >>>> Patrick Donnelly, Ph.D. >>>> He / Him / His >>>> Red Hat Partner Engineer >>>> IBM, Inc. >>>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> -- >>>> This message has been checked by Libraesva ESG and is found to be clean. >>>> Follow this link to report as spam/bad: >>>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad >>>> Follow this link to blocklist sender: >>>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Issue with encrypted OSDs when upgrading from version 19.2.4 to 19.2.6 using cephadm
by Eugeniusz Pokora 04 Sep '26

04 Sep '26

When upgrading Ceph cluster from version 19.2.4 to 19.2.6 using cephadm, an issue occurred with encrypted OSDs. 2026-09-02 10:53:28,494 7f24e08141c0 DEBUG Determined image: 'quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089' 2026-09-02 10:53:28,501 7f24e08141c0 INFO Reconfig daemon osd.36 ... 2026-09-02 10:53:29,885 7f24e08141c0 INFO Stopping osd.36 to update osd_key bluestore label 2026-09-02 10:53:31,670 7f24e08141c0 INFO Opening osd.36 with crypsetup 2026-09-02 10:53:33,955 7f24e08141c0 INFO Non-zero exit code 5 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /tmp/cryptsetup_action.sh --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 -e NODE_NAME=k08r06cph03 -v /dev:/dev -v /tmp/cephadm-osd-36-rotate-helper.sh:/tmp/cryptsetup_action.sh quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 2026-09-02 10:53:33,956 7f24e08141c0 INFO /usr/bin/podman: stderr Cannot use device /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or mounted). 2026-09-02 10:53:33,956 7f24e08141c0 ERROR ERROR: Got error rotating osd keyring while using cryptsetup tool Out: Err:Cannot use device /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or mounted). The issue did not occur on all hosts in the cluster (fortunately, it was a test env), but it affected all OSDs on affected host. After error occurred OSDs remained in a down state and had to be up manually. After several attempts, the update was successfully completed, but we had to rotate the OSD keyrings ourselves. A few hours latter when monmap was changed and cephadm scheduled a reconfig this problem happened again, on 2 out of 4 nodes of the cluster all osds goes down due to cephadm reconfig and this same error: "Cannot use device... which is in use (already mapped or mounted)". Has anyone encountered similar problems? Regards Eugeniusz P

1 0

Issue with encrypted OSDs when upgrading from version 19.2.4 to 19.2.6 using cephadm
by Eugeniusz Pokora 04 Sep '26

04 Sep '26

1 0

Upgrade 19.2.6 to 20.2.4 changes auth_cipher config
by Eugen Block 04 Sep '26

04 Sep '26

Hi, I just tested the upgrade from a single-node mini cluster (only mgr and mon daemons) bootstrapped with 19.2.6 to 20.2.4. After the bootstrap I have no warnings (except that I don't have any OSDs), this is the excerpt from the mon dump: min_mon_release 19 (squid) election_strategy: 1 0: [v2:192.168.124.6:3300/0,v1:192.168.124.6:6789/0] mon.tentacle auth_epoch 0 auth_service_cipher aes256k auth_allowed_ciphers aes256k auth_preferred_cipher aes256k After the upgrade to 20.2.4 the mon dump looks like this (look at auth_allowed_ciphers): min_mon_release 20 (tentacle) election_strategy: 1 0: [v2:192.168.124.6:3300/0,v1:192.168.124.6:6789/0] mon.tentacle auth_epoch 0 auth_service_cipher aes256k auth_allowed_ciphers aes, aes256k auth_preferred_cipher aes256k Obviously, the cluster now warns about the MONs allowing insecure keys: > Monitors are configured to allow auth using insecure key types > Monitors are configured to allow creation of insecure key types

1 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Stig Telfer 03 Sep '26

03 Sep '26

I hit the same issue in the docs. Following the upgrade from 19.2.5 to 19.2.6, I still see MDS, OSD and MGR services failing the health check AUTH_INSECURE_SERVICE_KEY_TYPE. It looks like Cephadm hit an unexpected issue and never completed the job of key rotation. Is there a way to trigger Cephadm to attempt this again? There's a lot of daemons to manually process otherwise. Thanks, Stig > On 20 Aug 2026, at 14:47, Adam King <ceph-users(a)ceph.io> wrote: > > That command got removed for the time being as it used an old way of doing > the key rotation that wasn't built to work with the changes for the cve and > seemed to be broken for a number of daemon types. Since the name would make > people want to use it it was decided to just temporarily disables it so we > can fix it and reintroduce it in a new version as an easier way to rotate > the keys. For any keys tied to a cephadm deployed daemon that aren't > automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings > during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type > aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using > nfs exports managed through the nfs module the export's keyrings also need > to be rotated. `ceph auth rotate <export-keyring-entity> --key-type > aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > > export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. > Between rotating the keyrings for the nfs cluster/daemons and rotating the > keyring for the individual export(s) those export(s) are unavailable iirc. > > On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > >> >> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: >>> Hi, >>> >>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The >> following cephadm managed client keys were not rotated automatically: >>> >>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) >>> client.ceph-exporter. >>> client.crash. >>> client.osd-lockbox.. >>> client.rgw. >>> >>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* >> this command does not exist. >> >> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing >> there, but present on 20.2.3. >> >>> >> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… >>> >> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… >>> >>> What is the correct way to rotate these client keys via cephadm as part >> of the cve fix? >>> >>> Thanks for any guidance. >>> Best Regards >>> Max >>> >>> ________________________________ >>> From: Patrick Donnelly <ceph-users(a)ceph.io> >>> Sent: 19 August 2026 18:08 >>> To: ceph-users <ceph-users(a)ceph.io> >>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 >> released >>> >>> Warning: This email is from an unusual correspondent. >>> Warning: Make sure this is someone you trust. >>> >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… >>> >>> ## Getting Ceph >>> ------------ >>> * Source at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… >>> * v19.2.6 Tarball at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>> * v20.2.4 Tarball at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>> * Containers at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… >>> * For packages, see >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * For upgrade with cephadm, see: >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >>> -- >>> Patrick Donnelly, Ph.D. >>> He / Him / His >>> Red Hat Partner Engineer >>> IBM, Inc. >>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> -- >>> This message has been checked by Libraesva ESG and is found to be clean. >>> Follow this link to report as spam/bad: >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad >>> Follow this link to blocklist sender: >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph Days Chicago Call For Presentations
by Anthony Middleton 03 Sep '26

03 Sep '26

Hello, Ceph friends! I hope this week is going well. I'm sending this email on behalf of the Ceph Days Chicago <https://ceph.io/en/community/events/2026/ceph-days-chicago/> organizers. We are pleased to announce that Ceph Days Chicago 2026 is accepting submissions for presentations and talks. This is an opportunity to share your experience, insights, and contributions with the Ceph community. We are seeking presentations that cover: - Project updates and recent improvements - Ceph deployment and operational experience - Use cases and real-world applications - Technical architecture and design discussions - Roadmap and future directions Important Dates - Call for Papers closes: *Thursday, October 15, 2026* - Event date: *November 16, 2026* - Location: *University of Chicago, 1201 E 60th St, Chicago, IL 60637* Submission Details To submit your proposal, please visit: https://forms.gle/vdcHycWuWTEyDbxz8 The Ceph Days Chicago team welcomes submissions from community members at all experience levels. For questions about your submission or event details, please contact ceph-days-chicago(a)ceph.io. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

umbrella 21.1.1 RC2 QE validation status
by Yuri Weinstein 03 Sep '26

03 Sep '26

Details of this release are summarized here: https://tracker.ceph.com/issues/78413 Release Notes - email version for RCs https://github.com/ceph/ceph/pull/70240#issuecomment-5232852937 @Neha Ojha <nojha(a)redhat.com> @Patrick Donnelly <pdonnell(a)redhat.com> LRC upgrade - TBD PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: PASSED rados: Kamoltat, Radek, Nitzan, Afreen (dashboard), Travis (rook) rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane, Guillaume rbd: Ilya krbd: Ilya nvmeof: PASSED upgrade: Kamoltat upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: PASSED upgrade-clients:client-upgrade-squid-umbrella-umbrella: PASSED perf-basic: PASSED crimson-rados: Matan powercycle: Brad (same as before) TIA

1 1