- ceph-users - lists.ceph.io

RGW fix for CVE-2026-54330 can break some S3 clients
by Sascha Lucas 31 Aug '26

31 Aug '26

Hi, it seems not spread into this list yet? If you have upgraded to ceph 20.2.4/19.2.6 some S3 clients like restic[1] or php[2] can break due to a new introduced ceph BUG: https://tracker.ceph.com/issues/79674 Affected users can temporary disable the fix at runtime without any restarts: $ ceph config set client.rgw rgw_sigv4_insecure true until 20.2.5/19.2.7 gets releases. Thanks, Sascha. [1] https://github.com/restic/restic/issues/22033 [2] https://lp.zeroservices.eu/articles/ceph-sigv4-when-a-cve-fix-breaks-your-c…

1 0

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Eugen Block 31 Aug '26

31 Aug '26

If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. Zitat von Brent Kennedy <ceph-users(a)ceph.io>: > Question, sort of related. If upgrading from squid to tentacle, > will it automatically try to rotate keys are part of the cluster > upgrade or is that still a manual process? We would like to avoid > the key rotation for now as we have some proxmox nodes using ceph > storage in some of the clusters. > > Regards, > -Brent > > Existing Clusters: > US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > > > > -----Original Message----- > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Saturday, August 29, 2026 10:38 AM > To: Saif Mohammad <samdto987(a)gmail.com> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Clarification on Client Key Rotation > During Upgrade to 19.2.6/20.2.4 > > Hello, > > On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: >> >> Hi, >> >> While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these >> versions introduce the secure key type, I have a clarification >> regarding client key rotation. >> For the client.admin key, we need to rotate the key and then >> import/update the keyring. >> For the other client keys, do we only need to rotate the key and >> verify that the key has changed? Is there no need to import/update >> the keyring for the other clients, since there is no local keyring >> available? > > Step (9) of [1] says: > > "Then copy and import the key to each machine using that client.$ID > credential." > > [1] > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Brent Kennedy 30 Aug '26

30 Aug '26

Question, sort of related. If upgrading from squid to tentacle, will it automatically try to rotate keys are part of the cluster upgrade or is that still a manual process? We would like to avoid the key rotation for now as we have some proxmox nodes using ceph storage in some of the clusters. Regards, -Brent Existing Clusters: US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds -----Original Message----- From: Patrick Donnelly <ceph-users(a)ceph.io> Sent: Saturday, August 29, 2026 10:38 AM To: Saif Mohammad <samdto987(a)gmail.com> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 Hello, On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: > > Hi, > > While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. > For the client.admin key, we need to rotate the key and then import/update the keyring. > For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? Step (9) of [1] says: "Then copy and import the key to each machine using that client.$ID credential." [1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Patrick Donnelly 29 Aug '26

29 Aug '26

Hello, On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: > > Hi, > > While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. > For the client.admin key, we need to rotate the key and then import/update the keyring. > For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? Step (9) of [1] says: "Then copy and import the key to each machine using that client.$ID credential." [1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Missing Ubuntu 24.04 (noble) packages for Tentacle 20.2.3 and 20.2.4
by Homayoon Sadeghi 29 Aug '26

29 Aug '26

Hello, Tracker: https://tracker.ceph.com/issues/79633 Ubuntu 24.04 LTS (noble) Debian packages were published for Tentacle 20.2.1 and 20.2.2, but are absent from 20.2.3 and 20.2.4. The rolling debian-tentacle repository has also lost its noble suite. Because 20.2.4 is the hotfix for CVE-2025-30156, CVE-2026-39944, CVE-2026-50152 and CVE-2026-54330, operators running cephadm on Ubuntu 24.04 currently have no upstream path to install patched host tooling (cephadm, ceph-common, ceph-volume) at a version matching the 20.2.4 container image. download.ceph.com coverage: Release jammy noble bookworm 20.2.0 present missing (pulled) present 20.2.1 present present present 20.2.2 present present present 20.2.3 present missing present 20.2.4 present missing present debian-tentacle (rolling) present missing present Squid (19.2.2–19.2.6) never published a noble suite. Noble in Tentacle therefore only exists for 20.2.1–20.2.2. 20.2.0 noble was published and then removed on 20 Nov 2025 because ceph-mgr crashed on Python 3.12 (tracker #73930). The announcement said Ubuntu 24.04 would return in a later point release. 20.2.1 and 20.2.2 did; 20.2.3 and 20.2.4 did not. Reproduce: curl -sI https://download.ceph.com/debian-20.2.2/dists/noble/Release # HTTP/1.1 200 OK curl -sI https://download.ceph.com/debian-20.2.3/dists/noble/Release curl -sI https://download.ceph.com/debian-20.2.4/dists/noble/Release # HTTP/1.1 404 Not Found curl -s https://download.ceph.com/debian-20.2.4/dists/ curl -s https://download.ceph.com/debian-tentacle/dists/ # bookworm/ jammy/ curl -s https://download.ceph.com/debian-20.2.2/dists/noble/main/binary-amd64/Packa… \ | grep -A3 -E '^Package: (cephadm|ceph-common|ceph-volume)$' # Version: 20.2.2-1noble The Ubuntu archive only has Squid 19.2.3-0ubuntu0.24.04.3 for noble, so download.ceph.com is the only upstream Tentacle source for this distro. Jammy 20.2.4-1jammy debs are not usable on noble (ABI). This looks like a release-matrix / publish gap, not "noble cannot build": v20.2.4 commit 7f793731f1b39eb4f465e960113d2363c311b964 Jenkins ceph-dev-pipeline/8318 https://shaman.ceph.com/builds/ceph/tentacle/7f793731f1b39eb4f465e960113d23… That job built jammy (20.2.4-1jammy), bookworm, Rocky 10, CentOS 9. It did not build noble. Those other distros were copied to download.ceph.com; noble was not. v20.2.3 commit 06c2f9c35b67055a8a6fb99d1be236b3c4832ace got a late noble CI rebuild on 27 Aug (20.2.3-1noble) that was never copied to debian-20.2.3/dists/noble/. tentacle/latest/ubuntu/noble currently points at that unpublished 20.2.3 repo. Expected: 20.2.4-1noble under debian-20.2.4/dists/noble/ and debian-tentacle/dists/noble/, matching 20.2.1 and 20.2.2. Questions: 1. Was noble coverage dropped after 20.2.2, or is this a break in the release/publishing pipeline? 2. If intentional, can this be documented and announced, given 20.2.1/20.2.2 already shipped noble? 3. Can noble packages be produced for 20.2.4 out-of-band, given the CVE severity and the "upgrade as soon as possible" guidance? Thanks, Homayoon Sadeghi

1 0

Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Saif Mohammad 28 Aug '26

28 Aug '26

Hi, While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? We’re managing the cluster using cephadm. Could someone please confirm if my understanding is correct? Thanks, Mohammad Saif

1 0

Re: auth_service_cipher after 19.2.6 upgrade and other clients
by Christopher Durham 28 Aug '26

28 Aug '26

Eugen, Here is what you requested. # ceph mon dump | grep cipherdumped monmap epoch 5auth_service_cipher aesauth_allowed_ciphers aes, aes256kauth_preferred_ciphers aes I suspect that I am able set set auth_service_cipher to aes256k but wanted to be sure before I do so. -Chris On Friday, August 28, 2026 at 11:14:57 AM MDT, Eugen Block <eblock(a)nde.ag> wrote: Hi, can you share 'ceph mon dump |grep cipher'? If you use cephadm, the orchestrator should already have that done for you. The warning you're seeing will clear after a few hours automatically. If you don't use cephadm, you can safely set 'ceph mon set auth_service_cipher aes256k' because it really does only affect the core services, otherwise I'd have a problem. ;-) I upgraded our production (cephadm) cluster this week and all went well. We also can't upgrade our clients at this time, so we're all in the same boat, I guess. :-) Regards, Eugen Zitat von Christopher Durham <ceph-users(a)ceph.io>: > Hi: > After update to 19.2.6 I am so far successfully updating key types > for various daemons. I have the following messages in the output of > 'ceph health detail': > > [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth > service keys using insecure key types > rotating service keys for mon using insecure key type: aes > rotating service keys for mds using insecure key type: aes > rotating service keys for osd using insecure key type: aes > rotating service keys for mgr using insecure key type: aes > The doc says to set auth_service_cipher to aes256k. Before I change > that, I have a question: > Is this setting ONLY for mon/mds/osd/mgr? > I ask because i have several client keys that cannot be updated to > aes256k yet, such as the keys for ceph-csi in kubernetes. I want to > be sure that if update auth_service_cipher to aes256k, I am ONLY > affecting the rotating keys for mon/mds/osd/mgr. I do not want to > set auth_service_cipher to aes256k and then have the ceph-csi still > able to authenticate with older aes keys but not able to do rotating > keys after authentication ( I do not know if it does ).. Thanks for > any information > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: auth_service_cipher after 19.2.6 upgrade and other clients
by Eugen Block 28 Aug '26

28 Aug '26

Hi, can you share 'ceph mon dump |grep cipher'? If you use cephadm, the orchestrator should already have that done for you. The warning you're seeing will clear after a few hours automatically. If you don't use cephadm, you can safely set 'ceph mon set auth_service_cipher aes256k' because it really does only affect the core services, otherwise I'd have a problem. ;-) I upgraded our production (cephadm) cluster this week and all went well. We also can't upgrade our clients at this time, so we're all in the same boat, I guess. :-) Regards, Eugen Zitat von Christopher Durham <ceph-users(a)ceph.io>: > Hi: > After update to 19.2.6 I am so far successfully updating key types > for various daemons. I have the following messages in the output of > 'ceph health detail': > > [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth > service keys using insecure key types > rotating service keys for mon using insecure key type: aes > rotating service keys for mds using insecure key type: aes > rotating service keys for osd using insecure key type: aes > rotating service keys for mgr using insecure key type: aes > The doc says to set auth_service_cipher to aes256k. Before I change > that, I have a question: > Is this setting ONLY for mon/mds/osd/mgr? > I ask because i have several client keys that cannot be updated to > aes256k yet, such as the keys for ceph-csi in kubernetes. I want to > be sure that if update auth_service_cipher to aes256k, I am ONLY > affecting the rotating keys for mon/mds/osd/mgr. I do not want to > set auth_service_cipher to aes256k and then have the ceph-csi still > able to authenticate with older aes keys but not able to do rotating > keys after authentication ( I do not know if it does ).. Thanks for > any information > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

auth_service_cipher after 19.2.6 upgrade and other clients
by Christopher Durham 28 Aug '26

28 Aug '26

Hi: After update to 19.2.6 I am so far successfully updating key types for various daemons. I have the following messages in the output of 'ceph health detail': [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes The doc says to set auth_service_cipher to aes256k. Before I change that, I have a question: Is this setting ONLY for mon/mds/osd/mgr? I ask because i have several client keys that cannot be updated to aes256k yet, such as the keys for ceph-csi in kubernetes. I want to be sure that if update auth_service_cipher to aes256k, I am ONLY affecting the rotating keys for mon/mds/osd/mgr. I do not want to set auth_service_cipher to aes256k and then have the ceph-csi still able to authenticate with older aes keys but not able to do rotating keys after authentication ( I do not know if it does ).. Thanks for any information -Chris

1 0

ceph orch device zap can't wipe db/wal remainders
by Eugen Block 28 Aug '26

28 Aug '26

Hi, while I was investigating something else I noticed that the orchestrator still seems to fail wiping DB/WAL devices, it only wipes device paths like /dev/sdX. I couldn't find anything on tracker, only an old thread where I stated the same: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/H3KQGPIZXH… So the 'ceph orch device zap' command would only work for a standalone OSD. If the OSD has a separate DB/WAL you can't specify the entire device path of the fast device since it's usually the DB/WAL device for multiple OSDs. But you also can't specify the VG/LV syntax as you can with 'cephadm ceph-volume lvm zap --destroy VG/LV': host1:~ # ceph orch device zap host6 /dev/ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18 --force Error EINVAL: Device path '/dev/ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18' not found on host 'host6' Obviously, the preferred choice in such a case would be utilizing ceph-volume which can zap both VGs/LVs: host6:~ # cephadm ceph-volume lvm zap --destroy ceph-f0b8e0c6-840e-4ed5-ade3-de232e35a4b4/osd-block-6e58bd57-86ed-463b-8e8c-8ab64eb3e42f ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18 Is there anything planned for the orchestrator to support that as well? Thanks! Eugen

1 0