- ceph-users - lists.ceph.io

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Patrick Donnelly 29 Aug '26

29 Aug '26

Hello, On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: > > Hi, > > While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. > For the client.admin key, we need to rotate the key and then import/update the keyring. > For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? Step (9) of [1] says: "Then copy and import the key to each machine using that client.$ID credential." [1] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Missing Ubuntu 24.04 (noble) packages for Tentacle 20.2.3 and 20.2.4
by Homayoon Sadeghi 29 Aug '26

29 Aug '26

Hello, Tracker: https://tracker.ceph.com/issues/79633 Ubuntu 24.04 LTS (noble) Debian packages were published for Tentacle 20.2.1 and 20.2.2, but are absent from 20.2.3 and 20.2.4. The rolling debian-tentacle repository has also lost its noble suite. Because 20.2.4 is the hotfix for CVE-2025-30156, CVE-2026-39944, CVE-2026-50152 and CVE-2026-54330, operators running cephadm on Ubuntu 24.04 currently have no upstream path to install patched host tooling (cephadm, ceph-common, ceph-volume) at a version matching the 20.2.4 container image. download.ceph.com coverage: Release jammy noble bookworm 20.2.0 present missing (pulled) present 20.2.1 present present present 20.2.2 present present present 20.2.3 present missing present 20.2.4 present missing present debian-tentacle (rolling) present missing present Squid (19.2.2–19.2.6) never published a noble suite. Noble in Tentacle therefore only exists for 20.2.1–20.2.2. 20.2.0 noble was published and then removed on 20 Nov 2025 because ceph-mgr crashed on Python 3.12 (tracker #73930). The announcement said Ubuntu 24.04 would return in a later point release. 20.2.1 and 20.2.2 did; 20.2.3 and 20.2.4 did not. Reproduce: curl -sI https://download.ceph.com/debian-20.2.2/dists/noble/Release # HTTP/1.1 200 OK curl -sI https://download.ceph.com/debian-20.2.3/dists/noble/Release curl -sI https://download.ceph.com/debian-20.2.4/dists/noble/Release # HTTP/1.1 404 Not Found curl -s https://download.ceph.com/debian-20.2.4/dists/ curl -s https://download.ceph.com/debian-tentacle/dists/ # bookworm/ jammy/ curl -s https://download.ceph.com/debian-20.2.2/dists/noble/main/binary-amd64/Packa… \ | grep -A3 -E '^Package: (cephadm|ceph-common|ceph-volume)$' # Version: 20.2.2-1noble The Ubuntu archive only has Squid 19.2.3-0ubuntu0.24.04.3 for noble, so download.ceph.com is the only upstream Tentacle source for this distro. Jammy 20.2.4-1jammy debs are not usable on noble (ABI). This looks like a release-matrix / publish gap, not "noble cannot build": v20.2.4 commit 7f793731f1b39eb4f465e960113d2363c311b964 Jenkins ceph-dev-pipeline/8318 https://shaman.ceph.com/builds/ceph/tentacle/7f793731f1b39eb4f465e960113d23… That job built jammy (20.2.4-1jammy), bookworm, Rocky 10, CentOS 9. It did not build noble. Those other distros were copied to download.ceph.com; noble was not. v20.2.3 commit 06c2f9c35b67055a8a6fb99d1be236b3c4832ace got a late noble CI rebuild on 27 Aug (20.2.3-1noble) that was never copied to debian-20.2.3/dists/noble/. tentacle/latest/ubuntu/noble currently points at that unpublished 20.2.3 repo. Expected: 20.2.4-1noble under debian-20.2.4/dists/noble/ and debian-tentacle/dists/noble/, matching 20.2.1 and 20.2.2. Questions: 1. Was noble coverage dropped after 20.2.2, or is this a break in the release/publishing pipeline? 2. If intentional, can this be documented and announced, given 20.2.1/20.2.2 already shipped noble? 3. Can noble packages be produced for 20.2.4 out-of-band, given the CVE severity and the "upgrade as soon as possible" guidance? Thanks, Homayoon Sadeghi

1 0

Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Saif Mohammad 29 Aug '26

29 Aug '26

Hi, While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these versions introduce the secure key type, I have a clarification regarding client key rotation. For the client.admin key, we need to rotate the key and then import/update the keyring. For the other client keys, do we only need to rotate the key and verify that the key has changed? Is there no need to import/update the keyring for the other clients, since there is no local keyring available? We’re managing the cluster using cephadm. Could someone please confirm if my understanding is correct? Thanks, Mohammad Saif

1 0

Re: auth_service_cipher after 19.2.6 upgrade and other clients
by Christopher Durham 28 Aug '26

28 Aug '26

Eugen, Here is what you requested. # ceph mon dump | grep cipherdumped monmap epoch 5auth_service_cipher aesauth_allowed_ciphers aes, aes256kauth_preferred_ciphers aes I suspect that I am able set set auth_service_cipher to aes256k but wanted to be sure before I do so. -Chris On Friday, August 28, 2026 at 11:14:57 AM MDT, Eugen Block <eblock(a)nde.ag> wrote: Hi, can you share 'ceph mon dump |grep cipher'? If you use cephadm, the orchestrator should already have that done for you. The warning you're seeing will clear after a few hours automatically. If you don't use cephadm, you can safely set 'ceph mon set auth_service_cipher aes256k' because it really does only affect the core services, otherwise I'd have a problem. ;-) I upgraded our production (cephadm) cluster this week and all went well. We also can't upgrade our clients at this time, so we're all in the same boat, I guess. :-) Regards, Eugen Zitat von Christopher Durham <ceph-users(a)ceph.io>: > Hi: > After update to 19.2.6 I am so far successfully updating key types > for various daemons. I have the following messages in the output of > 'ceph health detail': > > [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth > service keys using insecure key types > rotating service keys for mon using insecure key type: aes > rotating service keys for mds using insecure key type: aes > rotating service keys for osd using insecure key type: aes > rotating service keys for mgr using insecure key type: aes > The doc says to set auth_service_cipher to aes256k. Before I change > that, I have a question: > Is this setting ONLY for mon/mds/osd/mgr? > I ask because i have several client keys that cannot be updated to > aes256k yet, such as the keys for ceph-csi in kubernetes. I want to > be sure that if update auth_service_cipher to aes256k, I am ONLY > affecting the rotating keys for mon/mds/osd/mgr. I do not want to > set auth_service_cipher to aes256k and then have the ceph-csi still > able to authenticate with older aes keys but not able to do rotating > keys after authentication ( I do not know if it does ).. Thanks for > any information > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: auth_service_cipher after 19.2.6 upgrade and other clients
by Eugen Block 28 Aug '26

28 Aug '26

Hi, can you share 'ceph mon dump |grep cipher'? If you use cephadm, the orchestrator should already have that done for you. The warning you're seeing will clear after a few hours automatically. If you don't use cephadm, you can safely set 'ceph mon set auth_service_cipher aes256k' because it really does only affect the core services, otherwise I'd have a problem. ;-) I upgraded our production (cephadm) cluster this week and all went well. We also can't upgrade our clients at this time, so we're all in the same boat, I guess. :-) Regards, Eugen Zitat von Christopher Durham <ceph-users(a)ceph.io>: > Hi: > After update to 19.2.6 I am so far successfully updating key types > for various daemons. I have the following messages in the output of > 'ceph health detail': > > [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth > service keys using insecure key types > rotating service keys for mon using insecure key type: aes > rotating service keys for mds using insecure key type: aes > rotating service keys for osd using insecure key type: aes > rotating service keys for mgr using insecure key type: aes > The doc says to set auth_service_cipher to aes256k. Before I change > that, I have a question: > Is this setting ONLY for mon/mds/osd/mgr? > I ask because i have several client keys that cannot be updated to > aes256k yet, such as the keys for ceph-csi in kubernetes. I want to > be sure that if update auth_service_cipher to aes256k, I am ONLY > affecting the rotating keys for mon/mds/osd/mgr. I do not want to > set auth_service_cipher to aes256k and then have the ceph-csi still > able to authenticate with older aes keys but not able to do rotating > keys after authentication ( I do not know if it does ).. Thanks for > any information > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

auth_service_cipher after 19.2.6 upgrade and other clients
by Christopher Durham 28 Aug '26

28 Aug '26

Hi: After update to 19.2.6 I am so far successfully updating key types for various daemons. I have the following messages in the output of 'ceph health detail': [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes The doc says to set auth_service_cipher to aes256k. Before I change that, I have a question: Is this setting ONLY for mon/mds/osd/mgr? I ask because i have several client keys that cannot be updated to aes256k yet, such as the keys for ceph-csi in kubernetes. I want to be sure that if update auth_service_cipher to aes256k, I am ONLY affecting the rotating keys for mon/mds/osd/mgr. I do not want to set auth_service_cipher to aes256k and then have the ceph-csi still able to authenticate with older aes keys but not able to do rotating keys after authentication ( I do not know if it does ).. Thanks for any information -Chris

1 0

ceph orch device zap can't wipe db/wal remainders
by Eugen Block 28 Aug '26

28 Aug '26

Hi, while I was investigating something else I noticed that the orchestrator still seems to fail wiping DB/WAL devices, it only wipes device paths like /dev/sdX. I couldn't find anything on tracker, only an old thread where I stated the same: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/H3KQGPIZXH… So the 'ceph orch device zap' command would only work for a standalone OSD. If the OSD has a separate DB/WAL you can't specify the entire device path of the fast device since it's usually the DB/WAL device for multiple OSDs. But you also can't specify the VG/LV syntax as you can with 'cephadm ceph-volume lvm zap --destroy VG/LV': host1:~ # ceph orch device zap host6 /dev/ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18 --force Error EINVAL: Device path '/dev/ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18' not found on host 'host6' Obviously, the preferred choice in such a case would be utilizing ceph-volume which can zap both VGs/LVs: host6:~ # cephadm ceph-volume lvm zap --destroy ceph-f0b8e0c6-840e-4ed5-ade3-de232e35a4b4/osd-block-6e58bd57-86ed-463b-8e8c-8ab64eb3e42f ceph-2575e9e1-ba63-4923-a418-230d328fe8f8/osd-db-b59a0d82-76f4-40f0-b0dd-45d96e6b7c18 Is there anything planned for the orchestrator to support that as well? Thanks! Eugen

1 0

Re: v20.2.4 staggered upgrade
by Eugen Block 27 Aug '26

27 Aug '26

It got worse: reef1:~ # ceph orch ps --daemon-type mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.reef1.eqokui reef1 *:9283,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765 running (3h) 7m ago 23h 198M - 20.2.4 6facee348180 472d9e5cd38a mgr.reef3.wiuxme reef3 *:8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765 running (3h) 7m ago 23h 144M - 20.2.4 6facee348180 820c2422172f I started from scratch (purged the previous cluster) and upgraded from: 18.2.2 -> 18.2.5 -> 18.2.6 -> 18.2.7 -> 18.2.8 -> 19.2.0 -> 19.2.1 -> 19.2.2 -> 19.2.3 -> 19.2.4 -> 19.2.5 -> 20.2.0 -> 20.2.1 -> 20.2.3 -> 20.2.4 So these are 14 Upgrades but I'm counting 39 and 27 entries for port 8765, so that doesn't seem to correlate. I have no idea yet where that comes from :-D Maybe there were some leftovers on those VMs that somehow confused something. I have other test clusters and our production cluster as well on either 20.2.4 or 19.2.6 and can't see this effect anywhere else (yet). Zitat von Jean-Marie de Boer <ceph-users(a)ceph.io>: > I have tested a staggered upgrade from 19.2.4 to 19.2.6. > > Indeed the OSD keys got rotated multiple times, I guess it may > always happen when upgrading, even if it's a single daemon type > that's later in the sequence. It's not really a problem but it does > make everything take quite a bit longer (we have 1000+ OSD's on > production clusters and must do a staggered upgrade because we need > to manage the MDS failover). I think the devs are aware though. > > I do not observe this port multiplication however. > > mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 > running (7h) 42s ago 5w 478M - 19.2.6 > ff836bb28e7d 251c5d067daf > mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 > running (14m) 42s ago 5w 475M - 19.2.6 > ff836bb28e7d 45c38fb90d31 > mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 > running (7h) 42s ago 5w 675M - 19.2.6 > ff836bb28e7d 1d69113ae7ed > > However I have been observing some strangeness with daemons showing > non-standard status: > > mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 > running (6h) 3m ago 5w 478M - 19.2.6 > ff836bb28e7d 251c5d067daf > mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 > unknown 8m ago 5w 339M - 19.2.6 > ff836bb28e7d 98add61a7280 > mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 > running 0s ago 5w - - <unknown> > <unknown> <unknown> A node failed to finish starting its > containers without a clear reason why, it also refused to exit > maintenance, and a mon on another node failed and fixed itself > without intervention. I have not really determined a clear cause. > > Most of that fixed itself eventually although I now notice a single > crash daemon playing up. > > Of course we're making it hard because we do a staggered upgrade and > we use multiple routed subnets too :) > > Regards, > JM > > > On 8/26/26 12:21, Eugen Block wrote: >> I haven't seen anything like this yet: >> >> mgr.reef1.zfjbzv reef1 >> *:9283,8765,8765,8765,8765,8765,8765 running (2m) 109s ago >> 18h 129M - 20.2.4 6facee348180 ed9ae39b8f5f >> mgr.reef2.omwexa reef2 >> *:8765,8765,8765,8765,8765,8765,8765 running (2m) 108s ago >> 14h 167M - 20.2.4 6facee348180 b24464d5c98b >> >> I just did a staggered upgrade in a test cluster from 18.2.7 to >> 20.2.4 and I noticed that it goes through redeploying all daemons >> with a new keyring everytime although they already had been >> upgraded, for example MGR, MON and OSD when I was trying to upgrade >> MDS only (--daemon-type MDS). >> >> A 'ceph mgr fail' does not clean up those redundant port entries. >> Should I create a tracker for that or is this already known? >> >> I haven't tried the same on 19.2.6 yet. >> >> Regards, >> Eugen >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > > Jean-Marie de Boer > Specialist > > SURF | Science Park 140 | 1098 XG Amsterdam | www.surf.nl > E: jean-marie.deboer(a)surf.nl | M: +31610961787 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 2

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Anthony D'Atri 27 Aug '26

27 Aug '26

I'd no idea either > On Aug 27, 2026, at 6:12 AM, Eugen Block <ceph-users(a)ceph.io> wrote: > > Apparently I either forgot that I have or really never have used 'ceph config set --force' before. :-D

1 0

Best way to find out the client key support
by Albert Shih 27 Aug '26

27 Aug '26

Hi, Our ceph client (cephfs) mostly run Debian 12/13 with the native pkg of ceph-common. And currenlty if I check the https://download.ceph.com/debian-20.2.4 they are still not the ceph-common & co for Debian Trixie. We also have proxmox client (rdb) and openshift (rooks) client. So what's the best way to check if the client can use aes256k key ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 27 août 2026 14:22:01 CEST

1 0