- ceph-users - lists.ceph.io

Re: v20.2.4 staggered upgrade
by Eugen Block 27 Aug '26

27 Aug '26

It got worse: reef1:~ # ceph orch ps --daemon-type mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.reef1.eqokui reef1 *:9283,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765 running (3h) 7m ago 23h 198M - 20.2.4 6facee348180 472d9e5cd38a mgr.reef3.wiuxme reef3 *:8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765 running (3h) 7m ago 23h 144M - 20.2.4 6facee348180 820c2422172f I started from scratch (purged the previous cluster) and upgraded from: 18.2.2 -> 18.2.5 -> 18.2.6 -> 18.2.7 -> 18.2.8 -> 19.2.0 -> 19.2.1 -> 19.2.2 -> 19.2.3 -> 19.2.4 -> 19.2.5 -> 20.2.0 -> 20.2.1 -> 20.2.3 -> 20.2.4 So these are 14 Upgrades but I'm counting 39 and 27 entries for port 8765, so that doesn't seem to correlate. I have no idea yet where that comes from :-D Maybe there were some leftovers on those VMs that somehow confused something. I have other test clusters and our production cluster as well on either 20.2.4 or 19.2.6 and can't see this effect anywhere else (yet). Zitat von Jean-Marie de Boer <ceph-users(a)ceph.io>: > I have tested a staggered upgrade from 19.2.4 to 19.2.6. > > Indeed the OSD keys got rotated multiple times, I guess it may > always happen when upgrading, even if it's a single daemon type > that's later in the sequence. It's not really a problem but it does > make everything take quite a bit longer (we have 1000+ OSD's on > production clusters and must do a staggered upgrade because we need > to manage the MDS failover). I think the devs are aware though. > > I do not observe this port multiplication however. > > mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 > running (7h) 42s ago 5w 478M - 19.2.6 > ff836bb28e7d 251c5d067daf > mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 > running (14m) 42s ago 5w 475M - 19.2.6 > ff836bb28e7d 45c38fb90d31 > mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 > running (7h) 42s ago 5w 675M - 19.2.6 > ff836bb28e7d 1d69113ae7ed > > However I have been observing some strangeness with daemons showing > non-standard status: > > mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 > running (6h) 3m ago 5w 478M - 19.2.6 > ff836bb28e7d 251c5d067daf > mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 > unknown 8m ago 5w 339M - 19.2.6 > ff836bb28e7d 98add61a7280 > mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 > running 0s ago 5w - - <unknown> > <unknown> <unknown> A node failed to finish starting its > containers without a clear reason why, it also refused to exit > maintenance, and a mon on another node failed and fixed itself > without intervention. I have not really determined a clear cause. > > Most of that fixed itself eventually although I now notice a single > crash daemon playing up. > > Of course we're making it hard because we do a staggered upgrade and > we use multiple routed subnets too :) > > Regards, > JM > > > On 8/26/26 12:21, Eugen Block wrote: >> I haven't seen anything like this yet: >> >> mgr.reef1.zfjbzv reef1 >> *:9283,8765,8765,8765,8765,8765,8765 running (2m) 109s ago >> 18h 129M - 20.2.4 6facee348180 ed9ae39b8f5f >> mgr.reef2.omwexa reef2 >> *:8765,8765,8765,8765,8765,8765,8765 running (2m) 108s ago >> 14h 167M - 20.2.4 6facee348180 b24464d5c98b >> >> I just did a staggered upgrade in a test cluster from 18.2.7 to >> 20.2.4 and I noticed that it goes through redeploying all daemons >> with a new keyring everytime although they already had been >> upgraded, for example MGR, MON and OSD when I was trying to upgrade >> MDS only (--daemon-type MDS). >> >> A 'ceph mgr fail' does not clean up those redundant port entries. >> Should I create a tracker for that or is this already known? >> >> I haven't tried the same on 19.2.6 yet. >> >> Regards, >> Eugen >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > > Jean-Marie de Boer > Specialist > > SURF | Science Park 140 | 1098 XG Amsterdam | www.surf.nl > E: jean-marie.deboer(a)surf.nl | M: +31610961787 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 2

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Anthony D'Atri 27 Aug '26

27 Aug '26

I'd no idea either > On Aug 27, 2026, at 6:12 AM, Eugen Block <ceph-users(a)ceph.io> wrote: > > Apparently I either forgot that I have or really never have used 'ceph config set --force' before. :-D

1 0

Best way to find out the client key support
by Albert Shih 27 Aug '26

27 Aug '26

Hi, Our ceph client (cephfs) mostly run Debian 12/13 with the native pkg of ceph-common. And currenlty if I check the https://download.ceph.com/debian-20.2.4 they are still not the ceph-common & co for Debian Trixie. We also have proxmox client (rdb) and openshift (rooks) client. So what's the best way to check if the client can use aes256k key ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 27 août 2026 14:22:01 CEST

1 0

Re: Rotation key after upgrade to 20.2.4
by Albert Shih 27 Aug '26

27 Aug '26

Le 27/08/2026 à 10:10:09+0000, Eugen Block a écrit Hi, > > please read all the sections in the docs ;-) > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > Although you use cephadm, step 6 is still relevant because it explains the > warning you're seeing: I read that, the problem is I just understand they are two types of keys :-( :-( > > > 6. Wipe the rotating service keys. > > Warning > > This is not recommended for most deployments. It is best to let your > > rotating service keys expire after a few hours (using default TTL). > > > Warning > > Only perform this step if all service daemons have upgraded binaries > > that understand the new cipher type. > > If you want to immediately clear the > > AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning, you can wipe the > > existing rotating service key database on the Monitors: > > > This is just a transitional state, the warning will go away after a few > hours (I'm not sure how long exactly). You're 100% right. The message just vannish. Thanks. Regards. -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 27 août 2026 14:19:14 CEST

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 27 Aug '26

27 Aug '26

Apparently I either forgot that I have or really never have used 'ceph config set --force' before. :-D Thanks for pointing it out, Boris and Chris. Zitat von Christopher Durham <ceph-users(a)ceph.io>: > Boris, et al: > What this appears to mean is that the rgw_sigv4_insecure option does > not exist in releases prior to 19.2.6/20.2.4, but using 'force' sets > it anyway, so that when a 19.2.6 or 20.2.4 version of radosgw > starts, it will see it and use it. > I suspect that if a 19.2.6 or 20.2.4 radosgw starts without this > option it will fail to do replication to say an earlier radosgw > without the signature fix on the other side of a multisite that has > not yet updated. As we always have, we plan on doing the update on > one side of the multisite on day 1, and the other side on day 2. The > force option seems to make this possible without having to modify > ceph.conf everywhere. > If I am wrong, please someone correct me. Thanks. > -ChrisOn Tuesday, August 25, 2026 at 01:49:30 AM MDT, Boris > <bb(a)kervyn.de> wrote: > > Hi Matthew, > > we've set it with: > ceph config set client.rgw rgw_sigv4_insecure true --force > > Best > Boris > > > > Am Di., 25. Aug. 2026 um 09:41 Uhr schrieb Matthew Vernon > <ceph-users(a)ceph.io>: >> >> Hi, >> >> On 19/08/2026 17:09, Patrick Donnelly wrote: >> >> > If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> > option to true before you begin to upgrade. After all clusters are >> > upgraded, set the option to ``false`` again. >> >> Stupid question - how/where does one set this, and will this be >> understood by older versions of Ceph? >> >> I ask because I have a Reef multisite setup, which should be supported >> to upgrade to Tentacle. Which I fear I'm now going to have to do in a hurry! >> >> AFAICT this is a new config setting, so I'd expect older versions of >> Ceph won't understand it... [and searching the reef docs doesn't find it] >> >> Thanks, >> >> Matthew >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend > im groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Rotation key after upgrade to 20.2.4
by Eugen Block 27 Aug '26

27 Aug '26

Hi, please read all the sections in the docs ;-) https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… Although you use cephadm, step 6 is still relevant because it explains the warning you're seeing: > 6. Wipe the rotating service keys. > Warning > This is not recommended for most deployments. It is best to let your > rotating service keys expire after a few hours (using default TTL). > Warning > Only perform this step if all service daemons have upgraded binaries > that understand the new cipher type. > If you want to immediately clear the > AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE warning, you can wipe the > existing rotating service key database on the Monitors: This is just a transitional state, the warning will go away after a few hours (I'm not sure how long exactly). Regards, Eugen Zitat von Albert Shih <ceph-users(a)ceph.io>: > Hi everyone > > So today I upgrade (one of) my ceph cluster (cephadm), and indeed after > upgrading some time all osd come back to life and all osd key seem to be in > aes256k > > But still with ceph health detail I've got > > [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth > service keys using insecure key types > rotating service keys for mon using insecure key type: aes > rotating service keys for mds using insecure key type: aes > rotating service keys for osd using insecure key type: aes > rotating service keys for mgr using insecure key type: aes > > When I look in the documentation > > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > section 3 it seem I had to copy the keyring directly to the nodes of my > cluster who run those service, but inside > > /var/lib/ceph/CEPHID/* > > I didn't find any keyring file related to mon/mds/osd/mgr with aes key. > > So I'm a little confuse...in the documentation it's say in a note: > > The mon. historically has not been managed by the Monitor auth > database; it exists soley in each Monitor’s keyring inside its data > directory. This suggested rotation procedure now puts the > authoritative copy in the auth database alongside other keys. The > Monitor keyring persists as a fallback or emergency key. > > so where are those rotation key ? > > Regards > > -- > Albert SHIH 🦫 🐸 > France > Heure locale/Local time: > jeu. 27 août 2026 11:50:08 CEST > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Rotation key after upgrade to 20.2.4
by Albert Shih 27 Aug '26

27 Aug '26

Hi everyone So today I upgrade (one of) my ceph cluster (cephadm), and indeed after upgrading some time all osd come back to life and all osd key seem to be in aes256k But still with ceph health detail I've got [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes When I look in the documentation https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… section 3 it seem I had to copy the keyring directly to the nodes of my cluster who run those service, but inside /var/lib/ceph/CEPHID/* I didn't find any keyring file related to mon/mds/osd/mgr with aes key. So I'm a little confuse...in the documentation it's say in a note: The mon. historically has not been managed by the Monitor auth database; it exists soley in each Monitor’s keyring inside its data directory. This suggested rotation procedure now puts the authoritative copy in the auth database alongside other keys. The Monitor keyring persists as a fallback or emergency key. so where are those rotation key ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 27 août 2026 11:50:08 CEST

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Albert Shih 27 Aug '26

27 Aug '26

Le 20/08/2026 à 19:55:02+0000, Salatiel Filho a écrit > Hi, I also miss in the docs the steps for those clients that are not > daemons? ( cephadm managed ) > > entity client.admin using insecure key type: aes They are a section (8) for the client.admin key rotation in the official documentation : https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… I didn't do it currently so no way to tell if that's work or not. Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 27 août 2026 11:48:06 CEST

1 0

Re: Ubuntu 24.04 Noble package support for Ceph 20.2.x
by Casey Bodley 27 Aug '26

27 Aug '26

according to Patrick in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/UEGCE3TKWY…, ubuntu 24.04 support will be added for tentacle in 20.2.5 On Wed, Aug 26, 2026 at 11:41 AM Martin Vobruba <ceph-users(a)ceph.io> wrote: > > Hi, > > I'd like to ask about the status of Ubuntu 24.04 Noble package support for > the Ceph 20.2.x release series. When I originally deployed the cluster, > packages for Noble were available and our clients are currently running > Ceph 20.2.2 on Ubuntu 24.04. However, I noticed that Noble packages are no > longer available for newer 20.2.x releases. I would like to upgrade the > clients to 20.2.4, primarily to get the fix for the insecure keys issue. > > Is the lack of Noble packages temporary? If so, is there any estimate for > when Ubuntu 24.04 packages for the 20.2.x branch will become available > again? Also, is there a recommended upgrade path for Noble clients > currently running 20.2.2 in the meantime? > > Thanks for any clarification. > > Best regards, > Martin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Christopher Durham 26 Aug '26

26 Aug '26

Boris, et al: What this appears to mean is that the rgw_sigv4_insecure option does not exist in releases prior to 19.2.6/20.2.4, but using 'force' sets it anyway, so that when a 19.2.6 or 20.2.4 version of radosgw starts, it will see it and use it. I suspect that if a 19.2.6 or 20.2.4 radosgw starts without this option it will fail to do replication to say an earlier radosgw without the signature fix on the other side of a multisite that has not yet updated. As we always have, we plan on doing the update on one side of the multisite on day 1, and the other side on day 2. The force option seems to make this possible without having to modify ceph.conf everywhere. If I am wrong, please someone correct me. Thanks. -ChrisOn Tuesday, August 25, 2026 at 01:49:30 AM MDT, Boris <bb(a)kervyn.de> wrote: Hi Matthew, we've set it with: ceph config set client.rgw rgw_sigv4_insecure true --force Best Boris Am Di., 25. Aug. 2026 um 09:41 Uhr schrieb Matthew Vernon <ceph-users(a)ceph.io>: > > Hi, > > On 19/08/2026 17:09, Patrick Donnelly wrote: > > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > > option to true before you begin to upgrade. After all clusters are > > upgraded, set the option to ``false`` again. > > Stupid question - how/where does one set this, and will this be > understood by older versions of Ceph? > > I ask because I have a Reef multisite setup, which should be supported > to upgrade to Tentacle. Which I fear I'm now going to have to do in a hurry! > > AFAICT this is a new config setting, so I'd expect older versions of > Ceph won't understand it... [and searching the reef docs doesn't find it] > > Thanks, > > Matthew > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0