- ceph-users - lists.ceph.io

Ubuntu 24.04 Noble package support for Ceph 20.2.x
by Martin Vobruba 26 Aug '26

26 Aug '26

Hi, I'd like to ask about the status of Ubuntu 24.04 Noble package support for the Ceph 20.2.x release series. When I originally deployed the cluster, packages for Noble were available and our clients are currently running Ceph 20.2.2 on Ubuntu 24.04. However, I noticed that Noble packages are no longer available for newer 20.2.x releases. I would like to upgrade the clients to 20.2.4, primarily to get the fix for the insecure keys issue. Is the lack of Noble packages temporary? If so, is there any estimate for when Ubuntu 24.04 packages for the 20.2.x branch will become available again? Also, is there a recommended upgrade path for Noble clients currently running 20.2.2 in the meantime? Thanks for any clarification. Best regards, Martin

1 0

Re: Cephadm staggered upgrade --limit parameter not always applied
by Eugen Block 26 Aug '26

26 Aug '26

Hi, I haven't found anything explaining this yet, but I also haven't looked for a while. I just retried limiting RGW upgrade (from Reef to Tentacle): reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.4 --daemon-types rgw --limit 1 But the limit is still ignored: 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:26.554+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:26.555+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef3.boujxu (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef3.boujxu (1/1) All three RGW daemons have been upgraded automatically. I'm pretty sure that it worked at some point with RGWs, in my notes from one of our customer upgrades where this worked: ceph orch upgrade start --image registry:5000/ceph/ceph:v18.2.5 --daemon-types rgw --limit 1 I can't tell when it stopped working properly. Regards, Eugen Zitat von Andreas Eriksson <norrland(a)nullbyte.se>: > Hi, > > Were you able to find anything more about this? I'm looking at > upgrading a bunch of clusters from previous major versions, but do > not want to upgrade all RGW services. And also limit the number of > osds at the time that are being upgraded, to keep the recovery > traffic lower for example. > > Br, > > On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote: >> Adding the dev list >> >> Zitat von Eugen Block <eblock(a)nde.ag>: >> >>> Hi, >>> >>> I did a quick search on tracker but couldn't find anything >>> related. A customer reported this, and I can confirm the behaviour >>> on a lab cluster. I usually perform a staggered upgrade with >>> --daemon-types and --limit, but not for all daemon types. So I >>> haven't stumbled across this yet myself, but our customer did (I >>> can reproduce with --daemon-types as well). They upgraded from >>> latest Reef to latest Squid and reported that despite providing >>> the --limit parameter, the mons were all upgraded. So I tried to >>> reproduce, and the behaviour is not really clear to me, I'll try >>> to clarify. >>> >>> # Start with MGRs >>> >>> reef1:~ # ceph orch upgrade start --image >>> quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1 >>> >>> Upgrading MGRs with limit works, but it doesn't reflect in the MGR >>> log. Usually, I expect a line like this: >>> >>> ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade >>> >>> But there is no such line in the logs. Then I upgrade the rest of the MGRs. >>> >>> # Continue with MONs >>> >>> reef1:~ # ceph orch upgrade start --image >>> quay.io/ceph/ceph:v19.2.3 --services mon --limit 1 >>> >>> This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. >>> And again, no such line in the log (Hit upgrade limit). What I >>> noticed was a MGR respawn after the first MON had been upgraded >>> successfully. Maybe some state of the upgrade progress gets lost >>> during the respawn? I then upgraded the remaining MON. Then >>> ceph-crash is upgraded successfully. >>> >>> # Upgrade OSD >>> >>> reef1:~ # ceph orch upgrade start --image >>> quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1 >>> >>> And this is the first service that actually reports the upgrade limit: >>> >>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO >>> cephadm.upgrade] Upgrade: Updating osd.0 (1/1) >>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Upgrade: Updating osd.0 (1/1) >>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] >>> Hit upgrade limit of 1. Stopping upgrade >>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Hit upgrade limit of 1. Stopping upgrade >>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO >>> cephadm.upgrade] Upgrade: Setting container_image for all nvmeof >>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Upgrade: Setting container_image for all nvmeof >>> 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO >>> cephadm.upgrade] Upgrade: Finalizing container_image settings >>> 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Upgrade: Finalizing container_image settings >>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO >>> cephadm.upgrade] Upgrade: Complete! >>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) >>> log [INF] : Upgrade: Complete! >>> >>> >>> This is really irritating and inconsistent: If the orchestrator >>> does honor --limit with other services than OSD, why isn't that >>> visible in the logs? And what's with the MONs? Why 2 out of 3? >>> >>> Any pointers appreciated! Not sure which Ceph versions might be >>> affected by this, I'll try out a couple more upgrade paths. >>> >>> Thanks, >>> Eugen >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

August Ceph Foundation Board Meeting
by Anthony Middleton 26 Aug '26

26 Aug '26

Hey Ceph friends, I hope you're doing well. Here is the recording of the August Ceph Foundation Board meeting: https://www.youtube.com/watch?v=e7xt4h37rSg. There are several committees community members can join, including Hardware, Ceph Days, Ceph Ambassadors, and Community Outreach. Please reach out to me if you're interested in connecting with any of the committee leaders. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: v20.2.4 staggered upgrade
by Jean-Marie de Boer 26 Aug '26

26 Aug '26

I have tested a staggered upgrade from 19.2.4 to 19.2.6. Indeed the OSD keys got rotated multiple times, I guess it may always happen when upgrading, even if it's a single daemon type that's later in the sequence. It's not really a problem but it does make everything take quite a bit longer (we have 1000+ OSD's on production clusters and must do a staggered upgrade because we need to manage the MDS failover). I think the devs are aware though. I do not observe this port multiplication however. mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 running (7h) 42s ago 5w 478M - 19.2.6 ff836bb28e7d 251c5d067daf mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 running (14m) 42s ago 5w 475M - 19.2.6 ff836bb28e7d 45c38fb90d31 mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 running (7h) 42s ago 5w 675M - 19.2.6 ff836bb28e7d 1d69113ae7ed However I have been observing some strangeness with daemons showing non-standard status: mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 running (6h) 3m ago 5w 478M - 19.2.6 ff836bb28e7d 251c5d067daf mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 unknown 8m ago 5w 339M - 19.2.6 ff836bb28e7d 98add61a7280 mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 running 0s ago 5w - - <unknown> <unknown> <unknown> A node failed to finish starting its containers without a clear reason why, it also refused to exit maintenance, and a mon on another node failed and fixed itself without intervention. I have not really determined a clear cause. Most of that fixed itself eventually although I now notice a single crash daemon playing up. Of course we're making it hard because we do a staggered upgrade and we use multiple routed subnets too :) Regards, JM On 8/26/26 12:21, Eugen Block wrote: > I haven't seen anything like this yet: > > mgr.reef1.zfjbzv reef1 *:9283,8765,8765,8765,8765,8765,8765 > running (2m) 109s ago 18h 129M - 20.2.4 > 6facee348180 ed9ae39b8f5f > mgr.reef2.omwexa reef2 *:8765,8765,8765,8765,8765,8765,8765 > running (2m) 108s ago 14h 167M - 20.2.4 > 6facee348180 b24464d5c98b > > I just did a staggered upgrade in a test cluster from 18.2.7 to 20.2.4 > and I noticed that it goes through redeploying all daemons with a new > keyring everytime although they already had been upgraded, for example > MGR, MON and OSD when I was trying to upgrade MDS only (--daemon-type MDS). > > A 'ceph mgr fail' does not clean up those redundant port entries. Should > I create a tracker for that or is this already known? > > I haven't tried the same on 19.2.6 yet. > > Regards, > Eugen > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Jean-Marie de Boer Specialist SURF | Science Park 140 | 1098 XG Amsterdam | www.surf.nl E: jean-marie.deboer(a)surf.nl | M: +31610961787

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 26 Aug '26

26 Aug '26

Hi, did you add the new pub key to all the host's authorized_keys? clear-key and then generate-key is the correct way. Zitat von Chris Palmer <ceph-users(a)ceph.io>: > I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and > now it is broken: > > The note below points towards the SSH Configuration section. That > doesn't have anything specific about rotation, so I had to make some > guesses. > > It says that a new SSH key can be generated with "ceph cephadm > generate-key". It doesn't give any error, but using "ceph cephadm > get-pub-key" suggests it hasn't actually done anything (I still get > the original ssh-rsa key). The doc probably needs clarifying. > > So I tried "ceph cephadm clear-key" which worked, and then "ceph > cephadm generate-key" again. This time I can see a new ssh-ed25519 > key. I then failed all MGRs. > > Now however I have this for every host: > > root@c1:~# ceph cephadm check-host c2 > check-host failed: > Failed to connect to c2 at address (192.168.nnn.162): Failed to > connect to c2 (192.168.nnn.162). Permission denied > Log: Opening SSH connection to 192.168.nnn.162, port 22 > [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 > [conn=9] Local address: 192.168.nnn.161, port 35514 > [conn=9] Peer address: 192.168.nnn.162, port 22 > [conn=9] Beginning auth for user root > [conn=9] Auth failed for user root > [conn=9] Connection failure: Permission denied > [conn=9] Aborting connection > > Have I done this incorrectly, or is the documentation wrong? What is > the correct way to rectify this? > > Thanks, Chris > > On 19/08/2026 17:07, Patrick Donnelly wrote: >> Today, August 19th, 2026, the Ceph project has released Tentacle >> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >> components. >> >> This is the sixth backport release in the Squid series. >> >> This is the fourth backport release in the Tentacle series. >> >> Additional details will be posted on the Ceph website/blog: >> https://ceph.io/en/news/blog/ >> >> We strongly recommend that all Ceph operators upgrade to one of these >> releases as soon as possible. >> >> >> ## Critical Upgrade Steps >> >> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >> This is the first time Ceph has introduced a new key type for CephX >> credentials. Therefore, a new procedure exists for upgrading and >> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >> cluster, the operator should familiarize themselves with the procedure >> documented at: >> >> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >> >> These steps are for package-based deployments of Ceph. In other >> sections, the document also includes new information about key >> rotation procedures, monitor authentication controls, and emergency >> recovery procedures. >> >> Deployments using cephadm will automate the process except for client >> keys. You may notice that Cephadm spends more time than normal on the >> upgrade after updating all daemon images. This is due to a new process >> rotating the OSD and MDS keys. Please be patient. There are plans to >> improve this in the future. >> >> Deployments using Rook will also automate rotation of some client keys >> with some exclusions. Look to the Rook project’s announcement for more >> details. >> >> Client and kernel upgrades are recommended to support aes256k. Note >> that upstream Linux kernel client support began in kernel 7.0 and has >> been backported to CentOS Stream 9 and 10. Check with your >> distribution vendor for backported key support before rotating client >> keys used by the kernel. >> >> As part of upgrading a Ceph cluster, six new health warnings and >> errors will be generated. This is normal. As part of the process to >> upgrade/rotate entity keys, you will address the warnings and errors >> as you go. These are documented beginning here: >> >> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >> >> The CephX documentation has been brought up to date and vetted for >> accuracy. Those changes are live here: >> >> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >> >> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >> requests with `host` and `x-amz-` headers not included in the signed >> subset. Unfortunately, the REST client used in multisite was >> generating such improperly signed requests. >> >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. >> >> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >> secrets stored in the Monitor config-key store will be forthcoming. In >> the meantime, operators of cephadm-managed clusters are strongly >> encouraged to rotate the cephadm SSH key, as an established process >> exists: >> >> See: >> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >> >> Rotating other secrets is advised but the process is not yet >> established. Operators should assess their cluster's potential >> exposure to determine if immediate Monitor config-key secret rotation >> is required for their environment. >> >> >> ## CVE Fixes >> >> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >> of AES-CBC. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >> >> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >> in improper verification of a cryptographic signature. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >> >> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >> subscription handler. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >> >> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >> cryptographic signatures in RGW's SigV4 verifier. >> >> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >> >> ## Other Media >> >> A Cephalocon 2025 talk covers some of the details surrounding the new >> CephX key upgrade process. >> >> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >> >> ## Getting Ceph >> ------------ >> * Source at https://github.com/ceph/ceph >> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >> * Containers at https://quay.io/repository/ceph/ceph >> * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ >> * For upgrade with cephadm, see: >> https://docs.ceph.com/en/latest/cephadm/upgrade/ >> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 26 Aug '26

26 Aug '26

I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and now it is broken: The note below points towards the SSH Configuration section. That doesn't have anything specific about rotation, so I had to make some guesses. It says that a new SSH key can be generated with "ceph cephadm generate-key". It doesn't give any error, but using "ceph cephadm get-pub-key" suggests it hasn't actually done anything (I still get the original ssh-rsa key). The doc probably needs clarifying. So I tried "ceph cephadm clear-key" which worked, and then "ceph cephadm generate-key" again. This time I can see a new ssh-ed25519 key. I then failed all MGRs. Now however I have this for every host: root@c1:~# ceph cephadm check-host c2 check-host failed: Failed to connect to c2 at address (192.168.nnn.162): Failed to connect to c2 (192.168.nnn.162). Permission denied Log: Opening SSH connection to 192.168.nnn.162, port 22 [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 [conn=9] Local address: 192.168.nnn.161, port 35514 [conn=9] Peer address: 192.168.nnn.162, port 22 [conn=9] Beginning auth for user root [conn=9] Auth failed for user root [conn=9] Connection failure: Permission denied [conn=9] Aborting connection Have I done this incorrectly, or is the documentation wrong? What is the correct way to rectify this? Thanks, Chris On 19/08/2026 17:07, Patrick Donnelly wrote: > Today, August 19th, 2026, the Ceph project has released Tentacle > 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several > components. > > This is the sixth backport release in the Squid series. > > This is the fourth backport release in the Tentacle series. > > Additional details will be posted on the Ceph website/blog: > https://ceph.io/en/news/blog/ > > We strongly recommend that all Ceph operators upgrade to one of these > releases as soon as possible. > > > ## Critical Upgrade Steps > > The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. > This is the first time Ceph has introduced a new key type for CephX > credentials. Therefore, a new procedure exists for upgrading and > rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph > cluster, the operator should familiarize themselves with the procedure > documented at: > > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… > > These steps are for package-based deployments of Ceph. In other > sections, the document also includes new information about key > rotation procedures, monitor authentication controls, and emergency > recovery procedures. > > Deployments using cephadm will automate the process except for client > keys. You may notice that Cephadm spends more time than normal on the > upgrade after updating all daemon images. This is due to a new process > rotating the OSD and MDS keys. Please be patient. There are plans to > improve this in the future. > > Deployments using Rook will also automate rotation of some client keys > with some exclusions. Look to the Rook project’s announcement for more > details. > > Client and kernel upgrades are recommended to support aes256k. Note > that upstream Linux kernel client support began in kernel 7.0 and has > been backported to CentOS Stream 9 and 10. Check with your > distribution vendor for backported key support before rotating client > keys used by the kernel. > > As part of upgrading a Ceph cluster, six new health warnings and > errors will be generated. This is normal. As part of the process to > upgrade/rotate entity keys, you will address the warnings and errors > as you go. These are documented beginning here: > > https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… > > The CephX documentation has been brought up to date and vetted for > accuracy. Those changes are live here: > > https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… > > Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 > requests with `host` and `x-amz-` headers not included in the signed > subset. Unfortunately, the REST client used in multisite was > generating such improperly signed requests. > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. > > Regarding the fix for CVE-2026-50152, formal guidance on rotating all > secrets stored in the Monitor config-key store will be forthcoming. In > the meantime, operators of cephadm-managed clusters are strongly > encouraged to rotate the cephadm SSH key, as an established process > exists: > > See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration > > Rotating other secrets is advised but the process is not yet > established. Operators should assess their cluster's potential > exposure to determine if immediate Monitor config-key secret rotation > is required for their environment. > > > ## CVE Fixes > > * CVE-2025-30156 is an authentication bypass in CephX caused by misuse > of AES-CBC. > > See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 > > * CVE-2026-39944 shares the unauthenticated-encryption root cause of > CVE-2025-30156, but applies it to RGW's STS session tokens resulting > in improper verification of a cryptographic signature. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 > > * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor > subscription handler. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 > > * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 > cryptographic signatures in RGW's SigV4 verifier. > > See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 > > ## Other Media > > A Cephalocon 2025 talk covers some of the details surrounding the new > CephX key upgrade process. > > https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… > > ## Getting Ceph > ------------ > * Source at https://github.com/ceph/ceph > * v19.2.6 Tarball at https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz > * v20.2.4 Tarball at https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * For upgrade with cephadm, see: > https://docs.ceph.com/en/latest/cephadm/upgrade/ > * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 > * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >

1 0

v20.2.4 staggered upgrade
by Eugen Block 26 Aug '26

26 Aug '26

I haven't seen anything like this yet: mgr.reef1.zfjbzv reef1 *:9283,8765,8765,8765,8765,8765,8765 running (2m) 109s ago 18h 129M - 20.2.4 6facee348180 ed9ae39b8f5f mgr.reef2.omwexa reef2 *:8765,8765,8765,8765,8765,8765,8765 running (2m) 108s ago 14h 167M - 20.2.4 6facee348180 b24464d5c98b I just did a staggered upgrade in a test cluster from 18.2.7 to 20.2.4 and I noticed that it goes through redeploying all daemons with a new keyring everytime although they already had been upgraded, for example MGR, MON and OSD when I was trying to upgrade MDS only (--daemon-type MDS). A 'ceph mgr fail' does not clean up those redundant port entries. Should I create a tracker for that or is this already known? I haven't tried the same on 19.2.6 yet. Regards, Eugen

1 0

Re: Cephadm staggered upgrade --limit parameter not always applied
by Andreas Eriksson 26 Aug '26

26 Aug '26

Hi, Were you able to find anything more about this? I'm looking at upgrading a bunch of clusters from previous major versions, but do not want to upgrade all RGW services. And also limit the number of osds at the time that are being upgraded, to keep the recovery traffic lower for example. Br, On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote: > Adding the dev list > > Zitat von Eugen Block <eblock(a)nde.ag>: > >> Hi, >> >> I did a quick search on tracker but couldn't find anything related. A >> customer reported this, and I can confirm the behaviour on a lab >> cluster. I usually perform a staggered upgrade with --daemon-types >> and --limit, but not for all daemon types. So I haven't stumbled >> across this yet myself, but our customer did (I can reproduce with >> --daemon-types as well). They upgraded from latest Reef to latest >> Squid and reported that despite providing the --limit parameter, the >> mons were all upgraded. So I tried to reproduce, and the behaviour is >> not really clear to me, I'll try to clarify. >> >> # Start with MGRs >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 >> --services mgr --limit 1 >> >> Upgrading MGRs with limit works, but it doesn't reflect in the MGR >> log. Usually, I expect a line like this: >> >> ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade >> >> But there is no such line in the logs. Then I upgrade the rest of the >> MGRs. >> >> # Continue with MONs >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 >> --services mon --limit 1 >> >> This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. >> And again, no such line in the log (Hit upgrade limit). What I >> noticed was a MGR respawn after the first MON had been upgraded >> successfully. Maybe some state of the upgrade progress gets lost >> during the respawn? I then upgraded the remaining MON. Then >> ceph-crash is upgraded successfully. >> >> # Upgrade OSD >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 >> --services osd.osd.standalone --limit 1 >> >> And this is the first service that actually reports the upgrade limit: >> >> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating osd.0 (1/1) >> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating osd.0 (1/1) >> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] Hit >> upgrade limit of 1. Stopping upgrade >> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Hit upgrade limit of 1. Stopping upgrade >> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Setting container_image for all nvmeof >> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Upgrade: Setting container_image for all nvmeof >> 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Finalizing container_image settings >> 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Upgrade: Finalizing container_image settings >> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Complete! >> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) log >> [INF] : Upgrade: Complete! >> >> >> This is really irritating and inconsistent: If the orchestrator does >> honor --limit with other services than OSD, why isn't that visible in >> the logs? And what's with the MONs? Why 2 out of 3? >> >> Any pointers appreciated! Not sure which Ceph versions might be >> affected by this, I'll try out a couple more upgrade paths. >> >> Thanks, >> Eugen > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Robert Sander 26 Aug '26

26 Aug '26

Hi, Am 26.08.26 um 8:44 AM schrieb Eugen Block: > according to the "OS Recommendations" [0] Ceph Tentacle is not available > for Debian 13. Umbrella will be built for Trixie. > Are you using cephadm managed clusters or package based? With cephadm > you're not that strictly tied to the installed package versions. In the end all systems need updated client libraries to handle the new key format. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 26 Aug '26

26 Aug '26

Hi, according to the "OS Recommendations" [0] Ceph Tentacle is not available for Debian 13. Umbrella will be built for Trixie. Are you using cephadm managed clusters or package based? With cephadm you're not that strictly tied to the installed package versions. Regards, Eugen [0] https://docs.ceph.com/en/latest/start/os-recommendations/ Zitat von Johan <ceph-users(a)ceph.io>: > Ok, > > on Debian 12 it was not a problem upgrading to ceph 20.2.4 using apt > and https://download.ceph.com/debian-tentacle/ > > With Debian 13 which I have at my troublesome cluster I get several > dependency-issues. > ceph-common is dependent on libicu72 which doesn't have any candidates > python3-cephfs (and others) is dependent on python3 < 3.12 but > 3.13.5-1 will be installed > etc > > Is this because I perhaps hadn't upgraded ceph-common and others > properly before upgrading Debian from 12 to 13? > > /Johan > > Den 2026-08-24 kl. 11:06, skrev Johan: >> Im not a professional computer tech but have two ceph clusters >> (cephadm, Trixie, Debian 12/13). >> >> I upgraded the least sensitive one using the dashboard. When I >> upgrade the admin client I now get >> auth: error parsing file /etc/ceph/ceph.client.admin.keyring: >> error setting modifier for [client.admin] type=key val=<key>: >> Malformed input [buffer:3] >> auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) >> Input/ output error >> >> I get the same respons when using a ordinary user to connect the filesystem. >> >> I found this issue describing the same error >> https://github.com/rook/rook/issues/18240 >> >> As they, in above issue, discussed which version/image of the >> toolbox was used I checked my version of ceph at the command line. >> I get >> ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) >> reef (stable) >> I would guess this is the problem. >> >> How do I get to run up-to-date version on the command line? >> >> /Johan >> >> >> Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> https://ceph.io/en/news/blog/ >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/ >>> index.html#upgrading-and-rotating-cephx-keys >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/ >>> index.html#auth-insecure-keys-creatable >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> https://docs.ceph.com/en/latest/architecture/index.html#high- >>> availability-authentication >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh- >>> configuration >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx- >>> keys-patrick-donnelly-ibm-yehuda-sadeh-weinraub-ubiquiti >>> >>> ## Getting Ceph >>> ------------ >>> * Source at https://github.com/ceph/ceph >>> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ ceph-19.2.6.tar.gz >>> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ ceph-20.2.4.tar.gz >>> * Containers at https://quay.io/repository/ceph/ceph >>> * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ >>> * For upgrade with cephadm, see: >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0