- ceph-users - lists.ceph.io

v21.1.1 cephadm sudo hardening
by Eugen Block 30 Sep '26

30 Sep '26

Hi, I managed to bootstrap a single-node cluster with v21.1.1, I wanted to check and test the sudo hardening that's already documented in latest: https://docs.ceph.com/en/latest/cephadm/host-management/#sudo-hardening But the command 'ceph cephadm prepare-host-and-enable-sudo-hardening...' doesn't exist. root@debian:/# ceph -v ceph version 21.1.1 (815bc6b6f9b4e0fd98daed552ec8e7123a0a8f84) umbrella (dev) It only contains the usual prepare-host command: root@debian:/# ceph cephadm -h | grep prepare cephadm prepare-host <host> [<addr>] The respective PR (https://github.com/ceph/ceph/pull/66864) was merged into main in June, but I assume that this won't be part of Umbrella. Is that assumption correct? Thanks, Eugen

2 2

Uploaded 270TB of data to a pool without BULK flag set on.
by merp 30 Sep '26

30 Sep '26

Hello, On my new ceph cluster consisting of 3 servers (45 22TB disks on each) I have created an EC8+4 pool for rgw application and forgot to add BULK flag on before uploading 270TB worth of data on it. Now the data is there and everything is fine except that pool is with 128 PG which should probably be 1024 and cluster is stuck backfilling all the time and seems to reset progress whenever it reaches ~5% misplaced object threshold, probably because of ceph balancer. The actual question is what should I do - wait indefinitely until cluster recovers and add that BULK flag to the pool or add it right now and allow ceph to scale to some more reasonable amount of PG for this pool and let it then continue with backfill? So far I have temporarily disabled deepscrub as in combination with backfill that grinds recovery speed to a halt. Ceph health detail reports only warnings about not deep-scrubbed PGs, everything else looks in order, pg stat is as follows: 450 pgs: 2 active+remapped+backfill_wait, 12 active+remapped+backfilling, 436 active+clean; 276 TiB data, 442 TiB used, 2.6 PiB / 3.0 PiB avail; 93827081/1377582381 objects misplaced (6.811%); 33 MiB/s, 13 objects/s recovering Let me know if more information about the cluster might be helpful. Thanks in advance, Merp.

2 1

Ceph Foundation Q3 2026 Newsletter
by Anthony Middleton 29 Sep '26

29 Sep '26

Hey Ceph friends, I hope all is well. Here's the Ceph Foundation Q3 2026 Newsletter. During the third quarter, the Ceph Foundation focused on strengthening the project's governance structure and continuing to expand our global event footprint. This edition of our quarterly newsletter covers the trademark transition to the Linux Foundation, a new member joining the Foundation, recaps from KCD x Ceph x OpenInfra Day Korea and the Ceph Developer Summit for Vampire, an updated pitch deck for prospective members, and a new transparency initiative around board meeting recordings. If you have an idea for a Ceph-related event, outreach effort, or project, we encourage you to submit a funding request and work with the Foundation to help bring it to life. In This Issue - Ceph Trademark Moves to the Linux Foundation - CSC and Foundation Board meeting highlights - Welcome New Member: Long Van - KCD x Ceph x OpenInfra Day Korea 2026 Recap - Ceph Developer Summit: Vampire - Updated Ceph Foundation Pitch Deck - Board Meeting Recordings Now on YouTube ------------------------------ Ceph Trademark Moves to the Linux Foundation The Ceph trademark has officially transferred from IBM/Red Hat to the Linux Foundation. This change places ownership of the Ceph name and mark under the same neutral, vendor-independent umbrella that already supports the Foundation's governance, legal, and financial operations. This transfer marks the completion of the Ceph Foundation's integration into the Linux Foundation. We highlighted this in our Q1 newsletter <https://ceph.io/en/community/cq/Ceph_Quarterly_Update_2026_Q1.pdf>. The move reinforces Ceph's standing as an open, community-governed project rather than one tied to a single commercial sponsor. It also aligns trademark stewardship with how the project is actually run today, through the CSC and the Governing Board, rather than with the company that created it. Why it matters: Vendor-neutral trademark ownership is a marker of project maturity that many organizations look for before adopting or contributing to open source infrastructure. It signals that Ceph's direction is set by its community and governing bodies, not by any one company's roadmap. ------------------------------ Governance Roles and Accountability The Board and CSC used this quarter's joint meeting to clarify the roles and responsibilities of the three pillars supporting the project: the Linux Foundation, the CSC, and the Foundation Board. Matthew offered a working definition for each: the Linux Foundation provides stewardship along with business services such as legal, accounting, and banking, paid for as needed; the Foundation serves as a neutral place to raise funds and recruit members in support of the project; and the CSC sets the project's technical direction, with the Board contributing industry perspective without directing that work. The Foundation's financial health continues to improve. It now sits above its minimum cushion, defined as twice the highest membership tier, added two new members this year (one associate, one silver), and brought on two full-time staff, Anthony and Emmanuel. That stronger position means the Foundation can now consider funding lab hardware and has opened discussion on a 2027 conference. Key Takeaways - Roles across the Linux Foundation, CSC, and Foundation Board were clarified, though some open questions may remain. - Ceph trademark migration to the Linux Foundation is complete. - A full inventory of Linux Foundation benefits and tools will be requested so the CSC can evaluate them as a set. ------------------------------ Welcome New Members: Long Van The Ceph Foundation is pleased to welcome Long Van <https://longvan.net/> as a Silver member. New members bring fresh perspective and resources to the project, and their participation helps fund the events, documentation, and community programs covered elsewhere in this newsletter. We look forward to working with Long Van to expand their involvement in the Ceph community. To learn more about Long Van and other Ceph Foundation members, visit ceph.io/foundation <https://ceph.io/en/foundation/members/>. ------------------------------ Ceph Days Continue to GrowKCD x Ceph x OpenInfra Day Korea 2026 <https://ceph.io/en/community/events/2026/ceph-days-korea/> The Ceph community joined forces for a joint conference celebrating Kubernetes Community Days (KCD), Ceph Day, and OpenInfra Day in a single event with multiple technical tracks. Ceph Ambassadors Myoungwon Oh and Sungmin Lee played a major role in bringing this community-led event together. Thank you both for your support and dedication to the Ceph project. - Date: September 1, 2026 - Location: Baekbeom Kim Koo Museum & Library, Seoul, South Korea - Format: Joint conference with parallel technical tracks across the three communities - Attendance: Nearly 400 attendees - Call for Papers: Closed with 40 submissions across all three organizations. - Schedule: 7 Ceph-related talks - Registration & Sponsors: Managed through an external event platform; confirmed sponsors included 4 Platinum (AWS, Seagate, Nutanix, Hitachi), 1 Gold, and 4 Silver. Combining three community events into one gave attendees access to a wider range of technical content in a single event. It gave Ceph a stronger presence within the broader cloud-native and open infrastructure audience in the region. Check the upcoming and tentative Ceph Days events for the rest of 2026, including Ceph Days Zürich <https://ceph.io/en/community/events/2026/ceph-days-zurich/>, Chicago <https://ceph.io/en/community/events/2026/ceph-days-chicago/>, and Ceph Meetup Melbourne <https://ceph.io/en/community/events/2026/ceph-meetup-melbourne/>. ------------------------------ Ceph Developer Summit: Vampire The Ceph Developer Summit for the upcoming Vampire release, scheduled for Spring 2027, ran online from August 4–13, 2026. Nine sessions brought together contributors across the major Ceph components to plan the next release cycle. Sessions covered: - RADOS — Notes (August 4) <https://pad.ceph.com/p/cds-vampire-RADOS> - NVMe-oF Gateway — Notes (August 6) <https://pad.ceph.com/p/cds-vampire-NVMe-oFgateway> - Telemetry / Ceph-MGR — Notes (August 10) <https://ceph.io/en/community/events/2026/ceph-days-chicago/> - BlueStore — Notes (August 11) <https://pad.ceph.com/p/cds-vampire-BlueStore> - Cephadm — Notes (August 11) <https://pad.ceph.com/p/cds-vampire-cephadm> - Dashboard — Notes (August 12) <https://pad.ceph.com/p/cds-vampire-dashboard> - CephFS — Notes (August 12) <https://pad.ceph.com/p/cds-vampire-CephFS> - RGW — Notes (August 12) <https://pad.ceph.com/p/cds-vampire-rgw> - RBD — Notes (August 13) <https://pad.ceph.com/p/cds-vampire-RBD> - Crimson / SeaStore — Notes (August 13) <https://pad.ceph.com/p/cds-vampire-crimson> - Performance — Notes (August 13) <https://pad.ceph.com/p/cds-vampire-performance> The full schedule and session links are available at Ceph Events <https://t.ly/CDS-Vampire>, and recordings from every session are posted to the Ceph YouTube playlist <https://www.youtube.com/playlist?list=PLAYbnnO0qf9U>. These sessions give the community visibility into what's shaping the Vampire release well before it ships, and give anyone who couldn't attend live a way to catch up on the technical direction for RADOS, storage backends, gateways, and tooling. Thank you to every component lead for taking the time to lead a session and help keep the Ceph project moving forward. ------------------------------ Updated Ceph Foundation Pitch Deck The Ceph Foundation pitch deck has been refreshed for the first time in more than four years. The updated version is available on the Foundation's about page <https://ceph.io/en/foundation/join/>. The deck is built for organizations already running Ceph in production, giving them a clear case for joining the Foundation and supporting the project that underpins their infrastructure. If your organization works with a company using Ceph, this is a useful resource to share with their leadership. ------------------------------ Board Meeting Recordings Now on YouTube The Governing Board has started publishing meeting recordings on YouTube, a new step toward greater transparency around the Foundation's budget, priorities, and upcoming events. Community members can now follow board discussions directly rather than relying solely on written summaries, giving a clearer view into how funding and program decisions get made. Find the playlist of board meetings here <https://www.youtube.com/playlist?list=PLRSaZR21yGac>. ------------------------------ Get Involved Interested in organizing a meetup, hosting a Ceph Days, speaking at an event, or promoting Ceph in your region? Community participation is what makes these events successful. If you need support for a Ceph event, meetup, travel, or community campaign, submit a funding request to the Ceph Foundation. The Governing Board and Linux Foundation staff review requests and consider them based on available budget and community impact. Ceph Foundation Community Request Form <https://form.asana.com/?k=f6mdd2cDTRFLO-bgiN_xWw&d=9283783873717> Ceph Days Planning Guide <https://docs.google.com/document/d/1SY1UVUeVRJoOsUIGtZoD1ipko1QyBDCj15v2Y15…> Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

September Ceph Tech Talk Reminder
by Anthony Middleton 29 Sep '26

29 Sep '26

Join us tomorrow for the next Ceph Tech Talk: *Using AI and Slack to Streamline Datacenter Remote Hands Tasks*. Ceph Ambassador *Anthony D’Atri *will explain how AI and Slack automation can make remote hands operations, often the most manual and error-prone part of running distributed storage at scale, more efficient. If your team manages datacenter operations for Ceph clusters, you’ll learn practical tools you can use right away. Find the event on the Ceph Community Calendar using the link below. *September 30, 2026, at 10 am PDTJoin: https://meet.jit.si/ceph-tech-talks <https://meet.jit.si/ceph-tech-talks>* Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Grafana, Prometheus, client certificates and mgmt-gateway in 20.2.3
by Robert Sander 29 Sep '26

29 Sep '26

Hi, we stumbled across an issue that is introduced with the new mgmt-gateway service. As soon as it is active the Prometheus service is configured to require client certificates for its API. It looks like the Grafana service is configured with client certificates according to the new service spec but it does not seem to use them. The result is no metrics in the Grafana dashboards. Can anyone confirm this behaviour? After removing the mgmt-gateway service and redeploying all monitoring services everything works again. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 11

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Chris Palmer 27 Sep '26

27 Sep '26

Yeah, I manually replaced the old key in each hosts authorized_keys file (just before I saw your email!). Then it went away. This is a recurring sort of theme with cephadm though. It is supposed to abstract away many of the implementation details. But then the documentation doesn't cover it, so you have to guess what it should have done, and then fix it manually... Thanks, Chris On 26/08/2026 13:43, Eugen Block wrote: > Hi, > > did you add the new pub key to all the host's authorized_keys? > > clear-key and then generate-key is the correct way. > > Zitat von Chris Palmer <ceph-users(a)ceph.io>: > >> I've attempted to rotate the SSH keys on a 20.2.4 test cluster, and >> now it is broken: >> >> The note below points towards the SSH Configuration section. That >> doesn't have anything specific about rotation, so I had to make some >> guesses. >> >> It says that a new SSH key can be generated with "ceph cephadm >> generate-key". It doesn't give any error, but using "ceph cephadm >> get-pub-key" suggests it hasn't actually done anything (I still get >> the original ssh-rsa key). The doc probably needs clarifying. >> >> So I tried "ceph cephadm clear-key" which worked, and then "ceph >> cephadm generate-key" again. This time I can see a new ssh-ed25519 >> key. I then failed all MGRs. >> >> Now however I have this for every host: >> >> root@c1:~# ceph cephadm check-host c2 >> check-host failed: >> Failed to connect to c2 at address (192.168.nnn.162): Failed to >> connect to c2 (192.168.nnn.162). Permission denied >> Log: Opening SSH connection to 192.168.nnn.162, port 22 >> [conn=9] Connected to SSH server at 192.168.nnn.162, port 22 >> [conn=9] Local address: 192.168.nnn.161, port 35514 >> [conn=9] Peer address: 192.168.nnn.162, port 22 >> [conn=9] Beginning auth for user root >> [conn=9] Auth failed for user root >> [conn=9] Connection failure: Permission denied >> [conn=9] Aborting connection >> >> Have I done this incorrectly, or is the documentation wrong? What is >> the correct way to rectify this? >> >> Thanks, Chris >> >> On 19/08/2026 17:07, Patrick Donnelly wrote: >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> https://ceph.io/en/news/blog/ >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… >>> >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> https://docs.ceph.com/en/latest/rados/operations/health-checks/index.html#a… >>> >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> https://docs.ceph.com/en/latest/architecture/index.html#high-availability-a… >>> >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >>> https://docs.ceph.com/en/latest/cephadm/host-management/#ssh-configuration >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and-cephx-keys-… >>> >>> >>> ## Getting Ceph >>> ------------ >>> * Source at https://github.com/ceph/ceph >>> * v19.2.6 Tarball at >>> https://download.ceph.com/tarballs/ceph-19.2.6.tar.gz >>> * v20.2.4 Tarball at >>> https://download.ceph.com/tarballs/ceph-20.2.4.tar.gz >>> * Containers at https://quay.io/repository/ceph/ceph >>> * For packages, see >>> https://docs.ceph.com/en/latest/install/get-packages/ >>> * For upgrade with cephadm, see: >>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: Performance issue after tentacle upgrade
by Moti 24 Sep '26

24 Sep '26

Hi Oguzhan, Thanks for your reply I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening? Regards On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users(a)ceph.io> wrote: > Hey Moti, > > Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > > I recently upgraded from 18.2.4 to 20.2.1 > > > > I did notice this issue: > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… > > but that's not much similar to our case since we don't see any jump in > CPU or memory usage. > > Although the symptoms don't look the same, I'd recommend upgrading to > v20.2.3 to get the fix for this issue and the other fixes that went in: > https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle > > After v20.2.3 upgrade, if you still see the same issue, then I'd suggest > collecting "perf" output per process per component (rgw, osd mainly) to > further analyze the bottleneck. > > You can also quickly check rgw logs to see what RGW reports as the latency > for those PUT/GET/HEAD requests whether it matches the latency you see on > your end (your wkld generator); especially for HEAD where no data streaming > happens. RGW logs "start" and "end" of a request. This is to rule out > whether bottleneck is outside of ceph (e.g., load-balancers). > > Cheers, > Oguzhan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 3

September Ceph Foundation Board Meeting
by Anthony Middleton 23 Sep '26

23 Sep '26

Hey Ceph friends, Quick question! Do you remember.... the Ceph Foundation Board meeting of September? If not, you can watch it on YouTube. Here's the link: https://youtu.be/NZ6k62ZDmRM <https://youtu.be/NZ6k62ZDmRM>. During the meeting, the board discussed the new Ceph Foundation Pitch Deck, a request for technical content, Emmanuel's vision for the Ceph Docs, and upcoming events in the Ceph ecosystem. Please reach out to me if you're looking for ways to contribute to the Ceph project. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 22 Sep '26

22 Sep '26

Good morning to all! After a couple of months I had the time to look again into the problem with 'SignatureDoesNotMatch' when accessing the "Object" section of the Ceph Dashboard. I've upgraded my cluster to 20.2.4 but the dashboard was still getting the error. Trying to debug I notice that RGW_API_SSL_VERIFY was "false", I enabled it (switching it to True) (all the RGW are running with "legit" SSL certificates) and the error message changed in "Certificate did not match expected hostname". Ok, we are on something, I said. The cluster is running with short hostanmes and the certifcate has FQDN. I tinkered with some variables but to non avail... Until today when I was gathering some logs to ask further help on this mailing-list, I put RGW_API_SSL_VERIFY to False and the dashboard worked again!!! I don't think that switching RGW_API_SSL_VERIFY from False to True and then to False again, did the job. Yesterday I did a couple of "experiment" with different parameters, but once the tests were done I always revert them to the orginal values. Appart from "rgw_dns_name" which I deleted (and not re-created)... Could be "rgw_dns_name" the culprit? It was the FQDN of the virtual ip used to access the cluster. Have a great day! Cheers Iztok On 27/05/26 14:41, Eugen Block via ceph-users wrote: > I see what you mean, in our Squid cluster the curl request from within > the mgr container fails with the same error. So I should probably expect > that we're gonna hit the same dashboard issue once we upgrade to > Tentacle, I guess. > > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > >> On 27/05/26 12:47, Eugen Block via ceph-users wrote: >>> I'm not sure about the curl versions. >> >> Curl was only a tool and it could be unrelated to the problem I'm >> facing, but it was interesting that when I try to query >> >> # curl "https://rgw.domain.com/admin/metadata/user" --aws-sigv4 >> "aws:amz:ZONEGROUP:s3" --user "DASHBOARDKEY:DASHBOARDSECRET" >> >> it worked fine with newer versions. But the older ones (like the one >> shipped with the MGR container) failed with the >> 'SignatureDoesNotMatch' error. When I tried with more verbosity I see >> that the oldest version is "signing" only 2 headers >> (SignedHeaders=host;x-amz-date) and the newer ones 3 >> (SignedHeaders=host;x-amz-content-sha256;x-amz-date), but this could >> be due to '--aws-sigv4' option in curl. >> >> >>> Did you notice the response from > "C U" the day before yesterday? He >>> or she commented: >>> >>>> fqdn on the dashboard needs working certificate validation. >>> >>>> "Squid did cert work inline → bad cert raised crypto.Error → caught >>>> and rewrapped as ServerConfigException cleanly. Tentacle does cert >>>> work via a >>> subprocess; failures can now come from (a) the fork itself, (b) >>> PyO3 init crashing in the child (the documented subinterpreter >>> collision), or (c) >>> genuine cert errors marshalled back as JSON. When (b) happens, the >>> dashboard module's cert provisioning never completes, the module ends up >>> half-initialized, and downstream dashboard→RGW calls fail with >>> SignatureDoesNotMatch because the dashboard is operating without >>> properly loaded >>> credentials/sigv4 plumbing." >> >> I did some tests using openssl s_client from the mgr container to >> verify the certificates and I had no problems. Clearly I wasn't using >> python, but unless is not using the system paths for certificates, it >> shouldn't matter. >> >> Iztok >> >>> >>> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >>> >>>> On 27/05/26 12:17, Eugen Block via ceph-users wrote: >>>>> Hi, >>>>> >>>>> >>>>> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >>>>> >>>>>> On 27/05/26 09:36, Eugen Block via ceph-users wrote: >>>>>>> The description in this tracker [0] looks identical, but it is >>>>>>> supposed to be fixed. Do you have rgw_dns_name set? >>>>>> >>>>>> Yes, at certain point I configured client.rgw rgw_dns_name to the >>>>>> FQDN belonging to the virtual IP of the ingress service. >>>>> >>>>> does it mean that it works now? >>>> >>>> No, still doesn't work. To try I've also remove it and then again >>>> set it, but without luck. >>>> >>>> I also tried to call the API endpoint directly with curl using the >>>> dashboard credentials and found that from 'older' versions of curl >>>> (provided for example by Rocky 9) the command gives me the same >>>> error ('SignatureDoesNotMatch'), but when I was using newer one >>>> (8.5.0 on Ubuntu 22.04 and 8.9.1 on Rocky 10) it correctly answer my >>>> query. >>>> >>>> Could it be the problem? >>>> >>>>> >>>>>> BTW, how you 'unset' a configuration variable, you just put a >>>>>> empty string or you use the rm option? >>>>>> >>>>>> ceph config set client.rgw rgw_dns_name '' >>>>>> or >>>>>> ceph config rm client.rgw rgw_dns_name >>>>>> ? >>>>> >>>>> That's the correct one (config rm). >>>> >>>> Thank you! I sometimes get lost in the documentation :-) >>>> >>>> Cheers >>>> Iztok >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 3

Rotating cephx client keys in openstack VMs
by Bartosz Rabiega 21 Sep '26

21 Sep '26

Hello, I'm researching a way to rotate cephx client keys to aes256k 100% online for openstack VMs (qemu) with ceph rbd volumes attached (librbd). The problem ----------- Unfortunately I don't see any reasonable way to do it. Obviously I might be missing something (or a lot of something) Usually libvirt configures VM disk with following auth: <disk ...> <auth username='cephxuser'> <secret type='ceph' uuid='id-of-libvirt-secret'/> </auth> </disk> The cephx key is stored in a libvirt secret, it is read when VM starts and then stored in RAM for a VM lifetime (librbd AFAIK). If we rotate cephxuser key the living VMs will eventually stop working - when either auth_mon_ticket_ttl is exhausted or some connection to mon needs to be re-established librbd will try to authenticate with old key and fail. Potential solutions ------------------- A - rotate key, and distribute it as the same libvirt secret: - running VMs will eventually stop working - hard reboot or live migration shall fix them - stopped (SHUTOFF) VMs will be OK when started - new VMs and new attachments will be OK B - use new user and new key - new libvirt secret - running VMs will be OK, as long as we keep old user and key, live migration will probably fix the VM and use new user and new key - stopped (SHUTOFF) VMs will be OK when started, but will fail when old user/key is removed. username and libvirt secret uuid is kept in nova database and nothing will update it - new VMs and new attachments will be OK C - use the same user and have 2 keys for it - old and new. Eventually remove old one - running VMs will be OK, they will switch to new key during live migration - stopped (SHUTOFF) VMs will be OK, libvirt secret points to the new key so when VM is started it will use it. User is the same so no DB update is required - new VMs and new attachments will be OK A - not acceptable, breaks running VMs B - very difficult, requires a lot of VM migration, requires a lot of work to fix SHELVED/SHUTOFF VMs (no tooling for that AFAIK?) C - looks the best but not possible, ceph doesn't allow multiple keys per single user Summary ------- If my understanding is correct the effort to rotate the keys for openstack is tremendous and requires nova/cinder DB modification to fix SHUTOFF vms. Probably SHELVED vms will have the same problem. If you reached this point, thanks for reading :) Any ideas or comments? Bartosz Rabiega

3 2