- ceph-users - lists.ceph.io

Subject: Assistance Required: Vault Integration with RADOS Gateway for SSE-S3 Encryption
by Dhivya G 20 Mar '25

20 Mar '25

Dear Ceph Community, We have been researching encryption mechanisms for a while and have successfully deployed Ceph using Kubernetes. Currently, we are working on integrating HashiCorp Vault with RADOS Gateway to enable server-side encryption (SSE-S3) for object uploads. Despite configuring Vault with RADOS Gateway, we are encountering an "INVALID REQUEST" error when attempting to upload objects into an encrypted bucket. Additionally, we are unable to locate any relevant Vault logs that provide insights into this specific error. We would greatly appreciate any insights from those who have faced a similar issue or have successfully configured Vault transit with RADOS Gateway for SSE-S3 encryption. Any guidance or shared experiences would be immensely helpful in resolving this issue. Looking forward to your expert advice. Thanks & Regards, Dhivya G|Associate Software Engineer Ph No: (+91) 9894944910 email: dhivya.g(a)zybisys.com Zybisys Consulting LLP | Srinivasa Tower' No.29, 39th E Cross, 4th T Block, Jayanagar , Bangalore India - 560041http://zybisys.com/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 3

Attention: Documentation
by Joel Davidow 20 Mar '25

20 Mar '25

While reading about mons, I've come across two documentation issues. 1. The 2013 blog "Monitors and Paxos, a chat with Joao" has a broken link. - The broken link returns a 404 - The bad URL is http://ceph.com/dev-notes/cephs-new-monitor-changes/ - That bad URL is on line 8 at https://github.com/ceph/ceph.io/blob/main/src/en/news/blog/2013/monitors-an… - The working URL is https://ceph.io/en/news/blog/2013/cephs-new-monitor-changes/ 2. The 2013 blog "Ceph’s New Monitor Changes" has 3 broken links. - Figures 1, 2, and 3 each have URLs that return 404 - http://ceph.com/wp-content/uploads/2013/03/2013-03-07_17-57-12.png - http://ceph.com/wp-content/uploads/2013/03/2013-03-07_17-58-59.png - http://ceph.com/wp-content/uploads/2013/03/2013-03-07_17-59-49.png - I was unable to locate any of those three png files in Ceph's Github org - It'd be great if those three png files could be found and links updated so I can continue learning about how mons work. Thanks, Joel

4 5

Division by zero while upgrading
by Alex 20 Mar '25

20 Mar '25

Upgrading from Pacific to Quincy, anyone ever see this? This one looks FUN! I upgraded mgrs, mons, and 1/2 of the osd nodes. Hit this when upgrading the last OSD node. It's a test cluster. Running the command below: ceph orch upgrade start ... --hosts <one_host> --services osd.all-available-devices $ ceph health detail HEALTH_ERR Upgrade: failed due to an unexpected exception [ERR] UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception Unexpected exception occurred during upgrade process: division by zero

1 1

ceph-osd/bluestore using page cache
by Brian Marcotte 19 Mar '25

19 Mar '25

Some years ago when first switching to Bluestore, I could see that ceph-osd wasn't using the host page cache anymore. Some time later after a Ceph upgrade, I found that ceph-osd was now filling the page cache. I'm sorry I don't remember which upgrade that was. Currently I'm running pacific and reef clusters. Should ceph-osd (Bluestore) be going through the page cache? Can ceph-osd be configured to go direct? Thanks. -- - Brian

5 5

Issues with Ceph Cluster Behavior After Migration from ceph-ansible to cephadm and Upgrade to Quincy/Reef
by Jeremi-Ernst Avenant 19 Mar '25

19 Mar '25

Hi all, I recently migrated my Ceph cluster from *ceph-ansible* to *cephadm* (about five months ago) and upgraded from *Pacific 16.2.11* to *Quincy (latest at the time)*, followed by an upgrade to *Reef 18.2.4* two months later - due to running an unsupported version of Ceph. Since this migration and upgrade, I’ve noticed unexpected behavior in the cluster, particularly related to OSD state awareness and balancer efficiency. *1. OSD Nearfull Not Reported Until Restart* I had an OSD exceed its configured nearfull threshold, but *Ceph did not detect or report it* via ceph status. As a result, the cluster entered a degraded state without any warnings. Only after manually restarting the affected OSD did Ceph recognize the nearfull state and update the corresponding pools accordingly. This behavior did not occur in *Pacific/ceph-ansible*—Ceph would previously detect and act on the nearfull condition without requiring a restart. This has been a common recurrence since the migration/upgrade. *2. injectargs Not Taking Effect Until OSD Restart* I've also observed that ceph tell osd.X injectargs --command ... often has no effect. The OSD does not seem to apply the new arguments until it is *manually restarted*, at which point I can modify values via injectargs as expected. However, after a few hours or days, the issue reappears, requiring another restart to modify runtime settings. *3. Ceph Balancer and PG Remapping Issues* The Ceph balancer appears to be operating, but its behavior seems inefficient compared to what we experienced on *Pacific*. It often fails to optimize data distribution effectively, and I have to rely on the *pgremapper* tool to manually intervene. Restarting OSDs seems to improve the balancer’s effectiveness temporarily, suggesting that stale OSD state information may be contributing to the issue. Since this is a *high-performance computing (HPC) environment*, manually restarting OSDs on a regular basis is not a viable solution. These issues did not occur when we were running *Pacific with ceph-ansible*, and I’m wondering if others have experienced similar problems after migrating to *cephadm* and/or upgrading to *Quincy/Reef*. I noticed people on reddit with the same issue, but their resolution was that they "I switch off my whole ceph cluster & switch it back on - to get it working 100% again, - DAILY" Has anyone else encountered these behaviors? Are there any known bugs or workarounds that could help restore expected OSD state tracking and balancer efficiency? Any insights would be greatly appreciated! Thanks, -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

3 3

Is it safe to set multiple OSD out across multiple failure domain?
by Kai Stian Olstad 19 Mar '25

19 Mar '25

Say we have 10 host with 10 OSDs and the failure domain is host. host0 osd 0 to 9 host1 osd 10 to 19 host2 osd 20 to 29 host3 osd 30 to 39 host4 osd 40 to 49 host5 osd 50 to 59 host6 osd 60 to 69 host7 osd 70 to 79 host8 osd 80 to 89 host9 osd 90 to 99 A pool have EC 4+2 and PG 2.1 has UP and ACTING the following OSDs 0,10,20,30,40 and 50. I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out? I would think and hope that Ceph do the right thing and the PG remain accessible since all OSD is available and data is being copied to other OSDs. -- Kai Stian Olstad

6 10

All Github Actions immediately blocked, except GH-official and Ceph-hosted ones
by Ernesto Puerta 18 Mar '25

18 Mar '25

Hi Cephers, Due to the escalating situation in which leaked secrets from a previously compromised GH Action (tj-actions) [1] are used to compromise more popular GH Actions (reviewdog)[2], *we have decided to immediately disable all Github Actions in all repositories, except the official GH ones, and the ones under the Ceph organization*. *From a preliminary analysis, none of our repos has been impacted*. However, given that the Ceph org hosts more that 200 repositories (including forks), it's safer to take a conservative approach while a deeper analysis is undertaken (and the situation settles down). We know that this might have an immediate impact on Ceph teams, as it will break CI using unofficial Github Actions. The recommendation for *each team is to review their Github workflows in their repos* (.github/workflows/*.yaml) and ensure that: - *Version tag pinning is replaced with the corresponding SHA-1 hash*, even for GH official actions. Example <https://github.com/ceph/ceph/blob/8890c0f0df343897195ab97104be0a551d899361/…> . - Once the previous requirement is met, *GH Actions from unofficial sources can be requested* (via mail to security(a)ceph.io) to be included in the allow-list. - Except for actions from the "reviewdog" org <https://github.com/reviewdog>: their team is still analyzing the scope of the attack. In this case, we shouldn't yet trust SHA-1 hashes obtained after the attack until the analysis concludes. - Alternatively, in the long run, we are considering to start forking 3rd party GH Actions under the Ceph org, but carries extra overhead for the Ceph maintainers. We apologize for the inconvenience, but we did this to reduce the exposure of the Ceph community to a series of attacks that may continue to escalate. Kind Regards, Ernesto [1] https://www.stepsecurity.io/blog/harden-runner-detection-tj-actions-changed… [2] https://www.stepsecurity.io/blog/reviewdog-github-actions-are-compromised

1 0

Ceph User + Developer March Meetup happening tomorrow!
by Laura Flores 18 Mar '25

18 Mar '25

Hi all, Join us at the User + Dev meeting tomorrow, where we'll be presenting some highlights from the Ceph User Stories survey and discussing next steps! RSVP here: https://www.meetup.com/ceph-user-group/events/306792345/ Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Remove ... something
by Albert Shih 17 Mar '25

17 Mar '25

Hi, I got two pool one replica 3 and another with EC 4+2 for rbd. During my test to access from a KVM hyperviseur I create a image with meta data in the replica 3 pool and the data in the EC 4+2. After I end the test (where everything works) I can't see anything in erasure42 pool. rbd list --pool erasure42 return a empty list. But root@alopod:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 147 TiB 146 TiB 564 GiB 564 GiB 0.38 TOTAL 147 TiB 146 TiB 564 GiB 564 GiB 0.38 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 23 MiB 7 68 MiB 0 46 TiB erasure42 14 32 336 GiB 86.46k 504 GiB 0.36 92 TiB it still use 336GiB. How can I find where those 336Gib are and delete the image or whatever. Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: lun. 17 mars 2025 08:24:32 CET

2 2

My new osd is not normally ?
by Yunus Emre Sarıpınar 17 Mar '25

17 Mar '25

I have 6 ssd sata and 12 osd per server in a 24 server cluster. This environment was created when it was in the natilus version. I switched this environment to the Octopus version 6 months ago. The cluster is working healthily. I added 8 new servers, I created 6 ssd sata and 12 osd on these servers in the same way. I did not change the number of PGs in the environment, I have 8192 PGs. The problem is that in my ceph -s output, the remapped pg and missplaced object states are gone, but there is a warning of 6nearfull osds and 4pools nearfull. I saw in the ceph df output that my pools are also full above normal. In the output of the ceph osd df tree command, I observed that the occupancy percentages of the newly added osds were around 80%, while the percentages of the old osds were around 30%. How do I equalize this situation? Note: I am sharing the output of crushmap and osd df tree with you in the attachment. My new osds between 288-384. My new servers are ekuark13,14,15,16 and bkuark13,14,15,16.

2 1