February 2025 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

Announcing go-ceph v0.32.0
by Sven Anderson 12 Aug '25

12 Aug '25

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.32.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

Squid 19.2.1 dashboard javascript error
by Chris Palmer 09 Aug '25

09 Aug '25

Firstly thank you so much for the 19.2.1 release. Initial testing suggests that the blockers that we had in 19.2.0 have all been resolved, so we are proceeding with further testing. We have noticed one small problem in 19.2.1 that was not present in 19.2.0 though. We use the older-style dashboard (mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false). The problem happens on the Dashboard screen when health changes to WARN. If you click on WARN you get a small empty dropdown instead of the list of warnings. A javascript error is logged, and using browser inspection there is the additional bit of info that it happens in polyfill: 2025-02-07T15:59:00.970+0000 7f1d63877640 0 [dashboard ERROR frontend.error] (https://<redacted>:8443/#/dashboard): NG0901 Error: NG0901 at d.find (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3342365) at le.ngDoCheck (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3173112) at Qe (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225586) at bt (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225341) at cs (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225051) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259043) at jf (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3266563) at S1 (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259790) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259801) at fg (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3267248) Also, after this happens, no dropdowns work again until the page is forcibly refreshed. Environment is RPM install on Centos 9 Stream. I've created issue [0]. Thanks, Chris [0] https://tracker.ceph.com/issues/69867 <https://tracker.ceph.com/issues/69867?next_issue_id=69865&prev_issue_id=90>

4 9

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

constant increase in osdmap epoch
by Frank Schilder 15 Apr '25

15 Apr '25

Hi all, we observe a problem that has been reported before, but I can't find a resolution. This is related to an earlier thread "failed to load OSD map for epoch 2898146, got 0 bytes" (https://www.spinics.net/lists/ceph-users/msg84485.html) We run an octopus latest cluster and observe a constant increase in osdmap epoch every few seconds. There is no change in the contents between two successive epochs: # diff map.3075085.txt map.3075086.txt 1c1 < epoch 3075085 --- > epoch 3075086 4c4 < modified 2024-11-18T15:38:45.512100+0100 --- > modified 2024-11-18T15:38:47.858092+0100 This is exactly what others reported too, for example, "steady increasing of osd map epoch since octopus" (https://www.spinics.net/lists/ceph-users/msg69443.html) Its a real problem since it dramatically shortens the time window an OSD can be down before its latest OSD map is purged from the cluster. This, in turn, leads to serious follow-up problems with OSD restart as reported in the thread I'm referring to at the beginning. Related to that I also see the mgrs increasing the pgmap version constantly every 2 seconds. However, I believe this is intentional. I don't see this redundant pgp_num_actual setting by the mgrs reported here: https://tracker.ceph.com/issues/51433 . I can't find a resolution anywhere. Any help would be very much appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 7

Ceph Day Silicon Valley 2025 - Call for Proposals Now Open!
by Neha Ojha 14 Apr '25

14 Apr '25

Dear Ceph Community, We are very excited to announce that Ceph Day is returning to Silicon Valley after seven years! On March 25, 2025, we will be hosting Ceph Day at IBM Almaden Research Center in San Jose, CA. We are now accepting talk proposals! For more details, visit https://ceph.io/en/community/events/2025/ceph-days-almaden/. We look forward to bringing the Ceph community together once again! Cheers, Neha

2 2

UI for Object Gateway S3 ?
by Anthony Fecarotta 14 Apr '25

14 Apr '25

Hi everyone, Thanks to a member of this wonderful mailing list, I successfully, and easily deployed the RADOS Gateway S3 API on each of the nodes in one of our five node Proxmox clusters. I can see managing this at scale from the terminal being a little tricky, but more so inconvenient. Are there any options (closed or open-source) that offer a way GUI or TUI to manage the S3 Buckets? I was able to find this GitHub Gist ( https://gist.github.com/e02c6081b515881aeb97b26016d0c658.git) that lets Minio act as a GUI for "Ceph backed S3," but it hasn't been updated in nine months and it seems to be intended strictly for Kubernetes. Any of your recommendations are appreciated. I started building a TUI with GO, to solve this problem, but that will take some time to finish 🙃 Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

3 5

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

ceph.log using seconds since epoch instead of date/time stamp
by Stillwell, Bryan 07 Apr '25

07 Apr '25

Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps? Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok # ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1 This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04. I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not): https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.… Thanks, Bryan

3 4

Subject: Assistance Required: Vault Integration with RADOS Gateway for SSE-S3 Encryption
by Dhivya G 20 Mar '25

20 Mar '25

Dear Ceph Community, We have been researching encryption mechanisms for a while and have successfully deployed Ceph using Kubernetes. Currently, we are working on integrating HashiCorp Vault with RADOS Gateway to enable server-side encryption (SSE-S3) for object uploads. Despite configuring Vault with RADOS Gateway, we are encountering an "INVALID REQUEST" error when attempting to upload objects into an encrypted bucket. Additionally, we are unable to locate any relevant Vault logs that provide insights into this specific error. We would greatly appreciate any insights from those who have faced a similar issue or have successfully configured Vault transit with RADOS Gateway for SSE-S3 encryption. Any guidance or shared experiences would be immensely helpful in resolving this issue. Looking forward to your expert advice. Thanks & Regards, Dhivya G|Associate Software Engineer Ph No: (+91) 9894944910 email: dhivya.g(a)zybisys.com Zybisys Consulting LLP | Srinivasa Tower' No.29, 39th E Cross, 4th T Block, Jayanagar , Bangalore India - 560041http://zybisys.com/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 3

Issues with Ceph Cluster Behavior After Migration from ceph-ansible to cephadm and Upgrade to Quincy/Reef
by Jeremi-Ernst Avenant 19 Mar '25

19 Mar '25

Hi all, I recently migrated my Ceph cluster from *ceph-ansible* to *cephadm* (about five months ago) and upgraded from *Pacific 16.2.11* to *Quincy (latest at the time)*, followed by an upgrade to *Reef 18.2.4* two months later - due to running an unsupported version of Ceph. Since this migration and upgrade, I’ve noticed unexpected behavior in the cluster, particularly related to OSD state awareness and balancer efficiency. *1. OSD Nearfull Not Reported Until Restart* I had an OSD exceed its configured nearfull threshold, but *Ceph did not detect or report it* via ceph status. As a result, the cluster entered a degraded state without any warnings. Only after manually restarting the affected OSD did Ceph recognize the nearfull state and update the corresponding pools accordingly. This behavior did not occur in *Pacific/ceph-ansible*—Ceph would previously detect and act on the nearfull condition without requiring a restart. This has been a common recurrence since the migration/upgrade. *2. injectargs Not Taking Effect Until OSD Restart* I've also observed that ceph tell osd.X injectargs --command ... often has no effect. The OSD does not seem to apply the new arguments until it is *manually restarted*, at which point I can modify values via injectargs as expected. However, after a few hours or days, the issue reappears, requiring another restart to modify runtime settings. *3. Ceph Balancer and PG Remapping Issues* The Ceph balancer appears to be operating, but its behavior seems inefficient compared to what we experienced on *Pacific*. It often fails to optimize data distribution effectively, and I have to rely on the *pgremapper* tool to manually intervene. Restarting OSDs seems to improve the balancer’s effectiveness temporarily, suggesting that stale OSD state information may be contributing to the issue. Since this is a *high-performance computing (HPC) environment*, manually restarting OSDs on a regular basis is not a viable solution. These issues did not occur when we were running *Pacific with ceph-ansible*, and I’m wondering if others have experienced similar problems after migrating to *cephadm* and/or upgrading to *Quincy/Reef*. I noticed people on reddit with the same issue, but their resolution was that they "I switch off my whole ceph cluster & switch it back on - to get it working 100% again, - DAILY" Has anyone else encountered these behaviors? Are there any known bugs or workarounds that could help restore expected OSD state tracking and balancer efficiency? Any insights would be greatly appreciated! Thanks, -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

3 3

RGW multisite metadata sync issue
by Vahideh Alinouri 11 Mar '25

11 Mar '25

Hi guys, My Ceph release is Quincy 17.2.5. I need to change the master zone to decommission the old one and upgrade all zones. I have separated the client traffic and sync traffic in RGWs, meaning there are separate RGW daemons handling the sync process. I encountered an issue when trying to sync one of the zones in the zonegroup. The data sync is proceeding fine, but I have an issue with the metadata sync. It gets stuck behind on a shard. Here is the output from radosgw-admin sync status: metadata sync syncing full sync: 1/64 shards full sync: 135 entries to sync incremental sync: 63/64 shards metadata is behind on 1 shard behind shards: [0] In the RGW log, I see this error: 2024-12-15T21:30:59.641+0000 7f6dff472700 1 beast: 0x7f6d2f1cf730: 172.19.66.112 - s3-cdn-user [15/Dec/2024:21:30:59.641 +0000] "GET /admin/log/?type=data&id=56&marker=00000000000000000000%3A00000000000000204086&extra-info=true&rgwx-zonegroup=7c01d60f-88c6-4192-baf7-d725260bf05d HTTP/1.1" 200 44 - - - latency=0.000000000s 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 I’ve tried the following steps: - Changed the PG number of the metadata pool to force a rebalance, but everything was fine. - Ran metadata sync init and tried to run it again. - Restarted RGW services in both the zone and the master zone. - Created a user in the master zone to ensure metadata sync works, which was successful. - Checked OSD logs but didn’t see any specific errors. - Attempted to list metadata in the pool using rados ls -p s3-cdn-dc07.rgw.meta, but got an empty result. - Compared the code for listing OMAP keys between Quincy and Squid versions; there were no specific changes. I’m looking for any advice or suggestions to resolve this issue.

3 6

Move block.db to new ssd
by Roland Giesler 11 Mar '25

11 Mar '25

I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions. The new 4TB SSD's I have split into volumes with: # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 & osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this: # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami I then take osd4 out of the cluster and stop it. Then I do this: # ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami Thereafter I do this to load the changes: # ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 --path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami After that the configuration change has reverted to what it was before. I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 What am I missing to get these changes to be permanent? regards Roland

6 13

When 18.2.5 will be released?
by wojiaowugen 04 Mar '25

04 Mar '25

Hi, everyone, It's an honor to ask questions here. can I ask when 18.2.5 will be released?

2 1

Replace OSD while cluster is recovering?
by Gustavo Garcia Rondina 04 Mar '25

04 Mar '25

Hello list, We have a Ceph cluster (17.2.6 quincy) with 2 admin nodes and 6 storage nodes, each storage node connected to a JBOD enclosure. Each enclosure houses 28 HDD disks of 18 TB size, totaling 168 OSDs. The pool that houses the majority of the data is erasure-coded (4+2). We have recently had one disk failure, which brought one OSD down: # ceph osd tree | grep down 2 hdd 16.49579 osd.2 down 0 1.00000 This OSD is out of the cluster, but we haven't replaced it physically yet. The problem that we are facing is that the cluster was not in the best shape when this OSD failed. Currently we have the following: ################################################ cluster: id: <redacted> health: HEALTH_ERR 1026 scrub errors Possible data damage: 18 pgs inconsistent 2137 pgs not deep-scrubbed in time 2137 pgs not scrubbed in time services: mon: 5 daemons, quorum xyz-admin1,xyz-admin2,xyz-osd1,xyz-osd2,xyz-osd3 (age 17M) mgr: xyz-admin2.sipadf(active, since 17M), standbys: xyz-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 168 osds: 167 up (since 44h), 167 in (since 6w); 220 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 2137 pgs objects: 448.54M objects, 1.0 PiB usage: 1.6 PiB used, 1.1 PiB / 2.7 PiB avail pgs: 134404830/2676514497 objects misplaced (5.022%) 1902 active+clean 191 active+remapped+backfilling 26 active+remapped+backfill_wait 15 active+clean+inconsistent 2 active+remapped+inconsistent+backfilling 1 active+remapped+inconsistent+backfill_wait io: recovery: 597 MiB/s, 252 objects/s progress: Global Recovery Event (6w) [=========================...] (remaining: 5d) ################################################ I have noticed the number of active+clean increasing (was ~1750 two days ago), and objects misplaced very slowly decreasing. My question is, should I wait until recovery is complete, then repair the 18 damaged pg, and only then replace the disk? My thinking is that replacing the disk will trigger more backfilling which will slow down the recovering even more. Another question, should I disable scrubbing while the recovery is not finalized? Thank you for any insights you may be able to provide! - Gustavo

4 5

[Cephfs] Can't get snapshot under a subvolume
by ceph＠gurkan.in 02 Mar '25

02 Mar '25

Hello all, We're getting a "Operation not permitted" error while trying to create a snapshot on the client. It is somehow related to previously-asked Pacific issue mentioned here: https://www.spinics.net/lists/ceph-users/msg67908.html We are on squid (19.2.1) and the given workaround seems to be only temporary. Anyway, here are some details about the issue: Cephx side: ``` client.mount-update-production key: *** caps: [mds] allow rws fsname=production-cephfs path=/volumes/_nogroup/update-production caps: [mon] allow r fsname=production-cephfs caps: [osd] allow rw tag cephfs data=production-cephfs ``` FS flags: ``` ~> sudo ceph fs lsflags production-cephfs joinable allow_snaps allow_multimds_snaps ``` I got the path via following command: ``` ~> sudo ceph fs subvolume getpath production-cephfs update-production /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f ``` And mounted to client with mentioned key: ``` ~> mount | grep ceph mount-update-production@00000000-0000-0000-0000-000000000000.production-cephfs=/volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f on /opt/aptly/.aptly type ceph (rw,noatime,name=mount-update-production,secret=<hidden>,acl,mon_addr=[multiple mon addresses here]) ``` Everything works flawlessly. Yet, when we want to get a snapshot: ``` /opt/aptly/.aptly/pool/9c/0b ~> sudo mkdir .snap/test mkdir: cannot create directory ‘.snap/test’: Operation not permitted ``` If I mount the whole FS with admin rights and run `setfattr -n ceph.dir.subvolume -v 0` for _all_ directories of following path one by one: /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f, then it is solving the issue. But it does not last; not sure what is triggering it again, but if I try snapshotting anything next day, getting the same "Operation not permitted" error again. Is there something I can try, or am I missing something obvious? Thanks, Gürkan

3 5

Schrödinger's Server
by Tim Holloway 01 Mar '25

01 Mar '25

Ack. Another fine mess. I was trying to clean things up and the process of tossing around OSD's kept getting me reports of slow responses and hanging PG operations. This is Ceph Pacific, by the way. I found a deprecated server that claimed to have an OSD even though it didn't show in either "ceph osd tree" or the dashboard OSD list. I suspect that a lot of the grief came from it attempting to use resources that weren't always seen as resources. I shut down the server's OSD (removed the daemon using ceph orch), then foolishly deleted the server from the inventory without doing a drain first. Now cephadmin hates me (key not found), and there are still an MDS and MON listed as ceph orch ls daemons even after I powered the host off. I cannot do a ceph orch daemon delete because there's no longer an IP address available to the daemon delete, and I cannot clear the cephadmin queue: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'ceph07.internal.mousetech.com' Any suggestions?

3 9

Request for Assistance: OSDS Stability Issues Post-Upgrade to Ceph Quincy 17.2.8
by Aref Akhtari 28 Feb '25

28 Feb '25

Dear Ceph Community, I hope this message finds you well. I am reaching out to seek assistance regarding a stability issue we have encountered after upgrading our Ceph cluster from version Pacific 16.2.3 to Quincy 17.2.8. Following the upgrade, we have observed that several of our Object Storage Daemons (OSDs) are experiencing erratic behavior. These OSDs frequently exhibit a "flapping" condition, where they unexpectedly go down and then come back up. This issue has predominantly affected the recently upgraded OSDs within the cluster. Upon reviewing the logs from the affected OSDs, we encountered the following messages: > 2025-02-03T08:34:09.769+0000 7f0f11390780 -1 > bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 > 2025-02-03T08:38:22.920+0000 7feb9dd44780 -1 > bluestore::NCB::__restore_allocator::No Valid allocation info on disk > (empty file) In an attempt to resolve the issue, we executed the ceph-bluestore-tool fsck and repair commands. Although these commands executed successfully, they did not rectify the problem at hand. Additionally, we have captured the following crash information from the ceph logs: > ceph crash info > 2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039 > { > "assert_condition": "log.t.seq == log.seq_live", > "assert_file": > "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/c > eph-17.2.8/src/os/bluestore/BlueFS.cc", > "assert_func": "uint64_t BlueFS::_log_advance_seq()", > "assert_line": 3029, > "assert_msg": > "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ce > ph-17.2.8/src/os/bluestore/BlueFS.cc: In function 'uint64_t > BlueFS::_log_advance_seq()' thread 7ff983564640 time > 2025-02-03T09:19:08.738781+0000\n/home/jenkins-build/build/workspace/ceph-bu > ild/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/os/bluestore/BlueFS.cc: > 3029: FAILED ceph_assert > (log.t.seq == log.seq_live)\n", > "assert_thread_name": "bstore_kv_sync", > "backtrace": [ > "/lib64/libc.so.6(+0x3e730) [0x7ff9930f5730]", > "/lib64/libc.so.6(+0x8bbdc) [0x7ff993142bdc]", > "raise()", > "abort()", > "(ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x179) [0x55882dfb7fdd]", > "/usr/bin/ceph-osd(+0x36b13e) [0x55882dfb813e]", > "/usr/bin/ceph-osd(+0x9cff3b) [0x55882e61cf3b]", > "(BlueFS::_flush_and_sync_log_jump_D(unsigned long)+0x4e) > [0x55882e6291ee]", > "(BlueFS::_compact_log_async_LD_LNF_D()+0x59b) [0x55882e62e8fb]", > "/usr/bin/ceph-osd(+0x9f2b15) [0x55882e63fb15]", > "(BlueFS::fsync(BlueFS::FileWriter*)+0x1b9) [0x55882e631989]", > "/usr/bin/ceph-osd(+0x9f4889) [0x55882e641889]", > "/usr/bin/ceph-osd(+0xd74cd5) [0x55882e9c1cd5]", > "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x483) > [0x55882eade393]", > "(rocksdb::WritableFileWriter::Sync(bool)+0x120) [0x55882eae0b60]", > "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup > const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned > long)+0x337) [0x55882ea00ab7]", > "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, > rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned > long, bool, unsigned long*, unsigned long, rocksdb > ::PreReleaseCallback*)+0x1935) [0x55882ea07675]", > "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, > rocksdb::WriteBatch*)+0x35) [0x55882ea077c5]", > "(RocksDBStore::submit_common(rocksdb::WriteOptions&, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x83) [0x55882e992593]", > "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x99) > [0x55882e992ee9]", > "(BlueStore::_kv_sync_thread()+0xf64) [0x55882e578e24]", > "/usr/bin/ceph-osd(+0x8afb81) [0x55882e4fcb81]", > "/lib64/libc.so.6(+0x89e92) [0x7ff993140e92]", > "/lib64/libc.so.6(+0x10ef20) [0x7ff9931c5f20]" > ], > "ceph_version": "17.2.8", > "crash_id": > "2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039", > "entity_name": "osd.211", > "os_id": "centos", > "os_name": "CentOS Stream", > "os_version": "9", > "os_version_id": "9", > "process_name": "ceph-osd", > "stack_sig": > "ba90de24e2beba9c6a75249a4cce7c533987ca5127cfba5b835a3456174d6080", > "timestamp": "2025-02-03T09:19:08.749233Z", > "utsname_hostname": "afra-osd18", > "utsname_machine": "x86_64", > "utsname_release": "5.15.0-119-generic", > "utsname_sysname": "Linux", > "utsname_version": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024" > } The above crash logs highlight an assertion failure in the BlueFS component, specifically within the function BlueFS::_log_advance_seq(). Despite our efforts to analyze and resolve the issue, we have reached an impasse. For completeness, we have verified the health of our disks using smartctl, and they have all been deemed healthy. We kindly request guidance from the community on how to address this issue or any recommended steps for deeper diagnostics. We appreciate your support and expertise during this troubleshooting process. Thank you for your attention and assistance. Best regards, Aref Akhtari

2 1

Replace OSD while cluster is recovering?
by grondina＠uchicago.edu 28 Feb '25

28 Feb '25

Hello list, We have a Ceph cluster (v17.2.6 quincy) with 3 admin nodes and 6 storage nodes, each connected to a JBOD enclosure. Each enclosure houses 28 HDD disks of size 18 TB, totaling 168 OSDs. The pool that houses the majority of the data is erasure-coded (4+2). We have recently had one disk failure, which brought one OSD down: # ceph osd tree | grep down 2 hdd 16.49579 osd.2 down 0 1.00000 This OSD is out of the cluster, but we haven't replaced it physically yet. The problem that we are facing is that the cluster was not in the best shape when this OSD failed. Currently we have the following: ################ cluster: id: <redacted> health: HEALTH_ERR 1026 scrub errors Possible data damage: 18 pgs inconsistent 2122 pgs not deep-scrubbed in time 2122 pgs not scrubbed in time services: mon: 5 daemons, quorum xyz-admin1,xyz-admin2,xyz-osd1,xyz-osd2,xyz-osd3 (age 17M) mgr: xyz-admin2.sipadf(active, since 17M), standbys: xyz-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 168 osds: 167 up (since 40h), 167 in (since 6w); 226 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 2122 pgs objects: 448.54M objects, 1.0 PiB usage: 1.6 PiB used, 1.1 PiB / 2.7 PiB avail pgs: 133905796/2676514497 objects misplaced (5.003%) 1880 active+clean 201 active+remapped+backfilling 23 active+remapped+backfill_wait 16 active+clean+inconsistent 1 active+remapped+inconsistent+backfill_wait 1 active+remapped+inconsistent+backfilling io: recovery: 703 MiB/s, 281 objects/s progress: Global Recovery Event (6w) [=========================...] (remaining: 5d) ################ I have noticed the number of active+clean increasing, and objects misplaced very slowly decreasing. My question is, should I wait until recovery is complete, repair the 18 damaged pg, and then replace the disk? My thinking is that replacing the disk will trigger more backfilling which will slow down the recovering even more. Another question, should I disable scrubbing while the recovery is not finalized? Thank you for any insights you may be able to provide! - Gustavo

1 0

Squid: Grafana host-details shows total number of OSDs
by Eugen Block 28 Feb '25

28 Feb '25

Hi, as always, I looked briefly into the tracker but couldn't find anything related (yet). Not sure if this has been reported or not, but after I upgraded a customer cluster to Squid (19.2.0), Grafana was also upgraded to version 9.4.12. And now the host details show all 126 OSDs for each host instead of the number per host. The respecitve query is simply: "expr": "count(sum by (ceph_daemon) (ceph_osd_metadata{cluster=~\"$cluster\", }))", To me it looks like there's something missing after the comma, this query for example shows the actual number of OSDs on that host: count(sum by (ceph_daemon) (ceph_osd_metadata{cluster=~"9bafee50-775c-11eb-9ed6-4c52620b498e",hostname="ses01"})) If this has been already reported, could anyone point me to the according tracker? If not, I'd be happy to create a new one. Thanks! Eugen

2 2

external multipath disk not mounted after power off/on the server
by Iban Cabrillo 27 Feb '25

27 Feb '25

Dear cephers, We have a series of devices that mount several SATA disks via an external cabinet. These disks have 4 paths managed by multipath. (local disks about 20 work perfectly) cephosd23:~ # multipath -ll mpathe (35000c500d88657e3) dm-30 LENOVO-X,ST14000NM004J size=13T features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 2:0:9:0 sdae 65:224 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:21:0 sdap 66:144 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:33:0 sdba 67:64 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 2:0:45:0 sdbl 67:240 active ready running mpathd (35000c500d886ac87) dm-23 LENOVO-X,ST14000NM004J size=13T features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 2:0:2:0 sdx 65:112 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:14:0 sdai 66:32 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:26:0 sdat 66:208 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 2:0:38:0 sdbe 67:128 active ready running mpathc (35000c500d87a5c5f) dm-20 LENOVO-X,ST14000NM004J The disks are accessible from the server, but Ceph has not been able to mount them and they are not accessible from Ceph. ... tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-110 tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-113 tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-112 the directory is empty and of course: /usr/bin/ceph-osd -f --cluster ceph --id 110 --setuser ceph --setgroup ceph 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-110/keyring: (2) No such file or directory 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 AuthRegistry(0x55841ec96940) no keyring found at /var/lib/ceph/osd/ceph-110/keyring, disabling cephx 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-110/keyring: (2) No such file or directory 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 AuthRegistry(0x7ffd668863a0) no keyring found at /var/lib/ceph/osd/ceph-110/keyring, disabling cephx Is there any way to fix this error and mount them again? The data should be intact. Best regards -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

7 10

Re: Free space
by quaglio＠bol.com.br 27 Feb '25

27 Feb '25

1 0

Re: Free space
by quaglio＠bol.com.br 27 Feb '25

27 Feb '25

1 0

Statistics?
by Jan Marek 27 Feb '25

27 Feb '25

Hello, I have a new created ceph cluster, I had some issues with disks, and now I have this 'ceph -s' list: health: HEALTH_WARN Degraded data redundancy: 103798/888 objects degraded (11688.964%), 188 pgs degraded, 192 pgs undersized 24 daemons have recently crashed services: mon: 3 daemons, quorum ceph0,ceph1,ceph2 (age 22m) mgr: ceph1.xzkgcv(active, since 40m), standbys: ceph0.qseeqr mds: 1/1 daemons up, 1 standby osd: 64 osds: 64 up (since 22m), 64 in (since 22m); 198 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 2065 pgs objects: 296 objects, 965 MiB usage: 63 GiB used, 1.4 PiB / 1.4 PiB avail pgs: 103798/888 objects degraded (11688.964%) 3770/888 objects misplaced (424.550%) 1865 active+clean 188 active+recovery_wait+undersized+degraded+remapped 7 active+recovery_wait+remapped 4 active+recovering+undersized+remapped 1 active+remapped+backfill_wait io: client: 11 KiB/s wr, 0 op/s rd, 2 op/s wr recovery: 0 B/s, 198 objects/s Please, pay attention to percentage. It seems, that there is swapped numerator and denominator of fraction, isn't it? Sincerely Jan Marek --. Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

2 2

Discussion on issues encountered while creating osds
by hera sami 26 Feb '25

26 Feb '25

Hi team, glad to connect with you all. I am seeing an issue with ceph deployed using cephadm. I am using the "reef" version of ceph. I had written an ansible script for bringing up ceph as described in the ceph official documentation. My script works fine most of the time but on some vms I am seeing an issue. I am creating a cluster of 3 nodes. The bootstrapping went well and other 2 nodes got added to the cluster. [image: image.png] While my mon and mgr are added successfully, the osd on 2 nodes are not coming up. While the device for osd creation is AVAILABLE as seen from below, when i run the ceph orch add osd command, the ceph orch command hangs indefinitely. [image: image.png] [image: image.png] [image: image.png] I am trying to create osd on sftp-ntp-1 and ceph-non-sftp-1 with device /dev/sdb. Althought the device is AVAILABLE, but when I run the ceph och add osd command, the ceph orch command hangs indefinitely. Neither the osd gets added nor any ceph orch or ceph osd commands accessible. Sometimes ceph orch commands resumes when executed "sudo systemctl restart ceph.target", but no luck on osd creation. Also in these nodes, I dont see the logs of Reconfig daemon osd in /var/log/ceph/cephadm.log unlike other daemons like mon, mgr root@sftp-ntp-1:/var/log# cat /var/log/ceph/cephadm.log | grep "Reconfig daemon" 2025-02-27 06:49:09,426 7f37b62ce1c0 INFO Reconfig daemon mgr.sftp-ntp-1.hkvpch ... 2025-02-27 06:49:09,807 7fbbcc9d21c0 INFO Reconfig daemon mon.sftp-ntp-1 ... 2025-02-27 06:49:10,203 7f9cd5a691c0 INFO Reconfig daemon crash.sftp-ntp-1 ... 2025-02-27 06:49:10,581 7fbe2e4e11c0 INFO Reconfig daemon ceph-exporter.sftp-ntp-1 ... This is seen on only few vms and I am not able to understand what is the root cause behind the issue. It would be much appreciated if I can get any sort of idea on this. Thanks, Hera

1 0

Looking for 'rados df' output command explanation for some columns
by jbareapa＠redhat.com 26 Feb '25

26 Feb '25

Hello folks I'm trying to understand some columns of 'rados df' output command, and i appreciate some feedback around. Sorry if was posted before. This cluster is pretty little in HW specs # rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR .mgr 1.8 MiB 2 0 6 0 0 0 526 469 KiB 715 12 MiB 0 B 0 B cephfs1.data 32 KiB 4 0 8 0 0 0 21 20 KiB 47 13 KiB 0 B 0 B cephfs1.metadata 902 MiB 190 0 380 0 0 0 6941 5.3 GiB 909148 2.5 GiB 0 B 0 B cephfs2.data 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B cephfs2.metadata 151 KiB 22 0 66 0 0 0 40 40 KiB 47 13 KiB 0 B 0 B firstpoolrbd 30 KiB 5 0 15 0 0 0 3561 2.8 MiB 58 27 KiB 0 B 0 B I have been 'punishing' on purpose 'cephfs1.metadata' on some clients creating and deleting files/dirs. My question are for the columns below , the values showed are 'accumulatve' stats figures, right? I mean, from what i did, that makes sense. It cannot be values per second, right? Is there a way to reset such columns values? RD_OPS Total number of read operations requested for this pool. RD Total number of bytes read from this pool. WR_OPS Total number of write operations requested for this pool. WR Total number of bytes written to the pool. Note that , writing the same object many times will result is that the pool's usage will remain the same but the number of bytes written to the pool will grow. Thanks in advance for any comment

1 0

backfill_toofull not clearing on Reef
by Deep Dish 26 Feb '25

26 Feb '25

Hello, I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was replaced (same capacity). For the past week or so the cluster has been recovering, slowly, and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity. I tried setting the following without any result: ceph osd set-backfillfull-ratio 0.90 # ceph -s ... health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull 11 pgs not deep-scrubbed in time ... data: volumes: 1/1 healthy pools: 13 pools, 1665 pgs objects: 27.81M objects, 79 TiB usage: 197 TiB used, 413 TiB / 610 TiB avail pgs: 16341449/128092605 objects misplaced (12.758%) 1371 active+clean 150 active+remapped+backfill_wait 121 active+remapped+backfill_wait+backfill_toofull 16 active+remapped+backfill_toofull 6 active+clean+scrubbing+deep 1 active+remapped+backfilling io: client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr recovery: 16 MiB/s, 4 objects/s # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up 9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up 16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up 27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up 35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6 TiB 35.51 1.10 113 up 43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5 TiB 37.92 1.17 106 up 51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8 TiB 33.63 1.04 108 up 59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9 TiB 32.70 1.01 101 up 67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9 TiB 31.99 0.99 112 up 74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2 TiB 28.68 0.89 110 up 3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5 TiB 28.45 0.88 112 up 11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1 TiB 21.50 0.67 115 up 19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6 TiB 36.51 1.13 115 up 25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2 TiB 29.08 0.90 91 up 33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2 TiB 27.27 0.84 109 up 41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5 TiB 24.36 0.75 105 up 49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1 TiB 29.98 0.93 107 up 56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0 TiB 31.77 0.98 116 up 62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5 TiB 38.47 1.19 108 up 69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1 TiB 30.05 0.93 109 up 5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2 TiB 31.40 0.97 109 up 13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3 TiB 31.12 0.96 116 up 22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8 TiB 34.72 1.07 114 up 31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0 TiB 31.08 0.96 101 up 39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7 TiB 35.60 1.10 110 up 47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0 TiB 31.89 0.99 108 up 55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4 TiB 39.43 1.22 116 up 65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9 TiB 32.62 1.01 116 up 73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6 TiB 36.13 1.12 111 up 78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7 TiB 35.50 1.10 118 up 4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6 TiB 27.11 0.84 131 up 12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9 TiB 23.80 0.74 106 up 20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4 TiB 26.11 0.81 106 up 28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5 TiB 38.59 1.19 121 up 36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9 TiB 31.98 0.99 108 up 44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8 TiB 33.68 1.04 90 up 52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1 TiB 29.76 0.92 108 up 60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0 TiB 31.04 0.96 104 up 68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8 TiB 33.83 1.05 119 up 76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7 TiB 35.77 1.11 107 up 6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1 TiB 33.01 1.02 111 up 14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3 TiB 30.96 0.96 84 up 23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8 TiB 33.96 1.05 95 up 30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5 TiB 38.67 1.20 93 up 38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0 TiB 31.72 0.98 101 up 46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7 TiB 36.05 1.12 81 up 53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6 TiB 36.12 1.12 88 up 63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2 TiB 42.94 1.33 87 up 75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up 79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6 TiB 36.90 1.14 93 up 7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1 TiB 22.22 0.69 98 up 15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5 TiB 28.50 0.88 114 up 21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6 TiB 22.60 0.70 88 up 29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8 TiB 34.20 1.06 114 up 37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9 TiB 33.02 1.02 128 up 45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6 TiB 36.26 1.12 109 up 54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7 TiB 35.04 1.08 111 up 61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2 TiB 27.87 0.86 103 up 70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0 TiB 31.86 0.99 109 up 77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8 TiB 32.35 1.00 112 up 2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2 TiB 32.27 1.00 115 up 10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9 TiB 24.28 0.75 97 up 18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3 TiB 26.63 0.82 107 up 26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6 TiB 35.60 1.10 111 up 34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6 TiB 36.17 1.12 106 up 42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1 TiB 30.17 0.93 111 up 50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6 TiB 36.05 1.12 101 up 58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4 TiB 39.11 1.21 122 up 66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3 TiB 26.73 0.83 98 up 72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0 TiB 29.57 0.92 110 up 0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3 TiB 30.63 0.95 119 up 8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7 TiB 26.77 0.83 112 up 17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5 TiB 38.79 1.20 107 up 24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9 TiB 32.22 1.00 116 up 32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7 TiB 35.56 1.10 94 up 40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3 TiB 39.44 1.22 106 up 48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9 TiB 32.22 1.00 114 up 57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2 TiB 28.40 0.88 95 up 64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6 TiB 37.42 1.16 130 up 71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9 TiB 32.01 0.99 105 up TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413 TiB 32.31 MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 86.15359 root default -19 10.00000 host 01stor28 1 hdd 1.00000 osd.1 up 1.00000 1.00000 9 hdd 1.00000 osd.9 up 1.00000 1.00000 16 hdd 1.00000 osd.16 up 1.00000 1.00000 27 hdd 1.00000 osd.27 up 1.00000 1.00000 35 hdd 1.00000 osd.35 up 1.00000 1.00000 43 hdd 1.00000 osd.43 up 1.00000 1.00000 51 hdd 1.00000 osd.51 up 1.00000 1.00000 59 hdd 1.00000 osd.59 up 1.00000 1.00000 67 hdd 1.00000 osd.67 up 1.00000 1.00000 74 hdd 1.00000 osd.74 up 1.00000 1.00000 -21 10.00000 host 01stor32 3 hdd 1.00000 osd.3 up 1.00000 1.00000 11 hdd 1.00000 osd.11 up 1.00000 1.00000 19 hdd 1.00000 osd.19 up 1.00000 1.00000 25 hdd 1.00000 osd.25 up 1.00000 1.00000 33 hdd 1.00000 osd.33 up 1.00000 1.00000 41 hdd 1.00000 osd.41 up 1.00000 1.00000 49 hdd 1.00000 osd.49 up 1.00000 1.00000 56 hdd 1.00000 osd.56 up 1.00000 1.00000 62 hdd 1.00000 osd.62 up 1.00000 1.00000 69 hdd 1.00000 osd.69 up 1.00000 1.00000 -15 10.00000 host 02stor23 5 hdd 1.00000 osd.5 up 1.00000 1.00000 13 hdd 1.00000 osd.13 up 1.00000 1.00000 22 hdd 1.00000 osd.22 up 1.00000 1.00000 31 hdd 1.00000 osd.31 up 1.00000 1.00000 39 hdd 1.00000 osd.39 up 1.00000 1.00000 47 hdd 1.00000 osd.47 up 1.00000 1.00000 55 hdd 1.00000 osd.55 up 1.00000 1.00000 65 hdd 1.00000 osd.65 up 1.00000 1.00000 73 hdd 1.00000 osd.73 up 1.00000 1.00000 78 hdd 1.00000 osd.78 up 1.00000 1.00000 -11 10.00000 host 02stor27 4 hdd 1.00000 osd.4 up 1.00000 1.00000 12 hdd 1.00000 osd.12 up 1.00000 1.00000 20 hdd 1.00000 osd.20 up 1.00000 1.00000 28 hdd 1.00000 osd.28 up 1.00000 1.00000 36 hdd 1.00000 osd.36 up 1.00000 1.00000 44 hdd 1.00000 osd.44 up 1.00000 1.00000 52 hdd 1.00000 osd.52 up 1.00000 1.00000 60 hdd 1.00000 osd.60 up 1.00000 1.00000 68 hdd 1.00000 osd.68 up 1.00000 1.00000 76 hdd 1.00000 osd.76 up 1.00000 1.00000 -17 16.15359 host 03stor23 6 hdd 1.00000 osd.6 up 1.00000 1.00000 14 hdd 1.00000 osd.14 up 1.00000 1.00000 23 hdd 1.00000 osd.23 up 1.00000 1.00000 30 hdd 1.00000 osd.30 up 1.00000 1.00000 38 hdd 1.00000 osd.38 up 1.00000 1.00000 46 hdd 1.00000 osd.46 up 1.00000 1.00000 53 hdd 1.00000 osd.53 up 1.00000 1.00000 63 hdd 1.00000 osd.63 up 1.00000 1.00000 75 hdd 7.15359 osd.75 up 1.00000 1.00000 79 hdd 1.00000 osd.79 up 1.00000 1.00000 -13 10.00000 host 03stor27 7 hdd 1.00000 osd.7 up 1.00000 1.00000 15 hdd 1.00000 osd.15 up 1.00000 1.00000 21 hdd 1.00000 osd.21 up 1.00000 1.00000 29 hdd 1.00000 osd.29 up 1.00000 1.00000 37 hdd 1.00000 osd.37 up 1.00000 1.00000 45 hdd 1.00000 osd.45 up 1.00000 1.00000 54 hdd 1.00000 osd.54 up 1.00000 1.00000 61 hdd 1.00000 osd.61 up 1.00000 1.00000 70 hdd 1.00000 osd.70 up 1.00000 1.00000 77 hdd 1.00000 osd.77 up 1.00000 1.00000 -3 10.00000 host 04stor28 2 hdd 1.00000 osd.2 up 1.00000 1.00000 10 hdd 1.00000 osd.10 up 1.00000 1.00000 18 hdd 1.00000 osd.18 up 1.00000 1.00000 26 hdd 1.00000 osd.26 up 1.00000 1.00000 34 hdd 1.00000 osd.34 up 1.00000 1.00000 42 hdd 1.00000 osd.42 up 1.00000 1.00000 50 hdd 1.00000 osd.50 up 1.00000 1.00000 58 hdd 1.00000 osd.58 up 1.00000 1.00000 66 hdd 1.00000 osd.66 up 1.00000 1.00000 72 hdd 1.00000 osd.72 up 1.00000 1.00000 -7 10.00000 host 04stor32 0 hdd 1.00000 osd.0 up 1.00000 1.00000 8 hdd 1.00000 osd.8 up 1.00000 1.00000 17 hdd 1.00000 osd.17 up 1.00000 1.00000 24 hdd 1.00000 osd.24 up 1.00000 1.00000 32 hdd 1.00000 osd.32 up 1.00000 1.00000 40 hdd 1.00000 osd.40 up 1.00000 1.00000 48 hdd 1.00000 osd.48 up 1.00000 1.00000 57 hdd 1.00000 osd.57 up 1.00000 1.00000 64 hdd 1.00000 osd.64 up 1.00000 1.00000 71 hdd 1.00000 osd.71 up 1.00000 1.00000 # ceph health detail HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting [0,67,4] pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting [71,44,22] pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting [58,51,23] pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting [69,45,22] pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting [24,3,63] pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting [35,50,14] pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting [28,16,69] pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6] pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting [78,26,29] pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30] pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting [43,11,17] pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting [68,33,2] pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66] pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting [78,17,6] pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting [57,67,13] pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38] pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting [1,0,28] pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting [47,14,66] pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14] pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting [27,45,38] pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72] pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting [52,69,46] pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting [55,26,32] pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting [34,59,11] pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2] pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting [55,70,46] pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23] pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting [37,25,59] pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23] pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting [11,26,24] pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting [59,60,46] pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6] pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting [35,57,46] pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting [28,9,72] pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting [5,43,72] pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting [70,72,14] pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting [78,10,23] pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting [66,7,53] pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13] pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting [25,15,63] pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting [78,76,34] pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting [24,76,54,16,25,79,22,2] pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting [10,9,56,4,47,61,32,79] pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting [70,41,36,48,23,31,1,2] pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting [10,29,78,25,1,44,64,79] pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting [28,26,15,23,57,16,25,78] pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting [59,46,19,7,72,68,55,8] pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting [76,54,25,31,0,46,58,59] pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting [31,76,1,66,8,56,79,21] pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting [74,0,70,44,39,34,33,6] pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting [44,21,43,18,69,64,38,31] [WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000 pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000 pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000 pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000 pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000 pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000 pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000 pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000 pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000 pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000 pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000 pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000 pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000 pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000 pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000 pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000 pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000 pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000 pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000 pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000 pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000 pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000 pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000 pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000 pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000

7 9

Problem with S3 presigned URLs & CORS & Object tagging
by Haarländer, Markus 26 Feb '25

26 Feb '25

Hi list, We encountered a problem with presigned URLs for putting objects in connection with CORS and S3 Object tagging in Ceph v17.2.7 It works fine with v 16.2.15 and it works fine if the tagging is disabled. Here are the steps to reproduce: 1. Create a CORS rule for a bucket called "my-bucket": { "CORSRules": [ { "AllowedOrigins": ["*"], "AllowedHeaders": ["*"], "AllowedMethods": ["PUT", "GET"], "ExposeHeaders": ["ETag", "Accept-Ranges", "Content-Encoding", "Content-Range"] } ] } 2. Create a presigned URL to upload data to the key "test.txt". A tag called "test" should be applied to the resulting object. The following presigned url is returned eg. by the Java SDK. It automatically contains the "x-amz-tagging" in the X-Amz-Signed-Headers parameter. https://my-bucket.my-s3-server/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-… 3. Simulate a preflight OPTIONS request with Origin and Access-Control-Request-Method headers, as the browser would do when trying to PUT to the presigned URL curl --request OPTIONS 'https://my-bucket.my-s3-server/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-…' \ --header 'Origin: https://example.org' \ --header 'Access-Control-Request-Method: PUT' The server replies with 403, no CORS headers and the following body: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000006b5686df4e08cfcd-0067bdc7f3-218e3740-default</RequestId><HostId>218e3740-default-default</HostId></Error> Expectation: The server should reply with 200 and with CORS headers Additional observations: - The exact same workflow works on Ceph v16.2.15, with tagging - It works with Ceph v17.2.7 if we do not use the tagging (then no x-amz-tagging is returned in the X-Amz-Signed-Headers parameter) - It works with Ceph v17.2.7 for non-CORS environments (then no OPTIONS request is done, the PUT request itself works with tagging) - It works with Ceph v17.2.7 if we add the "x-amz-tagging" header to the OPTIONS request. But this does not lead anywhere, as for CORS, every browser creates the OPTIONS request itself for a preflight request and removes all custom headers. There's no possibility to change that. Any ideas or hints are very welcome. Thank you. Markus

2 2

S3 Bucket Upload - Boto3 - Disable or Enable Checksum on bucket/rgw
by Devender Singh 25 Feb '25

25 Feb '25

Hello Using boto3 trying to upload files to s3 bucket, when I am using boto version 1.35 it works fine, when upgrading it to 1.36 doesn’t work, facing below error, python3 s3_upload.py Error testing the new bucket: An error occurred (XAmzContentSHA256Mismatch) when calling the PutObject operation: None But works with AWS s3 Bucket. “https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/“ Is there a way to enable/disable checksum ? Regards Dev

3 2

is upgrade from quincy to squid supported?
by Simon Oosthoek 25 Feb '25

25 Feb '25

Hi list, we have a ceph cluster running ubuntu 22.04 and quincy packages from ubuntu. To keep up with the releases of Ceph, we want to upgrade to a more recent stable release... We could upgrade to ubuntu 24.04 LTS (re-install actually) but the ceph version then skips 18.2.x and goes directly to 19.2.x, if this is a supported upgrade, that would be the least amount of work, since upgrade of the OS will be needed at some point anyway. We are not planning a move to cephadm yet. I suppose if we need to install 18 before 19, we'd need to add the ceph repo for ubuntu package to upgrade and then upgrade to ubuntu 24.04's ceph 19.2.x after reinstalling all the ceph servers... Has anyone successfully upgraded from 17.2.x to 19.2.x in one upgrade step? (And is this supported?) Cheers /Simon -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

2 1

Monitors crash largely due to the structure of pg-upmap-primary
by Michal Strnad 25 Feb '25

25 Feb '25

Hi. We ran into an issue with "pg-upmap-primary", which resulted in our monitors crashing massively (around 1,000 crashes per day). According to [1], it should be possible to remove these pg-upmap-primaries. Unfortunately, we are unable to do so because these PGs, and therefore the pool, no longer exist. root(a)app001.clX ~ # ceph osd dump | grep 'pg_upmap_primary' | grep 24.ffc pg_upmap_primary 24.ffc 232 root(a)app001.clX ~ # ceph osd rm-pg-upmap-primary 24.ffc Error ENOENT: pgid '24.ffc' does not exist root(a)app001.clX ~ # ceph pg dump | grep "^24\." dumped all Is there any way to remove these structures? We also tried upgrading from the current version 18.2.1 to 18.2.4, but this led to a state on our three-node test cluster where one of the three monitors failed to start, along with a third of the OSDs, due to issues with the mentioned structure. Restarting the daemon didn’t help. Does anyone have a solution or an idea? This is becoming quite a problem for us. Below, I am attaching one of the many monitor crash logs. Thank you very much for any advice! Of course, we have also created a ticket in the tracker [https://tracker.ceph.com/issues/69760] where the same information I’m sending in this email is documented. Michal [1] https://tracker.ceph.com/issues/61948#note-32 { "assert_condition": "pg_upmap_primaries.empty()", "assert_file": "/builddir/build/BUILD/ceph-18.2.1/src/osd/OSDMap.cc",https://tracker.ceph.com/issues/69760 "assert_func": "void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const", "assert_line": 3239, "assert_msg": "/builddir/build/BUILD/ceph-18.2.1/src/osd/OSDMap.cc: In function 'void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const' thread 7f94216a3640 time 2025-02-02T19:16:03.629964+0100\n/builddir/build/BUILD/ceph-18.2.1/src/osd/OSDMap.cc: 3239: FAILED ceph_assert(pg_upmap_primaries.empty())\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib64/libc.so.6(+0x54db0) [0x7f9429054db0]", "/lib64/libc.so.6(+0xa365c) [0x7f94290a365c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f9429d630df]", "/usr/lib64/ceph/libceph-common.so.2(+0x163243) [0x7f9429d63243]", "/usr/lib64/ceph/libceph-common.so.2(+0x1a0f38) [0x7f9429da0f38]", "(OSDMonitor::reencode_full_map(ceph::buffer::v15_2_0::list&, unsigned long)+0xe2) [0x55ca54957e22]", "(OSDMonitor::get_version_full(unsigned long, unsigned long, ceph::buffer::v15_2_0::list&)+0x1de) [0x55ca549596ae]", "(OSDMonitor::build_latest_full(unsigned long)+0x2a3) [0x55ca549599a3]", "(OSDMonitor::check_osdmap_sub(Subscription*)+0xc8) [0x55ca5495be98]", "(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xf04) [0x55ca54834dd4]", "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x6a6) [0x55ca548359c6]", "(Monitor::_ms_dispatch(Message*)+0x779) [0x55ca54836d59]", "/usr/bin/ceph-mon(+0x2f3dfe) [0x55ca547f1dfe]", "(DispatchQueue::entry()+0x52a) [0x7f9429f5766a]", "/usr/lib64/ceph/libceph-common.so.2(+0x3e7321) [0x7f9429fe7321]", "/lib64/libc.so.6(+0xa1912) [0x7f94290a1912]", "/lib64/libc.so.6(+0x3f450) [0x7f942903f450]" ], "ceph_version": "18.2.1", "crash_id": "2025-02-02T18:16:03.632571Z_f5516ed0-6df5-4267-bada-71f5d8d764ba", "entity_name": "mon.mon001-clX", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mon", "stack_sig": "772ef523b041edc5147d1d9905926fb794d32b2635368a8199f6e2e4f2d688bf", "timestamp": "2025-02-02T18:16:03.632571Z", "utsname_hostname": "app001.clX", "utsname_machine": "x86_64", "utsname_release": "5.14.0-402.el9.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Dec 21 19:46:35 UTC 2023" }

3 4

CSC Meeting Minutes | 2025-02-24
by Afreen Misbah 25 Feb '25

25 Feb '25

Hi all, Here are the meeting minutes from the 2025-02-24 call. Joseph shared LFX insights on ceph! - Link to the insights <https://insights.lfx.linuxfoundation.org/foundation/cephfoundation/overview…> QE updates (Yuri) - 6 outstanding PRs <https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Av18.2.5> - Eric is going to take a look at them and provide inputs - can start release process as soon as the PRs merge Release scheduling (Greg) Discussions from ongoing mail thread Ceph Tentacle release timeline — when? <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PNJB3CW43NF…> - People want fast EC - EC work in upstream making progress - Move the technical discussions to mailing list - distro sync is important for a lot of operators due to tooling available-by-default, even if using Ceph upstream packages - containers vs packages: Packages are stable enough - tentacle when fast EC ready, and freeze "U" in January for an April release next year? - Do we need/want to extend our major-version upgrade commitments for the shorter time frames? - Ceph does not give release for an entire year - Need to prepare a plan and see what resources and costs will be needed - Better to go for stability rather than quickness Tentacle Kickoff PR (Patrick) - https://github.com/ceph/ceph/pull/61321 - QA: https://tracker.ceph.com/issues/70000 - Merge soon to avoid rebases Ceph build PR (Patrick) - https://github.com/ceph/ceph-build/pull/2315 - Blocked on Dan Mick ,probably take a look soon Ceph Labs presentation (David) - During Ceph Infra call 2/27 12:30 Eastern - Feel free to join -- Afreen -- Afreen

2 1

[How to mount cephFS on a k8s pod?]
by Baijia Ye 25 Feb '25

25 Feb '25

Hello all, Is it possible to mount cephFS on a k8s pod? I tried but failed. [image: image.png] Thanks for your help! Best, Baijia

1 0

Cephalocon 2025 Sponsorships - Early interest survey
by Dan van der Ster 24 Feb '25

24 Feb '25

Hi all, After a fantastic Cephalocon at CERN last year, and based on the early results of the Ceph Events Survey (thanks to those who responded!) we are aiming to hold the next Cephalocon in North America in the fall of 2025 (Sep/Oct). We need to capture informal (not contractual) commitments from potential sponsors early so we can move ahead with planning the event. Levels will be similar to 2024's prospectus [1], with a possible increase in cost. However, for anyone who responds to this early commitment, we plan to hold the 3 major sponsorships (Platinum, Gold, and Silver) to 2024 pricing! We really appreciate your support in sending a quick response (by EOD Thursday, Feb 27th): https://docs.google.com/forms/d/e/1FAIpQLSfbC1nDP5qlBYMwzEuctsGdCa60mkeVIHc… [1] https://events.linuxfoundation.org/wp-content/uploads/2024/10/Sponsor-Ceph-… -- Dan van der Ster Ceph Executive Council

1 0

Why is the default include_parent value of `export-diff` True , and is it not allowed for users to set it?
by Zacharias Turing 24 Feb '25

24 Feb '25

Hi， Why not allow users to specify include_parent=False when doing export-diff? Wouldn't this save capacity during migration?

1 0

cephfs-mirror and acl
by ronny.lippold 24 Feb '25

24 Feb '25

hi, hope, somebody can help me,i did not found any ideas in the internet. we plan to migrate our fileserver storage to cephfs. our customer uses acl. for standby, it would be great to have a cold standby ... thats why cephfs-mirror. after a lot of testing, i got cephfs-mirror working. but now, is see ... there where no acl synced. i see the acl in the snapshot, but not on the other side. is somebody using cephfs-mirror, also with acl? do you have any other ideas for cold standby (rsync, lsyncd)? many thanks for help and regards, ronny

1 0

ASSISTANCE REQUEST: OSDs Stability Issues Post-Upgrade
by Nima AbolhassanBeigi 23 Feb '25

23 Feb '25

Dear Ceph Community, I hope this message finds you well. I am reaching out to seek assistance regarding a stability issue we have encountered after upgrading our Ceph cluster from version Pacific 16.2.3 to Quincy 17.2.8. Following the upgrade, we have observed that several of our Object Storage Daemons (OSDs) are experiencing erratic behavior. These OSDs frequently exhibit a "flapping" condition, where they unexpectedly go down and then come back up. This issue has predominantly affected the recently upgraded OSDs within the cluster. Upon reviewing the logs from the affected OSDs, we encountered the following messages: 2025-02-03T08:34:09.769+0000 7f0f11390780 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-02-03T08:38:22.920+0000 7feb9dd44780 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) In an attempt to resolve the issue, we executed the ceph-bluestore-tool fsck and repair commands. Although these commands executed successfully, they did not rectify the problem at hand. Additionally, we have captured the following crash information from the ceph logs: ceph crash info 2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039 { "assert_condition": "log.t.seq == log.seq_live", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/c eph-17.2.8/src/os/bluestore/BlueFS.cc", "assert_func": "uint64_t BlueFS::_log_advance_seq()", "assert_line": 3029, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ce ph-17.2.8/src/os/bluestore/BlueFS.cc: In function 'uint64_t BlueFS::_log_advance_seq()' thread 7ff983564640 time 2025-02-03T09:19:08.738781+0000\n/home/jenkins-build/build/workspace/ceph-bu ild/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/os/bluestore/BlueFS.cc: 3029: FAILED ceph_assert (log.t.seq == log.seq_live)\n", "assert_thread_name": "bstore_kv_sync", "backtrace": [ "/lib64/libc.so.6(+0x3e730) [0x7ff9930f5730]", "/lib64/libc.so.6(+0x8bbdc) [0x7ff993142bdc]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55882dfb7fdd]", "/usr/bin/ceph-osd(+0x36b13e) [0x55882dfb813e]", "/usr/bin/ceph-osd(+0x9cff3b) [0x55882e61cf3b]", "(BlueFS::_flush_and_sync_log_jump_D(unsigned long)+0x4e) [0x55882e6291ee]", "(BlueFS::_compact_log_async_LD_LNF_D()+0x59b) [0x55882e62e8fb]", "/usr/bin/ceph-osd(+0x9f2b15) [0x55882e63fb15]", "(BlueFS::fsync(BlueFS::FileWriter*)+0x1b9) [0x55882e631989]", "/usr/bin/ceph-osd(+0x9f4889) [0x55882e641889]", "/usr/bin/ceph-osd(+0xd74cd5) [0x55882e9c1cd5]", "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x483) [0x55882eade393]", "(rocksdb::WritableFileWriter::Sync(bool)+0x120) [0x55882eae0b60]", "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x337) [0x55882ea00ab7]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb ::PreReleaseCallback*)+0x1935) [0x55882ea07675]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x35) [0x55882ea077c5]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x83) [0x55882e992593]", "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x99) [0x55882e992ee9]", "(BlueStore::_kv_sync_thread()+0xf64) [0x55882e578e24]", "/usr/bin/ceph-osd(+0x8afb81) [0x55882e4fcb81]", "/lib64/libc.so.6(+0x89e92) [0x7ff993140e92]", "/lib64/libc.so.6(+0x10ef20) [0x7ff9931c5f20]" ], "ceph_version": "17.2.8", "crash_id": "2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039", "entity_name": "osd.211", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-osd", "stack_sig": "ba90de24e2beba9c6a75249a4cce7c533987ca5127cfba5b835a3456174d6080", "timestamp": "2025-02-03T09:19:08.749233Z", "utsname_hostname": "afra-osd18", "utsname_machine": "x86_64", "utsname_release": "5.15.0-119-generic", "utsname_sysname": "Linux", "utsname_version": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024" } The above crash logs highlight an assertion failure in the BlueFS component, specifically within the function BlueFS::_log_advance_seq(). Despite our efforts to analyze and resolve the issue, we have reached an impasse. For completeness, we have verified the health of our disks using smartctl, and they have all been deemed healthy. We kindly request guidance from the community on how to address this issue or any recommended steps for deeper diagnostics. We appreciate your support and expertise during this troubleshooting process. OSD logs: https://paste.mozilla.org/6STm6eum Thank you for your attention and assistance. Best regards Nima

2 2

User new to ceph
by Christian Hansen 23 Feb '25

23 Feb '25

So i have battled the last days to reclaim my disk. Basically long story short i installed proxmox cluster with a ceph cluster. I reinstalled the proxmox node and didnt remove the osd as you should trough the gui found out later, i tried a bunch of commands like zap the disk wipe the disk and so on. But whatever i do i cannot wipe the ceph disk and i cant seem to wipe the disk so i can reuse the disk. How do you go about wiping a ceph disk when you reinstalled the original OS that had the disk as the OSD. i tried DBAN, i tried ShredOS i tried Dells wipe disk feature but it says its looked. i tried hirens boot and wiping the disk i tried in diskpart in windows to wipe the disk all things fail and i dont get it. Is there no way to reclaim my disk? Thanks in advance. Best regards, Christian

4 3

ceph iscsi gateway
by Iban Cabrillo 21 Feb '25

21 Feb '25

Good morning, I wanted to inquire about the status of the Ceph iSCSI gateway service. We currently have several machines installed with this technology that are working correctly, although I have seen that it appears to be discontinued since 2022. My question is whether to continue down this path, adding machines, or whether it will be discontinued in new Ceph distributions. Best regards and thanks in advance. -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

10 11

understanding Ceph OSD Interaction with the Linux Kernel
by Lina SADI 21 Feb '25

21 Feb '25

Dear Ceph Community, I hope you are doing well. I am trying to understand how Ceph OSD interacts with the Linux kernel. The Linux I/O stack consists of layers such as VFS, filesystem, block layer, and device driver, but since Ceph does not use a traditional filesystem and instead relies on BlueStore as its storage backend, I am struggling to find details on how it interacts with the kernel. Specifically, I would like to understand: - What are the layers in the Linux I/O stack that a Ceph OSD request passes through? - Does BlueStore use submit_bio() to enter the block layer, or does it bypass it and interact directly with the device driver? - Where can I find resources or documentation that explain this in detail? Any insights, documentation links, or pointers to relevant parts of the Ceph source code would be greatly appreciated. Thank you in advance for your help! Best regards.

2 1

rgw gateways query via /api/rgw/daemon
by Szabo, Istvan (Agoda) 21 Feb '25

21 Feb '25

Hi, I'd like to query the gateways via the rgw daemon api, but I got this error: [src/ceph_client.rs:97:9] s.clone() = "{\"status\": \"500 Internal Server Error\", \"detail\": \"The server encountered an unexpected condition which prevented it from fulfilling the request.\", \"request_id\": \"089f9e0a-3bce-4902-91ee-cdf9a4f57839\"} " [src/ceph_client.rs:193:13] l = Err( "expected array value", ) Checked couple of threads pointing to the rgw_dns_name setting in ceph.conf (didn't work) and also in the mailist there is a thread but without solution. Also went through this: https://docs.ceph.com/en/quincy/mgr/dashboard/#enabling-the-object-gateway-… but didn't help. On the dashboard I set our certificates, in the ceph logs I can see these: 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.186, port=8085, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.187, port=8081, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.187, port=8082, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.187, port=8085, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.187, port=8080, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.199.84, port=8080, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8082, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8085, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8081, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8080, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8083, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 258, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 190, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 108, in list 'port': int(metadata['frontend_config#0'].split('port=')[1]) IndexError: list index out of range 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard ERROR request] [10.105.176.214:49981] [GET] [500] [0.011s] [admin] [513.0B] /api/rgw/daemon 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "1739b2ab-5450-4c16-b934-82bf67198cbc"} '] 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO request] [10.105.176.214:49981] [GET] [500] [0.011s] [admin] [513.0B] /api/rgw/daemon 2025-02-21T09:51:49.474+0700 7f530ddce640 0 [dashboard INFO request] [10.105.176.214:49983] [GET] [200] [0.010s] [admin] [79.0B] /api/feature_toggles 2025-02-21T09:51:49.478+0700 7f530e5cf640 0 [dashboard INFO request] [10.105.176.214:49968] [GET] [200] [0.013s] [admin] [1.6K] /api/summary My ceph version is: 17.2.7 non containerized. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

ATTN: DOCS /api/cluster/user/export
by Kalló Attila 20 Feb '25

20 Feb '25

Hi, Could you please change this doc https://docs.ceph.com/en/quincy/mgr/ceph_api/#post--api-cluster-user-export as it doesn't accept simple strings, but a json array. i.e. I think this: { > "entities": "string" > } should be replaced by something like: > > { > "entities": ["string"] > } Here is an example cURL query just in case: > > ~$ curl -kL -H "Content-Type: application/json" -H "Accept: application/vnd.ceph.api.v1.0+json" -H "Authorization: Bearer $TOKEN" "${URL}api/cluster/user/export" -X POST -d '{ "entities": ["client.admin"] }' |jq -r . > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 206 100 174 100 32 6960 1280 --:--:-- --:--:-- --:--:-- 9809 > [client.admin] > key = ****KEY**** > caps mds = "allow *" > caps mgr = "allow *" > caps mon = "allow *" > caps osd = "allow *" Regards, Attila

4 4

RGW Squid radosgw-admin lc process not working
by Adam Prycki 20 Feb '25

20 Feb '25

Hello, this weekend I was trying to test if squid still suffers from orphaned objects from multipart uploads. https://tracker.ceph.com/issues/44660 I've previously encountered major issues with orphaned objects when using lifecycle transitions and server side copies. Especially on slower clusters. So far I couldn't induce orphaned objects during upload and server side copy :D I couldn't quickly test lifecycle transition because I cannot get `radosgw-admin lc process` to work on any bucket. I'm using 19.2.0 cephadm cluster with fresh rgw pools (I've removed all rgw and created a fresh instnace) Lifecycle config is very simple and based on aws example: <LifecycleConfiguration> <Rule> <ID>transition</ID> <Filter> <Prefix></Prefix> </Filter> <Status>Enabled</Status> <Transition> <Days>0</Days> <StorageClass>STANDARD_IA</StorageClass> </Transition> </Rule> </LifecycleConfiguration> I have STANDARD and STANDARD_IA storage classes in my placement. I'm testing on a bucket with 2 objects, one on STANDARD and one on STANDARD_IA. [ { "key": "default-placement", "val": { "index_pool": "n8.rgw.buckets.default-placement.index", "storage_classes": { "STANDARD": { "data_pool": "n8.rgw.buckets.default-placement.data.STANDARD" }, "STANDARD_IA": { "data_pool": "n8.rgw.buckets.default-placement.data.STANDARD_IA" } }, "data_extra_pool": "n8.rgw.buckets.default-placement.non-ec", "index_type": 0, "inline_data": true } } ] running `radosgw-admin lc process` results in this error lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.14 running `radosgw-admin lc process --bucket test1` results in no output ceph claims that lifecycle executed but no data was copied between pools radosgw-admin lc list [ { "bucket": ":test1:a0f7b0da-04a7-4053-b710-a15897233e86.334949.1", "shard": "lc.14", "started": "Sun, 02 Feb 2025 22:45:37 GMT", "status": "COMPLETE" } ] I've looked on ceph issue tracker but couldn't find anyone mentioning similar lifecycle issues on Squid. (My test buckets are not versioned.) Is this a bug or some kind of misconfiguration? Best regards Adam Prycki

2 2

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

Broke libvirt on compute node due to Ceph Luminous to Nautilus Upgrade
by Pardhiv Karri 18 Feb '25

18 Feb '25

Hi, We recently upgraded our Ceph from Luminous to Nautilus and upgraded the ceph clients on OpenStack (using rbd). All went well and after a few days, we randomly saw instances getting stuck with libvirt_qemu_exporter, which is getting the libvirt stuck on Openstack compute nodes. We had to kill those instances process, and then libvirt is returning. But the issue is happening again on the compute nodes with other instances. Upon doing some research, I found that we need to migrate the instances to use the latest (nautilus) ceph client, as they still use the old(luminous) client when spun up. The only way to get them to have the Nautilus client is to live migrate or reboot. We have thousands of instances, and doing any of those takes a long time without impacting the customer. Is there any other fix to solve this issue without migrating or rebooting the instances? Error on compute hosts: (renamed host and instance id) Feb 18 00:08:00 cmp03 libvirtd[5362]: 2025-02-18 00:08:00.510+0000: 5627: warning : qemuDomainObjBeginJobInternal:4933 : Cannot start job (query, none) for domain instance-009141b8; current job is (query, none) owned by (5628 remoteDispatchDomainBlockStats, 0 <null>) for (322330s, 0s) Thanks, Pardhiv

2 5

Ceph calculator
by Devender Singh 18 Feb '25

18 Feb '25

Hello all Do we have a good cluster design calculator which can suggest failure domain and pool size and min size according the number of nodes and drive and their size for both replicated and EC pools. Regards Dev

3 2

"event": "header_read"
by Szabo, Istvan (Agoda) 18 Feb '25

18 Feb '25

Hi, I haven't really see much about this kind of event in ceph KB: "event": "header_read", "time": "2025-02-18T14:20:00.610389+0700", "duration": 4294967295.999999 However we have huge latencies caused by this operation: "description": "osd_repop(client.3694832962.0:140533664 26.561 e748587/706771 26:86be7203:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.3654639937.1.43:head v 748587'789007122, mlcod=748587'789007120)", "initiated_at": "2025-02-18T14:20:00.610390+0700", "age": 87.067761466999997, "duration": 0.035560873999999999, "type_data": { "flag_point": "commit sent; apply or cleanup", "events": [ { "event": "initiated", "time": "2025-02-18T14:20:00.610390+0700", "duration": 0 }, { "event": "throttled", "time": "2025-02-18T14:20:00.610390+0700", "duration": 0 }, { "event": "header_read", "time": "2025-02-18T14:20:00.610389+0700", "duration": 4294967295.999999 }, { "event": "all_read", "time": "2025-02-18T14:20:00.610402+0700", "duration": 1.3264e-05 }, { "event": "dispatched", "time": "2025-02-18T14:20:00.610405+0700", "duration": 2.3549999999999999e-06 }, { "event": "queued_for_pg", "time": "2025-02-18T14:20:00.610409+0700", "duration": 3.8770000000000003e-06 }, { "event": "reached_pg", "time": "2025-02-18T14:20:00.610429+0700", "duration": 2.0027e-05 }, { "event": "started", "time": "2025-02-18T14:20:00.610438+0700", "duration": 9.8290000000000003e-06 }, { "event": "commit_sent", "time": "2025-02-18T14:20:00.645931+0700", "duration": 0.035492616999999997 }, { "event": "done", "time": "2025-02-18T14:20:00.645951+0700", "duration": 2.0007e-05 } Is there a way to track it down what does this mean? Thanks ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

How to reduce CephFS num_strays effectively?
by jinfeng.biao＠cba.com.au 18 Feb '25

18 Feb '25

Hello folks, We had an issue with the num_strays hit 1 million recently. As a workaround, max bal was increased to 125,000. The stray_num keeps growing at 25k per day. After a recent observation of 10TiB file deletion, the relevant application was stopped. Then we increased purging options to below values mds advanced filer_max_purge_ops 40 mds advanced mds_max_purge_files 1024 mds advanced mds_max_purge_ops 32768 mds advanced mds_max_purge_ops_per_pg 3 And run "du -hsx" to the top level directory mounted to the app that does massive deletion. Despite all above, strays still growing at 60K per day. There are a lot more applications using this CephFS filesystem and only this app is observed perform deletion at this scale. I'm wondering what would be the effective way to cleanup the strays in this situation while making the least impact to production. Note: We are on 14.2.6 thanks James Biao

5 4

Dashboard soft freeze with 19.2.1
by Nicola Mori 18 Feb '25

18 Feb '25

Dear Ceph users, after the upgrade to 19.2.1 I experience a soft freeze of the web UI (old version, activated with `ceph dashboard feature disable dashboard`). When I click on the "Cluster status" widget I get only a blank small tooltip window, instead of e.g. the list of current warnings. After this all the other widgets and menu entries become unresponsive until I refresh the page. Is this a known/common issue? Thanks, Nicola

2 1

Can I convert thick mode files to thin mode files in cephfs?
by 苏察哈尔灿 18 Feb '25

18 Feb '25

Does cephfs not support thin provisioning disk formats like vmware virtual machine disks? For example, the same vmware disk file in the rbd block is: root@backup-server:/mnt# du rbd/Win2003-002-198-bak/Win2003-002-198-bak_5-flat.vmdk -h 1.9G    rbd/Win2003-002-198-bak/ win2003-002-198-bak_5-flat-.vmdk After copying to cephfs: root@backup-server:/mnt# du cephfs/Win2003-002-198-bak_5-flat.vmdk -h 30G     cephfs/Win2003-002-198-bak_5-flat.vmdk The capacity was increased to a thick disk format. Is there any way to change this?  Thank you!

2 1

RBD Performance issue
by vignesh varma 17 Feb '25

17 Feb '25

Hi Team, I have set up a 2 Ceph cluster 3-node each cluster with a two-way RBD mirror. In this setup, Ceph 1 is configured two-way mirror to Ceph 2, and vice versa. The RBD pools are integrated with CloudStack. The Ceph cluster uses NVMe drives, but I am experiencing very low IOPS performance. I have provided the relevant details below. Could you please guide me on how to optimize the setup to achieve higher IOPS? fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --rwmixread=70 --filename=/root/ceph-rbd --name=write --size=1024m --bs=4k --rw=readwrite --iodepth=32 --numjobs=16 --group_reporting read: IOPS=92.5k, BW=361MiB/s (379MB/s)(11.2GiB/31718msec) write: IOPS=39.7k, BW=155MiB/s (163MB/s)(4922MiB/31718msec); 0 zone resets Hardware Specifications: - CPU: Intel(R) Xeon(R) Gold 5416S - RAM: 125 GB - Storage: 8 x 7 TB NVMe disks (Model: UP2A67T6SD004LX) [Drive specifications](https://www.techpowerup.com/ssd-specs/union-memory-uh711a-7… - Network: 4 x 25 Gbps interfaces configured with LACP bonding Each server in the setup is equipped with the above configuration.

3 4

Re: Posix backend for Radosgw
by Varada Kari 17 Feb '25

17 Feb '25

Anyone tried the posix backend for Ceph radosgw? Appreciate any pointers here on configuring and testing. Varada On Fri, Feb 14, 2025 at 12:40 PM Varada Kari <varada.kari(a)gmail.com> wrote: > Hi, > > I am trying to test the posix backend for radosgw in my test machine. It > is running Ubuntu 22.04 with the latest squid release(19.2.0) > I have configured a small single node cluster without any osd. > > $ sudo ceph -v > ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid > (stable) > $ sudo ceph -s > cluster: > id: 92d1770c-e83e-11ef-bcc9-5f5839c0a1fa > health: HEALTH_WARN > Failed to place 1 daemon(s) > Reduced data availability: 1 pg inactive > OSD count 0 < osd_pool_default_size 3 > > services: > mon: 1 daemons, quorum testserver (age 3d) > mgr: testserver.iavwwx(active, since 2d) > osd: 0 osds: 0 up, 0 in > > data: > pools: 1 pools, 1 pgs > objects: 0 objects, 0 B > usage: 0 B used, 0 B / 0 B avail > pgs: 100.000% pgs unknown > 1 unknown > > progress: > Global Recovery Event (0s) > [............................] > > > Test machine is running a raided xfs file system. > Using the below config in the radosgw.conf > > [client] > > rgw backend store = dbstore > rgw config store = dbstore > dbstore db dir = /mnt/raidx/rgw/dbstore > dbstore_config_uri = file:///mnt/raidx/dbstore/config.db > > rgw filter = posix > rgw posix base path = /mnt/raidx/posix/root > rgw posix database root = /mnt/raidx/rgw/posix/lmdb > > [client.radosgw.gateway] > rgw init timeout = 1200 > rgw frontends = beast port=80 > > Not able to start the radosgw, fails to initialize the rados. > > 2025-02-11T00:35:13.183-0800 7f76b02eca40 0 ceph version 19.2.0 > (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, > pid 2673837 > 2025-02-11T00:35:13.183-0800 7f76b02eca40 0 framework: beast > 2025-02-11T00:35:13.183-0800 7f76b02eca40 0 framework conf key: port, > val: 7480 > 2025-02-11T00:35:13.183-0800 7f76b02eca40 1 init_numa not setting numa > affinity > 2025-02-11T00:35:13.183-0800 7f76b02eca40 0 pidfile_write: ignore empty > --pid-file > 2025-02-11T00:35:13.187-0800 7f76b02eca40 -1 rgw main: ERROR: failed to > initialize config store 'dbstore': unable to open database file > 2025-02-11T00:35:13.187-0800 7f76b02eca40 -1 Couldn't init storage > provider (RADOS) > > My intent is to run a standalone radosgw with a filesystem backend, which > works as a s3 gateway. It would be better if i can avoid running a ceph-mon > and other daemons as it is just working as a s3 gateway. > Are there any configuration steps I am missing here? Please point me to > any documentation available on this. Would appreciate any help on this. > > > Best, > Varada > > > > >

2 1

(no subject)
by Vignesh Varma 17 Feb '25

17 Feb '25

Best Regards, Vignesh Varma G Cloud Engineer www.stackbill.com

1 0

Create a back network?
by Nicola Mori 15 Feb '25

15 Feb '25

Dear Ceph users, I have a question about the benefits of creating a back network for my cluster. Currently, my machines are all connected to a gigabit switch, each with 2 bonded interfaces. The cluster just have a front network. Recently I got a new gigabit switch, so now I have 2 and I was wondering if there would be any benefit in setting up a back network on one switch and a front network on the other, using one interface per network. In this way I'll loose the redundancy, but other benefits and drawbacks are not clear to me. Would some expert help me in better understanding this issue, please? Thanks in advance, Nicola

4 3

Automatic OSD activation after host reinstall
by Eugen Block 14 Feb '25

14 Feb '25

Good morning, this week I observed something new, I think. At least I can't recall having seen that yet. Last week I upgraded a customer cluster to 18.2.4 (everything worked fine except RGWs keep crashing [0]), this week I reinstalled the OS on one of the hosts. And after a successful registry login, the OSDs were automatically activated without me having to run 'ceph cephadm osd activate <host>' as documented in [1]. Zac improved the docs just last week, is that obsolete now? Or is that version specific? A few weeks ago we reinstalled our own Ceph servers as well, but we still run Pacific 16.2.15, and there I had to issue the OSD activation manually. Can anyone confirm this? Thanks! Eugen [0] https://tracker.ceph.com/issues/69885 [1] https://docs.ceph.com/en/latest/cephadm/services/osd/#activate-existing-osds

2 2

Does the number of PGs affect the total usable size of a pool?
by Work Ceph 13 Feb '25

13 Feb '25

Hello guys, Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single pool that consumes all OSDs of all nodes. After adding another host, I noticed that no extra space was added. Can this be a result of the number of PGs I am using? I mean, when adding more hosts/OSDs, should I always consider increasing the number of PGs from a pool?

4 11

Ceph Events Survey -- Your Input Wanted!
by Dan van der Ster 13 Feb '25

13 Feb '25

Hi all, The Ceph Foundation and Ambassadors are busy building a full calendar of Ceph events for 2025! As many of you know, Ceph has always thrived on its amazing community. Events like Ceph Days and Cephalocon are key points of interaction where we all can learn, connect, and share experience. Following the success of Cephalocon at CERN and Ceph Days in India, we’ve announced Ceph Days London & Silicon Valley -- check out https://ceph.io/en/community/events/ to get involved. And watch that space -- Ceph Days in Seattle, New York, and Berlin will be announced soon! Looking forward, we need your help to help shape our future events... and to plan our next Cephalocon! If you have a moment, please share your thoughts in our Ceph Events survey: https://forms.gle/Rm41d547Rb59S8xf9 Looking forward to seeing you at an event soon! --- Dan van der Ster Ceph Executive Council

1 0

Radosgw log Custom Headers
by Ansgar Jazdzewski 13 Feb '25

13 Feb '25

Hi folks, i like to make sure that the RadosGW is using the X-Forwarded-For as source-ip for ACL's However i do not find the information in the logs: i have set (using beast): ceph config set global rgw_remote_addr_param http_x_forwarded_for ceph config set global rgw_log_http_headers http_x_forwarded_for https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_log_http_he… I hope someone can point me in the right direction! Thanks, Ansgar

3 4

<---- breaks grouping of messages
by Marc 13 Feb '25

13 Feb '25

It is annoying that the subject modification of putting [ceph-users] first is breaking grouping of messages.

2 1

NFS recommendations
by Devin A. Bougie 12 Feb '25

12 Feb '25

Hi, All. We are new to Ceph, and looking for any general best practices WRT exporting a Cephfs file system over NFS. I see several options in the documentation and have tested several different configurations, but haven’t yet seen much difference in our testing and aren’t sure exactly which configuration is generally recommended to start with. We have a single Cephfs filesystem in our cluster of 10 hosts. Five of our hosts are OSDs with the spinning disks that make up our Cephfs data pool, and only run the osd services (osd, crash, ceph-exporter, node-exporter, and promtail). The other five hosts are “admin” hosts that run everything else (mds, mgr, mon, etc.). Our current setup follows the "HIGH-AVAILABILITY NFS” documentation, which gives us an Ingress.nfs.cephfs service with the haproxy and keepalived daemons and a nfs.cephfs service for the actual nfs daemons. If there are no downsides to this approach, are there any recommendations on placement for these two services? Given our cluster, would it be best to run both on the admin nodes? Or would it be better to have the ingress.nfs.cephfs service on the admin nodes, and the backend nfs.cephfs services on the osd nodes? Alternatively, are there advantages in using the “keepalive only” mode (only keepalived, no haproxy)? Or does anyone recommend doing something completely different, like using Pacemaker and Corosync to manage our NFS services? Any recommendations one way or another would be greatly appreciated. Many thanks, Devin

3 3

cephadm orchestrator feature request: scheduled rebooting of cluster nodes
by Robert Sander 12 Feb '25

12 Feb '25

Hi, it just came across my mind that it would be nice if the orchestrator would be able to reboot all nodes of a cluster in an un-interrupting way. This could be done similar to the upgrade where the orchestrator checks if daemons on a host can be restarted. Including "ceph osd add-noout $(hostname)" before the reboot and removing it afterwards. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 1

Upgrade quincy to reef
by cristian.tavarez＠hostpapa.com 12 Feb '25

12 Feb '25

Hello I'm trying to update my ceph cluster from the latest release of quincy 17.2.8 to reef 18.2.4. After starting the upgrade with 'ceph orch upgrade start --ceph-version 18.2.4' it stopped having upgraded 3/4 of my MGR daemons and everything else stayed on quincy. Any ceph command is not working saying orchestrator module is not found and Error ENOENT: Module not found. I have already tried looking at the osd issue hinted at https://tracker.ceph.com/issues/67329 but this happened on a test cluster without any OSDs on it as well. The current status given in the upgrade is the following ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v18.2.4", "progress_id": "a9e4d538-4c98-4307-8170-a82ca8ff300c", "target_id": "2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a", "target_digests": ["quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906", "quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77"], "target_version": "18.2.4", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": "UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host hostname-redacted at addr (IPv4-address)", "paused": true, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} How can I make the orchestrator work again to either rollback or upgrade to a newer version? the goal is to update this all the way to squid. Thank you

3 3

ceph rdb + libvirt
by Albert Shih 12 Feb '25

12 Feb '25

Hi, I need to create manually few VM with KVM. I would like to know if they are any difference between using a libvirt module and kernel module to access a ceph cluster. Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mer. 12 févr. 2025 16:14:15 CET

6 6

512e -> 4Kn hdd
by Marc 12 Feb '25

12 Feb '25

I used to have a tool from hgst/wd to 'convert' drives to 4Kn, but after a mentally challenging interaction with some wd support, I decided to try and use other solutions mentioned on the internet this seems to produce output similar to the previously used tool sg_format --format --size=4096 /dev/sdd Anyone have experience with this? This is first time the drives are already in the nodes, so I don't have a power cycle on them.

3 3

Grafana certificate issue
by Devender Singh 12 Feb '25

12 Feb '25

Hello all Recently upgraded to 19.2.1 and started facing .. Invalid grafana certificate on hostnode1: Invalid certificate key: [('PEM routines', '', 'no start line’)] Tried suggest steps, even re-deployed grafana but no solution? 1. [WRN] CEPHADM_CERT_ERROR: Invalid grafana certificate on host node1: Invalid certificate key: [('PEM routines', '', 'no start line')] Detected invalid grafana certificates. Please, use the following commands: > ceph config-key set mgr/cephadm/node1/grafana_crt -i <path-to-ctr-file> > ceph config-key set mgr/cephadm/node1/grafana_key -i <path-to-key-file> to set valid key and certificate or reset their value to an empty string in case you want cephadm to generate self-signed Grafana certificates. Once done, run the following command to reconfig the daemon: > ceph orch daemon reconfig grafana.ld6-converged03n.ld6.netskrt.org Also, when I exit my host from maintenance, I get below… but when I rerun same command, it works fine, why? 2. # ceph orch host maintenance exit node2 Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 805, in _host_maintenance_exit raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 238, in raise_if_exception e = pickle.loads(c.serialized_exception) TypeError: __init__() missing 2 required positional arguments: 'hostname' and ‘addr' Regards Dev

2 1

RGW - S3 bucket browser and/or S3 explorer
by notna＠ist-einmalig.de 12 Feb '25

12 Feb '25

Hi all. Is anyone aware of a ("S3 bucket browser" | "S3 explorer"), which works with RadosGW static websites? I've played around with those below, but none of them works (in my environment): - https://github.com/qoomon/aws-s3-bucket-browser - https://github.com/guimou/ceph-js-s3-explorer - https://github.com/awslabs/aws-js-s3-explorer - https://github.com/awslabs/aws-js-s3-explorer/tree/v2-alpha Any hints are highly appreciated!

4 9

Ceph RGW Cloud-Sync Issue
by Mark Selby 12 Feb '25

12 Feb '25

This is a 18.2.4 reef cluster Looking for help with cloud sync module. Here is my setup 1. Two Ceph clusters with RGW setup in a MultiSite configuration. There is no data synchronization between the clusters. No data sync groups, flows, or pipes. 2. We use a multi tenant setup with a small number of tenants (10). We have one tenant that is writing to the same bucket in different rgw zones. The names of files are unique in each zone. 3. We are using the cloud-sync module to copy data from our ceph clusters to aws. I have pasted the commands used to create the sync zone The cloud-sync module works but for one issue that I can not figure out. For some reason another bucket in another tenant is showing up as target to sync but was never specified in the original config. I can never get a clean radosgw-admin sync status because there are recovering shards from a bucket in another tenant that I never intended to sync to the cloud. The data sync status of the recovering shard is pointing to a bucket that never had any cloud sync specified. tenant1/bucket1 was never part of any tier config. radosgw-admin sync error list --rgw-zone cloud-sync shows a never ending list of objects that can not be synced from tenant1/bucket1 there are 4 different types of error messages. "message": "failed to sync bucket instance: (22) Invalid argument" "message": "failed to sync bucket instance: (39) Directory not empty" "message": "failed to sync object(16) Device or resource busy" "message": "failed to sync object(39) Directory not empty" One other point is that radosgw-admin sync info --rgw-zone cloud-sync --bucket tenant1/bucket1 seems to indicate that that is part of the cloud-sync config. What I just want to do is remove tenant1/bucket1 from the sync config and stop the data sync such that the shards recover and the error log stop accumulating messages. I would really like a clean sync status for my cloud tier zone. It would also be nice to help understand what is happening here. All and any help is greatly appreciated. # # sync and data sync status # radosgw-admin sync status --rgw-zone=cloud-sync realm 02b82a81-7b16-4760-ab94-08b79176397e (realm) zonegroup 0e44f33d-4e70-4ba6-8c75-794d3d0b69f8 (zonegroup) zone 34e5298b-4537-4ada-b0d9-75579dc9640a (cloud-sync) current time 2025-02-11T23:56:52Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 792edbc3-8516-4014-aad4-8e7b6ebe2757 (zone-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards 1 shards are recovering recovering shards: [72] source: f3e72377-2fb1-4ab6-aa0a-76b91d157fb3 (zone-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 4 shards behind shards: [72,75,77,78] oldest incremental change not applied: 2025-02-11T15:56:34.654956-0800 [75] 10 shards are recovering recovering shards: [68,69,70,71,72,73,74,76,77,78] radosgw-admin data sync status --rgw-zone cloud-sync --source-zone zone-2 --shard-id 68 { "shard_id": 68, "marker": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000357645", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2025-02-11T23:42:34.266680Z" }, "pending_buckets": [ "tenant1/bucket1:f3e72377-2fb1-4ab6-aa0a-76b91d157fb3.2251620.1:0" ], "recovering_buckets": [ "tenant1/bucket1:f3e72377-2fb1-4ab6-aa0a-76b91d157fb3.2251620.1:0[0]" ], "current_time": "2025-02-11T23:58:08Z" } # # cloud sync zone creation # radosgw-admin zone create \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --endpoints=https://ceph-server:4430 \ --tier-type=cloud radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=s-cloud-sync \ --access-key=<ACCESS> \ --secret=<SECRET> radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connections[].id=aws-cloud-sync radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connections[-1].endpoint=https://s3.amazonaws.com,connections[-1].access_key=<ACCESS>,connections[-1].secret=<SECRET> radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=profiles[].connection_id=aws-cloud-sync radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=profiles[-1].connection_id=aws-cloud-sync,profiles[-1].source_bucket='tenant0:bucket0',profiles[-1].target_path=aws-bucket radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connection_id=aws-cloud-sync,target_path=aws-bucket

1 0

Cephadm cluster setup with unit-dir and data-dir
by Rajmohan Ramamoorthy 11 Feb '25

11 Feb '25

Hi, I wanted to understand the behaviour of using --unit-dir and --data-dir while setting up the ceph cluster using cephadm. Can I pass on a reference to a mount points on a different disk (other than the root disk) for these options? The deployment I am trying to achieve is to persist the Ceph containers systemd unit files and their persistent cluster data on a different disk than the root disk. This way, the nodes retains its identity even if the Operating System disk is redeployed on the root disk and the persistent data (systemd unit files and container data) is retained on the other disk. Is this possible with cephadm? I tried pointing unit-dir and data-dir to a different disk mount point but it failed. The unit files are always created in the /etc/systemd/system directory in the root disk. Unit directory: https:// docs.ceph.com/en/latest/man/8/cephadm/?highlight=orch%20device%20zap#cmdopt… Data directory: https:// docs.ceph.com/en/latest/man/8/cephadm/?highlight=orch%20device%20zap#cmdopt… Regards, Rajmohan R

1 0

Ceph Steering Committee Notes 2025-02-10
by Ernesto Puerta 10 Feb '25

10 Feb '25

Hi Cephers, These are the topics covered in today's meeting: - *[Patrick] Any objections to running clean-ci now?* Do you have any important branches in ceph-ci? - https://github.com/ceph/ceph/pull/61709 - Joseph asked if it could delete branches similarly to GitHub UI/API so that they can be restored if needed? - *[Radek] Switching default build type for non-.git builds** to RelWithDebInfo.* This affects downstreams (if no build type is specified)! - Matan's PR: https://github.com/ceph/ceph/pull/61637/ - [Sam] Can it be done with CFLAGS instead? To be announced in ceph-dev. Set default/fall back to Release, which seems the safest. - [Greg] Not all releases are of "Release" type, some include debug info. - [Matan] 1. Release: `-O3 -DNDEBUG` 2. Debug: `-O0 -g` 3. RelWithDebInfo: `-O2 -g -DNDEBUG` 4. MinSizeRel: `-Os -DNDEBUG` - [JosephM] this needs to be backported. - *[Dan-probably absent today**] Upgrade 17.2.8 to 18.2.4 is currently broken*: https://tracker.ceph.com/issues/67660 (Cephadm feature backported to quincy before reef). - 18.2.5 is next, right? Can it be quick pls? - QA question: do we test upgrade from the candidate to the current maj_ver+1? This kind of thing should have been caught. - [Neha]To be followed up offline (QA is currently blocked by the ceph-ci too-many-branches issue). - *[Joseph Mundackal] - containerized builds* - https://github.com/ceph/ceph/pull/59841 - initial version of this merged. Looking for feedback and missing features. - Open trackers to build on ubuntu 24 - https://tracker.ceph.com/issues/62804 - https://tracker.ceph.com/issues/69240 (open PR: https://github.com/ceph/ceph/pull/61326) - https://tracker.ceph.com/issues/69525 - [Matan] There might be issues with compiler versions (e.g.: Ubuntu 22) - *[cbodley] stalebot on backport branches was enabled recently* in https://github.com/ceph/ceph/pull/60736 - can we revert? backport prs aren't tested promptly so someone would have to micromanage them to prevent closing - Revert PR: https://github.com/ceph/ceph/pull/61743 - *[Adam Kraitman] Virtualization solution for hosting the lab services* - VMWare (IBM paid/agreement with Broadcom) vs. Proxmox - [DavidG] temporary (~1 year) solution while infra is containerized. - [Greg] precedent of iSCSI. - NVMe-oF support in Proxmox? Not needed, librbd. Discussion on Tentacle schedule postponed till next week. Kind Regards, Ernesto

1 0

19.2.1: HEALTH_ERR 27 osds(s) are not reachable. (Yet working normally...)
by Harry G Coin 10 Feb '25

10 Feb '25

19.2.1 complains of all osd's being unreachable, as their public address isn't in the public subnet. However, they all are within the subnet, and are working normally as well. It's embarrassing for the dashboard to glow red of a totally crippled osd roster --- while all is working normally. This existed in the previous, but was working prior to 19. Detail: Notice, for osd.0, the dashboard lists public_addr [fc00:1002:c7::44]:6807/4160993080 But, we have in the logs: 7/2/25 03:35 PM[ERR] osd.0's public address is not in 'fc00:1002:c7::/64' subnet 7/2/25 03:35 PM[ERR][ERR] OSD_UNREACHABLE: 27 osds(s) are not reachable 7/2/25 03:35 PM[ERR]Health detail: HEALTH_ERR 27 osds(s) are not reachable However, as per the osd.0 attributes, the public address for osd.0 is well inside the stated public subnet. All the osd's are similarly configured, working, and held to be unreachable at the same time, for the same reason. Tell me there's a way to fix this without waiting a further half year....

2 3

Reef: rgw daemon crashes
by Eugen Block 10 Feb '25

10 Feb '25

Hi, yesterday I upgraded a customer cluster to Reef (18.2.4). The upgrade went quite well, nothing happened for hours, until it did. One of the two RGW daemons has crashed twice in the last 12 hours. Here's one backtrace: ---snip--- Feb 06 23:19:58 storage09 conmon[2501983]: radosgw: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/redhat-linux-build/boost/include/boost/context/posix/protected_fixedsize_stack.hpp:70: boost::context::stack_context boost::context::basic_protected_fixedsize_stack<traitsT>::allocate() [with traitsT = boost::context::stack_traits]: Assertion `0 == result' failed. Feb 06 23:19:58 storage09 conmon[2501983]: *** Caught signal (Aborted) ** Feb 06 23:19:58 storage09 conmon[2501983]: in thread 7f77a10b2640 thread_name:radosgw Feb 06 23:19:58 storage09 conmon[2501983]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Feb 06 23:19:58 storage09 conmon[2501983]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f78adeba6f0] Feb 06 23:19:58 storage09 conmon[2501983]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f78adf0794c] Feb 06 23:19:58 storage09 conmon[2501983]: 3: raise() Feb 06 23:19:58 storage09 conmon[2501983]: 4: abort() Feb 06 23:19:58 storage09 conmon[2501983]: 5: /lib64/libc.so.6(+0x2871b) [0x7f78adea471b] Feb 06 23:19:58 storage09 conmon[2501983]: 6: /lib64/libc.so.6(+0x37386) [0x7f78adeb3386] Feb 06 23:19:58 storage09 conmon[2501983]: 7: /usr/bin/radosgw(+0x361cb2) [0x561ed9ef3cb2] Feb 06 23:19:58 storage09 conmon[2501983]: 8: /usr/bin/radosgw(+0x361db8) [0x561ed9ef3db8] Feb 06 23:19:58 storage09 conmon[2501983]: 9: /usr/bin/radosgw(+0x36e15e) [0x561ed9f0015e] Feb 06 23:19:58 storage09 conmon[2501983]: 10: /usr/bin/radosgw(+0x357558) [0x561ed9ee9558] Feb 06 23:19:58 storage09 conmon[2501983]: 11: /usr/bin/radosgw(+0x34546c) [0x561ed9ed746c] Feb 06 23:19:58 storage09 conmon[2501983]: 12: /usr/bin/radosgw(+0x358f0a) [0x561ed9eeaf0a] Feb 06 23:19:58 storage09 conmon[2501983]: 13: /usr/bin/radosgw(+0xb705de) [0x561eda7025de] Feb 06 23:19:58 storage09 conmon[2501983]: 14: /usr/bin/radosgw(+0x3c6aed) [0x561ed9f58aed] Feb 06 23:19:58 storage09 conmon[2501983]: 15: /lib64/libstdc++.so.6(+0xdbad4) [0x7f78ae258ad4] Feb 06 23:19:58 storage09 conmon[2501983]: 16: /lib64/libc.so.6(+0x89c02) [0x7f78adf05c02] Feb 06 23:19:58 storage09 conmon[2501983]: 17: /lib64/libc.so.6(+0x10ec40) [0x7f78adf8ac40] ---snip--- I didn't find anything helpful in the tracker, only this report on this list [0] without a response from a year ago. The other daemon on a different host seems to be stable for now. This is no multi-site deployment, just two RGWs for a single zone. Any comments/pointers are appreciated! I can file a tracker issue if this is something new. Thanks! Eugen [0] https://www.spinics.net/lists/ceph-users/msg80956.html

1 2

Ceph upgrade quincy to reef
by cristian.tavarez＠hostpapa.com 08 Feb '25

08 Feb '25

Hello I'm trying to update my ceph cluster from the latest release of quincy 17.2.8 to reef 18.2.4. After starting the upgrade with 'ceph orch upgrade start --ceph-version 18.2.4' it stopped having upgraded 3/4 of my MGR daemons and everything else stayed on quincy. Any ceph command is not working saying orchestrator module is not found and Error ENOENT: Module not found. I have already tried looking at the osd issue hinted at https://tracker.ceph.com/issues/67329 but this happened on a test cluster without any OSDs on it as well. The current status given in the upgrade is the following ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v18.2.4", "progress_id": "a9e4d538-4c98-4307-8170-a82ca8ff300c", "target_id": "2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a", "target_digests": ["quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906", "quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77"], "target_version": "18.2.4", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": "UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host hostname-redacted at addr (IPv4-address)", "paused": true, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} How can I make the orchestrator work again to either rollback or upgrade to a newer version? the goal is to update this all the way to squid. Thank you

1 0

postgresql vs ceph, fsync
by Petr Holubec 08 Feb '25

08 Feb '25

We are evaluating pros and cons of running postgresql backed by ceph. We know that running pg on dedicated physical hw is highly recommended, but we've got our reasons. So to the question: What could happen if we switch fsync to off on postgre backed by ceph? The increase of perfomance is huge, which is the reason we are considering it. Ceph pool is running on osds connected to controllers with batteries so we could mitigate power losses. What happens when an osd run out of space? Are there other considerations?

2 1

Quick question: How to check if krbd is enabled?
by Alan Murrell 07 Feb '25

07 Feb '25

Hello, I Googled for this answer but could not find it. I am running Reef 18.2.2 . I need to find out if krbd is enabled on my cluster but I cannot figure out what command to check that. I have checked the Pool settings in the GUI but I see nothing about krbd. I suspect it is enabled, but I just need to confirm. All the posts I have found so far about enabling or disabling it have to do with doing so in Proxmox, and while we do have Proxmox connected to our cluster, the cluster was not created within Proxmox, so that option is not available in there either (nor would I expect it to be) Thanks! 😊

2 2

[RGW] Full replication gives stale recovering shard
by Gilles Mocellin 07 Feb '25

07 Feb '25

Hello Cephers, Since I was not confident in my replication status, I've done a radosgw sync init one after the other, in both of my zones. Since then, there art stale recovering shards. Incremental replication seems OK. Cluster A sync status: realm 421c9bbd-83cc-4d85-a25a-ca847f225bfe (R) zonegroup 2c5ccf52-88be-4b2f-b4b9-c55f73bbacd1 (ZG) zone 3e3b0a2b-1864-4923-93a6-237a13b51594 (Z1) current time 2025-02-06T19:03:10Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: aefd4003-1866-4b16-b1b3-2f308075cd1c (Z2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [65,66,77] oldest incremental change not applied: 2025-02-06T20:02:53.441838+0100 [66] 111 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101, 102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] Cluster B sync status: realm 421c9bbd-83cc-4d85-a25a-ca847f225bfe (R) zonegroup 2c5ccf52-88be-4b2f-b4b9-c55f73bbacd1 (ZG) zone aefd4003-1866-4b16-b1b3-2f308075cd1c (Z2) current time 2025-02-06T19:03:37Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 3e3b0a2b-1864-4923-93a6-237a13b51594 (Z1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 7 shards behind shards: [58,59,60,62,64,66,67] oldest incremental change not applied: 2025-02-06T20:03:27.831651+0100 [60] 81 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,57,58,59,60,61,62,63,64,65,66,67,68,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123 ,124,126,127] Restarting full sync does nothing, except eventually adding recovering shards ! I also have some large omap objects in Z1.rgw.log pool. Where I see a lot of objects with names beginning by bucket.sync-status.*. What can I do to remove those recovering shards ? Some trimming (bilog ?), deleting objects in Z1.rgw.log pool ? -- Gilles

1 0

Cannot stop OSD
by Alan Murrell 07 Feb '25

07 Feb '25

Hello, I am running a Ceph cluster installed with cephadm. Version is 18.2.2 reef (stable). I am moving the DB/WAL from my HDDs to SSD, and have been doing fine on all the OSDs until I got to one in particular (osd.14) From the cephadm shell, when I run 'ceph orch daemon stop osd.14', nothing happens: it does not get marked as Down. If I mark is as Down in the GUI, it does show as Down then a few seconds later it gets marks as Up again. I was running 'journalctl -xf | grep osd.14' to see if any errors came up, but nothing did. Not sure where to check next to try to sort this out?

2 7

Re: RGW: HEAD ok but GET fails
by Simon Campion 07 Feb '25

07 Feb '25

We see the same symptoms. We have an S3 object for which HEAD requests return 200, but GET requests return 404. We searched the S3 gateway logs for write requests that happened on the object before GET requests started failing. Strangely, we see succeeding GET requests and then failing GET requests without any write request (such as DELETE) being logged in between. So, we're unsure whether this is related to the known bug in DELETE requests mentioned earlier in this thread (https://tracker.ceph.com/issues/63935) Excerpts from the S3 gateway logs for the object in question: ``` ... # Last upload of the file 2025-01-16 08:09:23.112 debug 2025-01-16T08:09:23.111+0000 7f4b739f7640 1 beast: 0x7f4adc1094a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.105 +0000] "POST /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?uploads=&x-id=CreateMultipartUpload HTTP/1.1" 200 359 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.006000043s 2025-01-16 08:09:23.457 debug 2025-01-16T08:09:23.456+0000 7f137c405640 1 beast: 0x7f13049b64a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.168 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=4&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.288000762s 2025-01-16 08:09:23.460 debug 2025-01-16T08:09:23.460+0000 7fc386dfa640 1 beast: 0x7fc2fef834a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.161 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=2&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.298000038s 2025-01-16 08:09:23.461 debug 2025-01-16T08:09:23.460+0000 7f4b81212640 1 beast: 0x7f4adc1094a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.158 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=5&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.302002162s 2025-01-16 08:09:23.461 debug 2025-01-16T08:09:23.460+0000 7f4b629d5640 1 beast: 0x7f4adab5e4a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.162 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=3&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.298002154s 2025-01-16 08:09:23.461 debug 2025-01-16T08:09:23.460+0000 7f4ba5a5b640 1 beast: 0x7f4adaa5c4a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.162 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=1&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.298002154s 2025-01-16 08:09:23.520 debug 2025-01-16T08:09:23.519+0000 7f4bd8ac1640 1 beast: 0x7f4adab5e4a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.465 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=6&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 4225533 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.053000379s 2025-01-16 08:09:23.542 debug 2025-01-16T08:09:23.541+0000 7f4ba3a57640 1 beast: 0x7f4ada9db4a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.520 +0000] "POST /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=CompleteMultipartUpload HTTP/1.1" 200 548 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.021000152s ... # GET requests return 200 2025-01-16 17:27:57.574 debug 2025-01-16T17:27:57.573+0000 7f138d427640 1 beast: 0x7f13048334a0: 10.70.3.93 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:17:27:57.498 +0000] "GET /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?x-id=GetObject HTTP/1.1" 200 30439933 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.075000212s ... # Last write request we could find in the logs (a CopyObject request) 2025-01-17 08:11:23.721 debug 2025-01-17T08:11:23.719+0000 7fc394e16640 1 ====== starting new request req=0x7fc300cbd4a0 ===== 2025-01-17 08:11:23.721 debug 2025-01-17T08:11:23.720+0000 7fc3da6a1640 1 ====== req done req=0x7fc3009b74a0 op status=0 http_status=200 latency=0.007000001s ====== 2025-01-17 08:11:23.721 debug 2025-01-17T08:11:23.720+0000 7fc3da6a1640 1 beast: 0x7fc3009b74a0: 10.70.3.153 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:08:11:23.713 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?x-id=CopyObject HTTP/1.1" 200 252 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.007000001s ... # Last GET request returning 200 2025-01-17 09:34:50.256 debug 2025-01-17T09:34:50.255+0000 7fc420f2e640 1 beast: 0x7fc3009b74a0: 10.70.3.91 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:09:34:50.151 +0000] "GET /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?x-id=GetObject HTTP/1.1" 200 30439933 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.104000047s 2025-01-17 09:36:47.138 debug 2025-01-17T09:36:47.137+0000 7f13da4c1640 1 beast: 0x7f13061e64a0: 10.70.3.91 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:09:36:47.136 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000003s 2025-01-17 10:01:57.580 debug 2025-01-17T10:01:57.579+0000 7f4bf32f6640 1 beast: 0x7f4adace14a0: 10.70.3.93 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:01:57.578 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000007s 2025-01-17 10:02:16.805 debug 2025-01-17T10:02:16.804+0000 7f4b39983640 1 beast: 0x7f4ada4504a0: 10.70.3.28 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:02:16.803 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000008s 2025-01-17 10:23:35.706 debug 2025-01-17T10:23:35.705+0000 7f142f56b640 1 beast: 0x7f13058534a0: 10.70.3.91 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:23:35.704 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000003s 2025-01-17 10:25:38.406 debug 2025-01-17T10:25:38.405+0000 7f4b719f3640 1 beast: 0x7f4adbafd4a0: 10.70.3.28 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:25:38.404 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000007s # Now GET requests return 404s. Note there are only HEAD requests visible in the logs since the last GET request that returned a 200. 2025-01-17 10:31:56.619 debug 2025-01-17T10:31:56.617+0000 7fc37e5e9640 0 req 17300774944515556197 0.001000000s s3:get_obj iterate_obj() failed with -2 2025-01-17 10:31:56.619 debug 2025-01-17T10:31:56.618+0000 7fc35058d640 1 ====== req done req=0x7fc3011464a0 op status=-2 http_status=404 latency=0.002000001s ====== 2025-01-17 10:31:56.619 debug 2025-01-17T10:31:56.618+0000 7fc35058d640 1 beast: 0x7fc3011464a0: 10.70.3.93 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:31:56.615 +0000] "GET /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?x-id=GetObject HTTP/1.1" 404 294 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.002000001s # HEAD requets still return 200. 2025-01-17 10:32:48.196 debug 2025-01-17T10:32:48.195+0000 7fc3a2e32640 1 beast: 0x7fc2feb7b4a0: 10.70.3.91 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:32:48.194 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000001s ``` Here's the bucket index entry. The mtime matches the time of the last CopyObject request: ``` bash-5.1$ radosgw-admin bi list --bucket=buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4 ... { "type": "plain", "idx": "blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059", "entry": { "name": "blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059", "instance": "", "ver": { "pool": 20, "epoch": 0 }, "locator": "", "exists": true, "meta": { "category": 1, "size": 30439933, "mtime": "2025-01-17T08:11:23.715427Z", "etag": "ea3dc44b4f3e9fa8874c0022664edeaa-6", "storage_class": "", "owner": "obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1", "owner_display_name": "obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1", "content_type": "", "accounted_size": 30439933, "user_data": "", "appendable": false }, "tag": "26caa730-b29f-4262-ac51-6a47db5585d2.75677055.5371543061660465468", "flags": 0, "pending_map": [], "versioned_epoch": 0 } }, ``` There is a RADOS object with the relevant name, matching mtime, but with size 0: ``` bash-5.1$ rados -p general-s3.rgw.buckets.data ls | grep 7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 26caa730-b29f-4262-ac51-6a47db5585d2.75671339.1_blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 bash-5.1$ rados -p general-s3.rgw.buckets.data stat 26caa730-b29f-4262-ac51-6a47db5585d2.75671339.1_blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 general-s3.rgw.buckets.data/26caa730-b29f-4262-ac51-6a47db5585d2.75671339.1_blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 mtime 2025-01-17T08:11:23.000000+0000, size 0 ``` We run Ceph 19.2.0 on K8s, orchestrated with Rook. We have multiple replicas for the S3 gateway. The problems started to appear after updating from Reef to Squid. So far, we only noticed the problem for a specific bucket used as a BuildKit cache. We deleted and recreated the bucket for BuildKit, and the problem appeared again after running our BuildKit jobs against the new bucket for a few days. At first, we thought the problem was caused by a bug/unwanted behavior in the CopyObject implementation. It looks like CopyObject fails to update the index but nevertheless updates the RADOS objects holding the S3 object, replacing (multiple parts of?) a large S3 object with a single object of size 0. However, the fact that we see a successful GET request ten minutes after the last CopyObject request casts doubt on the hypothesis that CopyObject is the problem. We'd be very grateful for any advice on how to troubleshoot this further. Simon

1 0

RGW issue, lost/corrupted bucket metadata/index ?
by Cyril Duval 07 Feb '25

07 Feb '25

Hello folks ! This is my first time posting here, sorry in advance if I miss some of the etiquette. I'm facing some issues with RGW and especially the radosgw-admin tool, on what appears to be lost bucket metadata, or maybe corruption of some elements of the buckets. I have opened a ticket on https://tracker.ceph.com/issues/69818 where I put more details, but in summary I have the `failure: (2) No such file or directory:` kind of error on most radosgw-admin commands interacting with these buggy buckets. Lifecycle policies seem to be affected too, as I have a `lifecycle: RGWLC::process() failed to get obj head lc.5, ret=-5` error. I'm running Ceph 18.2.2 there. Is it something familiar to anyone ? Or does anyone know where I can have a look further to better understand the issue and solve it ? I tried giving a shot at the cpp code, but unfortunately I'm no expert there, I'm lost. Thanks in advance, -- Cyril Duval cyril.duval(a)diabolocom.com

1 0

Measuring write latency (ceph osd perf)
by Jan Kasprzak 07 Feb '25

07 Feb '25

Hello, ceph users, TL;DR: how can I look into ceph cluster write latency issues? Details: we have a HDD-based cluster (with NVMe for metadata), about 20 hosts, 2 OSD per host, mostly used as RBD storage for QEMU/KVM virtual machines. From time to time our users complain about write latencies inside their VMs. I would like to be able to see when the cluster is overloaded or when the write latency is bad. What did I try so far: 1) fio inside the KVM virtual machine: fio --ioengine=libaio --direct=1 --rw=write --numjobs=1 --bs=1M --iodepth=16 --size=5G --name=/var/tmp/fio-test [...] write: IOPS=63, BW=63.3MiB/s (66.4MB/s)(5120MiB/80863msec); 0 zone resets - I am usually getting about 60 to 150 IOPS for 1MB writes 2) PostgreSQL from the KVM virtual machine, running many tiny INSERTs as separate transactions for about 10 seconds. This is where I clearly see latency spikes: Wed Dec 18 09:20:21 PM CET 2024 406.062 txn/s Wed Dec 18 09:25:21 PM CET 2024 318.974 txn/s Wed Dec 18 09:30:21 PM CET 2024 285.591 txn/s Wed Dec 18 09:35:21 PM CET 2024 191.804 txn/s Wed Dec 18 09:40:22 PM CET 2024 246.679 txn/s Wed Dec 18 09:45:22 PM CET 2024 201.005 txn/s Wed Dec 18 09:50:22 PM CET 2024 153.206 txn/s Wed Dec 18 09:55:22 PM CET 2024 124.546 txn/s Wed Dec 18 10:00:23 PM CET 2024 33.094 txn/s Wed Dec 18 10:05:23 PM CET 2024 82.659 txn/s Wed Dec 18 10:10:23 PM CET 2024 292.544 txn/s Wed Dec 18 10:15:24 PM CET 2024 453.366 txn/s The drawback of both fio and postgresql benchmark is that I am unnecessarily loading the cluster with additional work, just to measure latency. And I am not covering the whole cluster, just the OSDs on which that VM happens to have its own data. 3) ceph osd perf I don't see any single obviously overloaded OSD here, but the latencies vary nevertheless. Here are statistics computed across all OSDs from the "ceph osd perf" output: Fri Jan 3 10:12:41 AM CET 2025 average 9 9 median 5 5 3rd-q 13 13 max 70 70 Fri Jan 3 10:13:42 AM CET 2025 average 5 5 median 3 3 3rd-q 10 10 max 31 31 Fri Jan 3 10:14:42 AM CET 2025 average 3 3 median 2 2 3rd-q 4 4 max 19 19 Fri Jan 3 10:15:42 AM CET 2025 average 5 5 median 1 1 3rd-q 3 3 max 63 63 However, I am not sure what exactly these numbers actually mean - what timespan do they cover? I would like to have something like "in the last 5 minutes, 99 % of all writes committed under XXX ms". Can Ceph tell me that? What else apart from buying faster hardware can I try in order to improve the write latency for QEMU/KVM-based VMs with RBD images? Thanks for any hints. -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

Ceph Tentacle release timeline — when?
by Gregory Farnum 07 Feb '25

07 Feb '25

Hi all, We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?) * Unfortunately, due to issues in the sepia lab and with testing infrastructure, we had big slips on Reef (August), and we tried to get Squid back on track but it ended up slipping farther (September). * The sepia lab needs to physically move this year out of the current Red Hat community hosting area in to a new lab (hosted by IBM). We're not sure how disruptive this will be, but it will definitely require changes as our current tests are somewhat dependent on the shape of the underlying hardware, which is changing. * There are not many major features available yet. Major upcoming features include the fast EC code, userspace and management support for cephfs encryption, and RBD mirroring groups. Different parts of these are coming throughout the calendar year but we don't have firm timetables yet. We have basically discarded attempting to do an early spring release, since there doesn't seem to be much point. The obvious options are 1) Do a fall release, targeting 12 months after Squid (so, late September, with a freeze in early July). 2) Wait until early spring 2026, with a freeze in January. (This doesn't preclude other options as we gather more input, but we are starting with these for discussion.) We are soliciting feedback on the impact these options will have on our various downstreams and direct upstream consumers. The CSC's *early and tentative* preference is for spring 2026, for the following reasons: 1) the sepia lab move is going to involve an unknown amount of development, and we are gunshy after issues the last couple years. 2) While there is feature work in development, we are unsure how much of interest will be both ready to freeze by mid-summer, and not suitable for backport to our stable releases. 3) Fall releases seem awkward with many organizations wanting to freeze infrastructure over the holidays. 4) While the data is conflicting, it seems like deployments are lagging more on major-version upgrades and so a longer release cycle may make future updates easier for everyone. If the Ceph release schedule has a large impact on your organization or planning, please let us know your thoughts and forward to your downstreams. :) Thanks! -The Ceph Steering Committee, by way of Greg

13 13

v19.2.1 Squid released
by Yuri Weinstein 06 Feb '25

06 Feb '25

We're happy to announce the 1st backport release in the Squid series. https://ceph.io/en/news/blog/2025/v19-2-1-squid-released/ Notable Changes --------------- * CephFS: The command `fs subvolume create` now allows tagging subvolumes by supplying the option `--earmark` with a unique identifier needed for NFS or SMB services. The earmark string for a subvolume is empty by default. To remove an already present earmark, an empty string can be assigned to it. Additionally, the commands `ceph fs subvolume earmark set`, `ceph fs subvolume earmark get`, and `ceph fs subvolume earmark rm` have been added to set, get and remove earmark from a given subvolume. * CephFS: Expanded removexattr support for CephFS virtual extended attributes. Previously one had to use setxattr to restore the default in order to "remove". You may now properly use removexattr to remove. You can also now remove layout on the root inode, which then will restore the layout to the default. * RADOS: A performance bottleneck in the balancer mgr module has been fixed. Related Tracker: https://tracker.ceph.com/issues/68657 * RADOS: Based on tests performed at scale on an HDD-based Ceph cluster, it was found that scheduling with mClock was not optimal with multiple OSD shards. For example, in the test cluster with multiple OSD node failures, the client throughput was found to be inconsistent across test runs coupled with multiple reported slow requests. However, the same test with a single OSD shard and with multiple worker threads yielded significantly better results in terms of consistency of client and recovery throughput across multiple test runs. Therefore, as an interim measure until the issue with multiple OSD shards (or multiple mClock queues per OSD) is investigated and fixed, the following change to the default HDD OSD shard configuration is made: - `osd_op_num_shards_hdd = 1` (was 5) - `osd_op_num_threads_per_shard_hdd = 5` (was 1) For more details, see https://tracker.ceph.com/issues/66289. * mgr/REST: The REST manager module will trim requests based on the 'max_requests' option. Without this feature, and in the absence of manual deletion of old requests, the accumulation of requests in the array can lead to Out Of Memory (OOM) issues, resulting in the Manager crashing. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 58a7fab8be0a062d730ad7da874972fd3fba59fb

2 1

RGW issue, lost bucket metadata ?
by Cyril Duval 06 Feb '25

06 Feb '25

Hello folks ! This is my first time posting here, sorry in advance if I miss some of the etiquette. I'm facing some issues with RGW and especially the radosgw-admin tool, on what appears to be lost bucket metadata, or maybe corruption of some elements of the buckets. I have opened a ticket on https://tracker.ceph.com/issues/69818 where I put more details, but in summary I have the `failure: (2) No such file or directory:` kind of error on most radosgw-admin commands interacting with these buggy buckets. Lifecycle policies seem to be affected too, as I have a `lifecycle: RGWLC::process() failed to get obj head lc.5, ret=-5` error. I'm running Ceph 18.2.2 there. Is it something familiar to anyone ? Or does anyone know where I can have a look further to better understand the issue and solve it ? I tried giving a shot at the cpp code, but unfortunately I'm no expert there, I'm lost. Thanks in advance, -- Cyril Duval cyril.duval(a)diabolocom.com

1 0

Best way to add back a host after removing offline - cephadm
by Kirby Haze 06 Feb '25

06 Feb '25

I have a 3 node test 17.2.7 cluster, and I decided to power one of the hosts down which contained a mon, 6 osds and a standby mgr. Then I used the `ceph orch host rm <host> --force --rm` to remove the host (after powering it down). All of this looks expected except the logs after removing the host. These pools have size 3 min_size 2. Actually I'm unsure what would happen if I were even to successfully add back the host as those objects would be degraded. Besides the error from cephadm, what is ceph thinking when I pulled the plug then removed that host? Am I guaranteed some type of recovery if I do successfully add back that host? ---- Before removing host root@ceph-test-2:/# ceph -s cluster: id: fca870d8-e431-11ef-8000-bc2411363b7d health: HEALTH_OK services: mon: 3 daemons, quorum ceph-test-2,ceph-test-3,ceph-test-4 (age 29m) mgr: ceph-test-2.vbjhdq(active, since 31m), standbys: ceph-test-4.jjubsa osd: 18 osds: 18 up (since 22m), 18 in (since 28m) rgw: 1 daemon active (1 hosts, 1 zones) data: pools: 5 pools, 129 pgs objects: 223 objects, 581 KiB usage: 5.2 GiB used, 8.8 TiB / 8.8 TiB avail pgs: 129 active+clean root@ceph-test-2:/# ceph orch host ls HOST ADDR LABELS STATUS ceph-test-2 10.0.0.52 _admin,rgw ceph-test-3 10.0.0.53 ceph-test-4 10.0.0.54 ------------- After removing host # ceph orch host rm ceph-test-4 --offline --force Removed offline host 'ceph-test-4' # ceph orch ps mon.ceph-test-4 ceph-test-4 stopped osd.1 ceph-test-4 error osd.7 ceph-test-4 error .... .... # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.85956 root default -5 2.92978 host ceph-test-2 2 hdd 0.48830 osd.2 up 1.00000 1.00000 4 hdd 0.48830 osd.4 up 1.00000 1.00000 8 hdd 0.48830 osd.8 up 1.00000 1.00000 11 hdd 0.48830 osd.11 up 1.00000 1.00000 14 hdd 0.48830 osd.14 up 1.00000 1.00000 16 hdd 0.48830 osd.16 up 1.00000 1.00000 -3 2.92978 host ceph-test-3 0 hdd 0.48830 osd.0 up 1.00000 1.00000 3 hdd 0.48830 osd.3 up 1.00000 1.00000 6 hdd 0.48830 osd.6 up 1.00000 1.00000 9 hdd 0.48830 osd.9 up 1.00000 1.00000 12 hdd 0.48830 osd.12 up 1.00000 1.00000 15 hdd 0.48830 osd.15 up 1.00000 1.00000 root@ceph-test-2:/rootfs/root# ceph -s cluster: id: fca870d8-e431-11ef-8000-bc2411363b7d health: HEALTH_WARN 6 failed cephadm daemon(s) Degraded data redundancy: 145/669 objects degraded (21.674%), 24 pgs degraded, 71 pgs undersized services: mon: 2 daemons, quorum ceph-test-2,ceph-test-3 (age 50m) mgr: ceph-test-2.vbjhdq(active, since 2h), standbys: ceph-test-3.wzmioq osd: 12 osds: 12 up (since 51m), 12 in (since 2h); 58 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones) data: pools: 5 pools, 129 pgs objects: 223 objects, 581 KiB usage: 3.5 GiB used, 5.9 TiB / 5.9 TiB avail pgs: 145/669 objects degraded (21.674%) 75/669 objects misplaced (11.211%) 54 active+clean+remapped 47 active+undersized 24 active+undersized+degraded 4 active+clean progress: Global Recovery Event (50m) [================............] (remaining: 37m) The cephadm logs show this as well 2025-02-06T06:27:46.936+0000 7f85026a4700 -1 log_channel(cephadm) log [ERR] : auth get failed: failed to find osd.7 in keyring retval: -2 Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1002, in _check_daemons self.mgr._daemon_action(daemon_spec, action=action) File "/usr/share/ceph/mgr/cephadm/module.py", line 2136, in _daemon_action daemon_spec) File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 494, in generate_config extra_ceph_config=daemon_spec.ceph_conf) File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 520, in get_config_and_keyring 'entity': entity, File "/usr/share/ceph/mgr/mgr_module.py", line 1593, in check_mon_command raise MonCommandFailed(f'{cmd_dict["prefix"]} failed: {r.stderr} retval: {r.retval}') mgr_module.MonCommandFailed: auth get failed: failed to find osd.7 in keyring retval: -2

2 1

EC pool - Profile change
by Devender Singh 06 Feb '25

06 Feb '25

Hello all Is there a way to change ec pool profile from host to osd? Regrds Dev

5 7

Backfills Not Increasing.
by Devender Singh 05 Feb '25

05 Feb '25

Hello all I tried disabling osd_mclock_override_recovery_settings to true and increased backfill from 1 to 15 after disabling scrub and deep-scrub. There was no change I started from 1 then 3 then 5 then 7 then 10 then 13 and 15. But no change. So it a default behaviour or bug? # ceph config get osd osd_mclock_override_recovery_settings true # ceph config get osd osd_max_backfills 15 data: volumes: 2/2 healthy pools: 20 pools, 4065 pgs objects: 21.55M objects, 80 TiB usage: 209 TiB used, 283 TiB / 493 TiB avail pgs: 3912380/77920842 objects misplaced (5.021%) 3873 active+clean 168 active+remapped+backfill_wait 24 active+remapped+backfilling Regards Dev

1 0

CephFS subdatapool in practice?
by Otto Richter (Codeberg e.V.) 05 Feb '25

05 Feb '25

Hello everyone, I am maintaining a cost-optimized Ceph cluster at Codeberg.org, that is serving a lot of small random IO (Git operations). Currently, the data is stored in a single CephFS pool on mixed SSD and HDD cluster, where the HDDs have a primary affinity of 0 to shift the reads to the SSDs. In a quest to further optimize our setup, I discovered the subdatapool feature described at https://github.com/TheJJ/ceph-cheatsheet#subdatapool (assigning multiple pools to a filesystem and using setfattr to allow writes in a certain folder to a separate pool). It feels like this would allow us to cut cost by moving certain directories with large files to a cheaper pool than that used for Git operations easily, without doing data migrations such as splitting data into multiple filesystems. Is anyone using this feature in production and can comment on it? Are there best practices regarding the usage of this feature? Any caveats to be aware of, such as overhead, latency, long-term maintainability? Since certain "interesting" features have been deprecated in the past, such as inline_data, I thought it would be better to ask here for feedback first. Thank you for the comments and have a nice day! Kind Regards Otto -- https://codeberg.org Codeberg e.V. – Arminiusstraße 2-4 – 10551 Berlin – Germany Registered at registration court Amtsgericht Charlottenburg VR36929.

3 2

Re: cephadm: Move DB/WAL from HDD to SSD
by Alan Murrell 05 Feb '25

05 Feb '25

Hello, I ran the commands from the "regular" cephadm shell ('cephadm shell --') but did not specify an OSD (I wasn't aware you could do that) When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting. I can give it a try and let you know how it goes, and if necessary go from there. Sent from my mobile device. Please excuse brevity and ttpos.

2 19

Merge DB/WAL back to the main device?
by Jan Kasprzak 05 Feb '25

05 Feb '25

Hi all, while reading a sibling thread about moving DB/WAL to a separate device, I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage? What I am trying to do: My OSD nodes are 1U boxes with 4 drive bays, two of which support NVMe. They have two small-ish NVMe drives and two large HDDs. On NVMe, there is a partition for an OS (configured as RAID-1 across both NVMe drives), and the rest of each NVMe is used as metadata for one of the HDD-based OSDs. So I have two OSDs per node. Now I am considering replacing those small NVMes with much larger ones, and use part of them for an OS as before, part of them as metadata for the HDD-based OSDs as before, and the rest as new NVMe-only OSDs. Can this be done without tearing the original HDD+NVMe-based OSDs down and recreating them again? Being able to remove a metadata device from the OSD would help in this case. Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

4 8

Spec file question: --dry-run not showing anything would be applied?
by Alan Murrell 05 Feb '25

05 Feb '25

Sorry, I just got on this list and am asking a number of questions… ☹ It was suggested in one of my other threads to create and apply a spec file that would have cephadm put the DB for rotational drives onto SSD when it creates the OSD on those drives. I came up with he following spec file: --- START --- service_type: osd service_id: osd_spec_hdd placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 model: 'Samsung SSD 990 PRO 1TB' limit: 1 --- service_type: osd service_id: osd_spec_ssd placement: host_pattern: '*' data_devices: model: 'KINGSTON SEDC1500M1920G' --- END --- But when I run ‘ceph orch apply osd -i ./osd_spec.yml --dry-run’, I get the following output: --- START --- WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any of these conditions change, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+-------------+ +---------+------+--------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+ --- END --- Which seems to indicate that nothing would be applied, but I am not sure if that is because my spec file is wrong or if it is because my rotational drives already have OSDs (and I have migrated their DB/WAL to the SSDs)?

3 2

Orphaned rbd_data Objects
by Stolte, Felix 05 Feb '25

05 Feb '25

Hi guys, we have a rbd pool we used for images exported via ceph-iscsi on a 17.2.7 cluster. The pool uses 10 times the diskspace i would suppose it should and after investigating we noticed a lot of rbd_data Objects which images are no longer present. I assume that the original images were deleted using the gwcli but not all Objects have been removed properly. What would be the best/most secure way to get rid of these orphaned objects and reclaim the diskspace? Best regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

4 11

ceph-osd segmentation fault on arm64 quay.io/ceph/ceph:v18.2.4
by Adam Prycki 05 Feb '25

05 Feb '25

Hi, today we attempted to upgrade our ceph cluster from 18.2.0 to 18.2.4. OSDs updated to 18.2.4 couldn't start. I've discovered that ceph-osd in segfaults. It happens always on our setup. I've tested it on ubunut 20.04 with docker and ubuntu 22.04 with podman Example: root@ktsh02:~# podman run --rm -it quay.io/ceph/ceph:v18.2.4 /bin/bash [root@20af860446bd /]# ceph-osd Segmentation fault (core dumped) 19.2.0 container seems to be fine. Do we have other users with arm64 ceph deployments? Do 18.2.4 containers work on your machines? Best regards Adam Prycki

2 1

Spec file: Possible typo in example:
by Alan Murrell 05 Feb '25

05 Feb '25

Hello, I am not sure if this is the right list to put this, but I was just looking over the documentation for the Service Spec file: https://docs.ceph.com/en/octopus/cephadm/drivegroups/ and under one of the "Advanced Cases" example, I believe there *might* be an error? Here is the example (for 20 HDDs, 12 SSDs, 2 NVMEs): --- START --- service_type: osd service_id: osd_spec_hdd placement: host_pattern: '*' data_devices: rotational: 0 db_devices: model: MC-55-44-XZ limit: 2 (db_slots is actually to be favoured here, but it's not implemented yet) --- service_type: osd service_id: osd_spec_ssd placement: host_pattern: '*' data_devices: model: MC-55-44-XZ db_devices: vendor: VendorC --- END --- For the 'data_devices' under the 'osd_spec_hdd' layout, should 'rotational' not be '1' instead of '0'? The earlier "Simple" example (20 HDDs, 2 SSDs) has 'rotational' as '1'.

2 4

Ceph User Stories Survey: Final Call
by Laura Flores 04 Feb '25

04 Feb '25

Hi all, Many thanks to everyone who has already taken the "Ceph User Stories" survey, the results of which will be discussed in one of the upcoming User + Dev Monthly Meetups. See https://www.meetup.com/ceph-user-group/events/ for updates on upcoming meetups. The survey will be closed this week on Friday, February 7th at 5pm CST. Until then, you may still take the survey by following this link: https://docs.google.com/forms/d/e/1FAIpQLSe66NedXh4gHLgk9G45eqP5V2wHlz4IKqR… Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Re: cephadm: Move DB/WAL from HDD to SSD
by Alan Murrell 04 Feb '25

04 Feb '25

One last question/clarification: when it becomes necessary to replace that SSD, I assume the process of moving the DB/WAL to the new SSD is the same, except in the 'ceph-volume migrate' command I would specify '--from db' now, yes? Sent from my mobile device. Please excuse brevity and ttpos. On Feb 4, 2025 00:18, Eugen Block <eblock(a)nde.ag> wrote: *** This is an EXTERNAL email. Please exercise caution. DO NOT open attachments or click links from unknown senders or unexpected email. *** I'm glad it worked. I thought those steps had already been added to the docs (cephadm specific), but I couldn't find them either. I'll ping Zac about it. Zitat von Alan Murrell <Alan(a)t-net.ca>: > OK, I think I am good now. I have completed the rest of the steps > (exit shell, start the osd.10 daemon back up again, waited for it to > be marked as "Up") and then I ran: > > ceph osd metadata 10 > > and all the bluestore_db items are pointing to the SSD now. > > OK, so now to note these steps/commands in to "Ceph Notes" and the > repeat the process for the other HDDs on this node and my other four 😊 > > Thank you *so* much for your help (and patience!) > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1