March 2025 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

Announcing go-ceph v0.32.0
by Sven Anderson 12 Aug '25

12 Aug '25

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.32.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

Squid 19.2.1 dashboard javascript error
by Chris Palmer 09 Aug '25

09 Aug '25

Firstly thank you so much for the 19.2.1 release. Initial testing suggests that the blockers that we had in 19.2.0 have all been resolved, so we are proceeding with further testing. We have noticed one small problem in 19.2.1 that was not present in 19.2.0 though. We use the older-style dashboard (mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false). The problem happens on the Dashboard screen when health changes to WARN. If you click on WARN you get a small empty dropdown instead of the list of warnings. A javascript error is logged, and using browser inspection there is the additional bit of info that it happens in polyfill: 2025-02-07T15:59:00.970+0000 7f1d63877640 0 [dashboard ERROR frontend.error] (https://<redacted>:8443/#/dashboard): NG0901 Error: NG0901 at d.find (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3342365) at le.ngDoCheck (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3173112) at Qe (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225586) at bt (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225341) at cs (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225051) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259043) at jf (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3266563) at S1 (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259790) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259801) at fg (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3267248) Also, after this happens, no dropdowns work again until the page is forcibly refreshed. Environment is RPM install on Centos 9 Stream. I've created issue [0]. Thanks, Chris [0] https://tracker.ceph.com/issues/69867 <https://tracker.ceph.com/issues/69867?next_issue_id=69865&prev_issue_id=90>

4 9

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Adding OSD nodes
by Sinan Polat 21 Jun '25

21 Jun '25

Hello, I am currently managing a Ceph cluster that consists of 3 racks, each with 4 OSD nodes. Each node contains 24 OSDs. I plan to add three new nodes, one to each rack, to help alleviate the high OSD utilization. The current highest OSD utilization is 85%. I am concerned about the possibility of any OSD reaching the osd_full_ratio threshold during the rebalancing process. This would cause the cluster to enter a read-only state, which I want to avoid at all costs. I am planning to execute the following commands: ceph orch host add new-node-1 ceph orch host add new-node-2 ceph orch host add new-node-3 ceph osd crush move new-node-1 rack=rack-1 ceph osd crush move new-node-2 rack=rack-2 ceph osd crush move new-node-3 rack=rack-3 ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 1 ceph config set osd osd_recovery_sleep 0.1 ceph orch apply osd --all-available-devices Before proceeding, I would like to ask if the above steps are safe to execute in a cluster with such high utilization. My main concern is whether the rebalancing could cause any OSD to exceed the osd_full_ratio and result in unexpected failures. Any insights or advice on how to safely add these nodes without impacting cluster stability would be greatly appreciated. Thanks!

7 7

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

CephFS Snapshot Mirroring
by Vladimir Cvetkovic 16 Apr '25

16 Apr '25

Hi everyone, We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync the snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup? Thanks!

4 5

constant increase in osdmap epoch
by Frank Schilder 15 Apr '25

15 Apr '25

Hi all, we observe a problem that has been reported before, but I can't find a resolution. This is related to an earlier thread "failed to load OSD map for epoch 2898146, got 0 bytes" (https://www.spinics.net/lists/ceph-users/msg84485.html) We run an octopus latest cluster and observe a constant increase in osdmap epoch every few seconds. There is no change in the contents between two successive epochs: # diff map.3075085.txt map.3075086.txt 1c1 < epoch 3075085 --- > epoch 3075086 4c4 < modified 2024-11-18T15:38:45.512100+0100 --- > modified 2024-11-18T15:38:47.858092+0100 This is exactly what others reported too, for example, "steady increasing of osd map epoch since octopus" (https://www.spinics.net/lists/ceph-users/msg69443.html) Its a real problem since it dramatically shortens the time window an OSD can be down before its latest OSD map is purged from the cluster. This, in turn, leads to serious follow-up problems with OSD restart as reported in the thread I'm referring to at the beginning. Related to that I also see the mgrs increasing the pgmap version constantly every 2 seconds. However, I believe this is intentional. I don't see this redundant pgp_num_actual setting by the mgrs reported here: https://tracker.ceph.com/issues/51433 . I can't find a resolution anywhere. Any help would be very much appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 7

Ceph Day Silicon Valley 2025 - Call for Proposals Now Open!
by Neha Ojha 14 Apr '25

14 Apr '25

Dear Ceph Community, We are very excited to announce that Ceph Day is returning to Silicon Valley after seven years! On March 25, 2025, we will be hosting Ceph Day at IBM Almaden Research Center in San Jose, CA. We are now accepting talk proposals! For more details, visit https://ceph.io/en/community/events/2025/ceph-days-almaden/. We look forward to bringing the Ceph community together once again! Cheers, Neha

2 2

UI for Object Gateway S3 ?
by Anthony Fecarotta 14 Apr '25

14 Apr '25

Hi everyone, Thanks to a member of this wonderful mailing list, I successfully, and easily deployed the RADOS Gateway S3 API on each of the nodes in one of our five node Proxmox clusters. I can see managing this at scale from the terminal being a little tricky, but more so inconvenient. Are there any options (closed or open-source) that offer a way GUI or TUI to manage the S3 Buckets? I was able to find this GitHub Gist ( https://gist.github.com/e02c6081b515881aeb97b26016d0c658.git) that lets Minio act as a GUI for "Ceph backed S3," but it hasn't been updated in nine months and it seems to be intended strictly for Kubernetes. Any of your recommendations are appreciated. I started building a TUI with GO, to solve this problem, but that will take some time to finish 🙃 Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

3 5

Experience with 100G Ceph in Proxmox
by Giovanna Ratini 11 Apr '25

11 Apr '25

Hello everyone, We are running Ceph in Proxmox with a 10G network. Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link <https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#p…> For this reason, we are considering purchasing a 100G switch for our servers. This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended. Does anyone have experience with 100G Ceph in Proxmox? Is upgrading to 100G a good idea, or will I have 60G sitting idle? Thanks in advance! Gio

13 32

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

ceph.log using seconds since epoch instead of date/time stamp
by Stillwell, Bryan 07 Apr '25

07 Apr '25

Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps? Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok # ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1 This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04. I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not): https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.… Thanks, Bryan

3 4

ceph-ansible LARGE OMAP in RGW pool
by Danish Khan 04 Apr '25

04 Apr '25

Dear All, My ceph cluster is giving Large OMAP warning from approx 2-3 Months. I tried a few things like : *Deep scrub of PGs* *Compact OSDs* *Trim log* But these didn't work out. I guess the main issue is that 4 shards in replication site are always recovering from 2-3 months. Any suggestions are highly appreciated. Sync status: root@drhost1:~# radosgw-admin sync status realm e259e0a92 (object-storage) zonegroup 7a8606d2 (staas) zone c8022ad1 (repstaas) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 2072ee2d32 (masterstaas) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [7,90,100] oldest incremental change not applied: 2025-03-12T13:14:10.268469+0530 [7] 4 shards are recovering recovering shards: [31,41,55,80] Master site: 1. *root@master1:~# for obj in $(rados ls -p masterstaas.rgw.log); do echo "$(rados listomapkeys -p masterstaas.rgw.log $obj | wc -l) $obj";done | sort -nr | head -10* 1225387 data_log.91 1225065 data_log.86 1224662 data_log.87 1224448 data_log.92 1224018 data_log.89 1222156 data_log.93 1201489 data_log.83 1174125 data_log.90 363498 data_log.84 258709 data_log.6 2. *root@master1:~# for obj in data_log.91 data_log.86 data_log.87 data_log.92 data_log.89 data_log.93 data_log.83 data_log.90; do rados stat -p masterstaas.rgw.log $obj; done* masterstaas.rgw.log/data_log.91 mtime 2025-02-24T15:09:25.000000+0530, size 0 masterstaas.rgw.log/data_log.86 mtime 2025-02-24T15:01:25.000000+0530, size 0 masterstaas.rgw.log/data_log.87 mtime 2025-02-24T15:02:25.000000+0530, size 0 masterstaas.rgw.log/data_log.92 mtime 2025-02-24T15:11:01.000000+0530, size 0 masterstaas.rgw.log/data_log.89 mtime 2025-02-24T14:54:55.000000+0530, size 0 masterstaas.rgw.log/data_log.93 mtime 2025-02-24T14:53:25.000000+0530, size 0 masterstaas.rgw.log/data_log.83 mtime 2025-02-24T14:16:21.000000+0530, size 0 masterstaas.rgw.log/data_log.90 mtime 2025-02-24T15:05:25.000000+0530, size 0 *3. ceph cluster log :* 2025-02-22T04:18:27.324886+0530 osd.173 (osd.173) 19 : cluster [WRN] Large omap object found. Object: 124:b2ddf551:::data_log.93:head PG: 124.8aafbb4d (124.d) Key count: 1218170 Size (bytes): 297085860 2025-02-22T04:18:28.735886+0530 osd.65 (osd.65) 308 : cluster [WRN] Large omap object found. Object: 124:f2081d70:::data_log.92:head PG: 124.eb8104f (124.f) Key count: 1220420 Size (bytes): 295240028 2025-02-22T04:18:30.668884+0530 mon.master1 (mon.0) 7974038 : cluster [WRN] Health check update: 3 large omap objects (LARGE_OMAP_OBJECTS) 2025-02-22T04:18:31.127585+0530 osd.18 (osd.18) 224 : cluster [WRN] Large omap object found. Object: 124:d1061236:::data_log.86:head PG: 124.6c48608b (124.b) Key count: 1221047 Size (bytes): 295398274 2025-02-22T04:18:33.189561+0530 osd.37 (osd.37) 32665 : cluster [WRN] Large omap object found. Object: 124:9a2e04b7:::data_log.87:head PG: 124.ed207459 (124.19) Key count: 1220584 Size (bytes): 295290366 2025-02-22T04:18:35.007117+0530 osd.77 (osd.77) 135 : cluster [WRN] Large omap object found. Object: 124:6b9e929a:::data_log.89:head PG: 124.594979d6 (124.16) Key count: 1219913 Size (bytes): 295127816 2025-02-22T04:18:36.189141+0530 mon.master1 (mon.0) 7974039 : cluster [WRN] Health check update: 5 large omap objects (LARGE_OMAP_OBJECTS) 2025-02-22T04:18:36.340247+0530 osd.112 (osd.112) 259 : cluster [WRN] Large omap object found. Object: 124:0958bece:::data_log.83:head PG: 124.737d1a90 (124.10) Key count: 1200406 Size (bytes): 290280292 2025-02-22T04:18:38.523766+0530 osd.73 (osd.73) 1064 : cluster [WRN] Large omap object found. Object: 124:fddd971f:::data_log.91:head PG: 124.f8e9bbbf (124.3f) Key count: 1221183 Size (bytes): 295425320 2025-02-22T04:18:42.619926+0530 osd.92 (osd.92) 285 : cluster [WRN] Large omap object found. Object: 124:7dc404fa:::data_log.90:head PG: 124.5f2023be (124.3e) Key count: 1169895 Size (bytes): 283025576 2025-02-22T04:18:44.242655+0530 mon.master1 (mon.0) 7974043 : cluster [WRN] Health check update: 8 large omap objects (LARGE_OMAP_OBJECTS) Replica site: 1. *for obj in $(rados ls -p repstaas.rgw.log); do echo "$(rados listomapkeys -p repstaas.rgw.log $obj | wc -l) $obj";done | sort -nr | head -10* 432850 data_log.91 432384 data_log.87 432323 data_log.93 431783 data_log.86 431510 data_log.92 427959 data_log.89 414522 data_log.90 407571 data_log.83 151015 data_log.84 109790 data_log.4 2. *ceph cluster log:* grep -ir "Large omap object found" /var/log/ceph/ /var/log/ceph/ceph-mon.drhost1.log:2025-03-12T14:49:59.997+0530 7fc4ad544700 0 log_channel(cluster) log [WRN] : Search the cluster log for 'Large omap object found' for more details. /var/log/ceph/ceph.log:2025-03-12T14:49:02.078108+0530 osd.10 (osd.10) 21 : cluster [WRN] Large omap object found. Object: 6:b2ddf551:::data_log.93:head PG: 6.8aafbb4d (6.d) Key count: 432323 Size (bytes): 105505884 /var/log/ceph/ceph.log:2025-03-12T14:49:02.389288+0530 osd.48 (osd.48) 37 : cluster [WRN] Large omap object found. Object: 6:d1061236:::data_log.86:head PG: 6.6c48608b (6.b) Key count: 431782 Size (bytes): 104564674 /var/log/ceph/ceph.log:2025-03-12T14:49:07.166954+0530 osd.24 (osd.24) 24 : cluster [WRN] Large omap object found. Object: 6:0958bece:::data_log.83:head PG: 6.737d1a90 (6.10) Key count: 407571 Size (bytes): 98635522 /var/log/ceph/ceph.log:2025-03-12T14:49:09.100110+0530 osd.63 (osd.63) 5 : cluster [WRN] Large omap object found. Object: 6:9a2e04b7:::data_log.87:head PG: 6.ed207459 (6.19) Key count: 432384 Size (bytes): 104712350 /var/log/ceph/ceph.log:2025-03-12T14:49:08.703760+0530 osd.59 (osd.59) 11 : cluster [WRN] Large omap object found. Object: 6:6b9e929a:::data_log.89:head PG: 6.594979d6 (6.16) Key count: 427959 Size (bytes): 103773777 /var/log/ceph/ceph.log:2025-03-12T14:49:11.126132+0530 osd.40 (osd.40) 24 : cluster [WRN] Large omap object found. Object: 6:f2081d70:::data_log.92:head PG: 6.eb8104f (6.f) Key count: 431508 Size (bytes): 104520406 /var/log/ceph/ceph.log:2025-03-12T14:49:13.799473+0530 osd.43 (osd.43) 61 : cluster [WRN] Large omap object found. Object: 6:fddd971f:::data_log.91:head PG: 6.f8e9bbbf (6.1f) Key count: 432850 Size (bytes): 104418869 /var/log/ceph/ceph.log:2025-03-12T14:49:14.398480+0530 osd.3 (osd.3) 55 : cluster [WRN] Large omap object found. Object: 6:7dc404fa:::data_log.90:head PG: 6.5f2023be (6.1e) Key count: 414521 Size (bytes): 100396561 /var/log/ceph/ceph.log:2025-03-12T14:50:00.000484+0530 mon.drhost1 (mon.0) 207423 : cluster [WRN] Search the cluster log for 'Large omap object found' for more details. Regards, Danish

2 10

reef 18.2.5 QE validation status
by Yuri Weinstein 02 Apr '25

02 Apr '25

Details of this release are summarized here: https://tracker.ceph.com/issues/70563#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke - Laura approved? rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? rgw - Adam E approved? fs - Venky is fixing QA suite, will need to be added and rerun orch - Adam King approved? rbd - Ilya approved? krbd - Ilya approved? upgrade-clients:client-upgrade-octopus-reef-reef - Ilya please take a look. upgrade/pacific-x (reef) - can this be deprecated? Josh? Neha? upgrade/quincy-x (reef) - Laura, Prashant please take a look. ceph-volume - Guillaume is fixing it. TIA

12 22

Ceph Tentacle release - dev freeze timeline
by Yaarit Hatuka 02 Apr '25

02 Apr '25

Hi everyone, In previous discussions, the Ceph Steering Committee tentatively agreed on a Tentacle dev freeze around the end of March or mid-April. We would like to revisit this and check in with all the tech leads to assess the readiness level and ensure we're aligned on the timeline. Please provide your input on the current status and any potential concerns you may have regarding the freeze. Thanks, on behalf of the CSC.

1 1

reshard stale-instances
by Richard Bade 01 Apr '25

01 Apr '25

Hi Everyone, We've just completed undoing multisite to bring us back down to a single zone/site. Now I'm working through cleaning things up. Heads up, I'll probably have a few more questions over the next couple of weeks :) Firstly though, regarding "radosgw-admin reshard stale-instances list". This brings back a decent list mostly of some test buckets that were being deleted and recreated by an automated system. This has left over 150k stale instances for each of three buckets. I know these ones will be fine to delete as even if it messes up the bucket I can delete and re-create them. So my question is, how do I clean these up without using the big hammer "radosgw-admin reshard stale-instances delete" to remove all of them which would include some production buckets that I'm not quite sure how to verify are stale? Another thought that I had is maybe if I were to reshard all the buckets up to the next prime now that we're on single zone would that guarantee that all the stale instances are actually stale? Any thoughts and tips would be much appreciated. Thanks, Rich

2 3

Prometheus anomaly in Reef
by Tim Holloway 01 Apr '25

01 Apr '25

I finally got brave and migrated from Pacific to Reef, did some banging and hammering and for the first time in a long time got a complete "HEALTH OK" status. However, the dashboard is still not happy. It cannot contact the Prometheus API on port 9095. I have redeployed Prometheus multiple times without result. I'm pretty sure that at one time there were no less than 3 different Prometheus containers running on one of the configured Prometheus servers, but now all I can get is the node-exporter. Worse, if I do: ceph orch reconfig prometheus I get: Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls" But if I do: ceph orch ls I get: prometheus ?:9095 0/1 - 116s ceph02.mousetech.com Suggestions?

2 40

How/when the osd_mclock_max_capacity_iops is updated?
by Michel Jouvin 31 Mar '25

31 Mar '25

Hi, Looking at our configuration during a recovery operation (misplaced objects after adding PG in a pool) that I found a bit slow, I saw that ~30 OSD over 500 had an entry for osd_mclock_max_capacity_iops_hdd in the configuration DB (and almost all the SSD). Despite all HDD and SSD being the same models (and the server HW being the same). As this value can have a significant impact on recovery time/speed with mclock, I was wondering if this value is periodically refreshed, in case one measurement was wrong (perturbated by external factors). The documentation is not very clear about this and seems to say that the bench used to estimate the IOPS capacity is run only when the OSD is activated (which may make sense as it is before there is any other activity on it)? I'm also wondering why we observe value like 350 on some OSD as the HW is the same on all servers? Currently, I tend to remove all the osd_mclock_max_capacity_iops_xxx entries from the DB periodically but it's probably not the right approach... Cheers, Michel

3 7

Ceph MDS stuck in reconnect -> rejoin -> failover loop
by Kasper Rasmussen 31 Mar '25

31 Mar '25

Hi Ceph pacific 16.2.15 I have 5 MDS hosts, 4 active (4 FS), and 1 standby. One MDS was restarted today (as part of OS Patching), resulting in a failover. This is usually not an issue but today,it got stuck in a reconnect -> rejoin -> failover loop for the specific FS. A ceph fs status shows that during the time the FS is in state "rejoin" the INOS rise to +50M (usually it is around 10-12M ) The memory on the MDS host is eaten, (MDS cache size is 36GB, but it rises to +140 GB.) Finaly it fails over, and the cycle starts over. We are currently restarting all clients, in an effort to rule out buggy clients. Any help on this issue will be very much appreciated. Thank you

2 2

Lifecycle question
by Szabo, Istvan (Agoda) 31 Mar '25

31 Mar '25

Hi, Is there a way to have hourly or minute based lifecycle rather than a 1 day set on a bucket? When I checked multiple xml rules all of them day based and can't set hour or minute value: Multipart cleaning: <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Incomplete Multipart Uploads</ID> <Prefix/> <Status>Enabled</Status> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> </Rule> </LifecycleConfiguration> Basic retention: <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>5 days retention</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>5</Days> </Expiration> </Rule> </LifecycleConfiguration> Tag based retention: <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>1 day object expiry</ID> <Filter> <Tag> <Key>expiration</Key> <Value>1d</Value> </Tag> </Filter> <Status>Enabled</Status> <Expiration> <Days>1</Days> </Expiration> </Rule> </LifecycleConfiguration> Prefix based retention <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>hermes-1-day-retention</ID> <Filter> <Prefix>her-</Prefix> </Filter> <Status>Enabled</Status> <Expiration> <Days>1</Days> //retention </Expiration> </Rule> </LifecycleConfiguration> ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 5

Deleting a pool with data
by Richard Bade 30 Mar '25

30 Mar '25

Hi Everyone, We're reducing back down from multisite to a single rgw zone. This will mean that some pools will be unused so I'd like to delete them. However there are some objects and data remaining in the pool even though the buckets are all deleted. It's just shadow objects. All the actual data has been deleted. So my question is around performance impact of deleting a pool with data in it. Will ceph handle things nicely or will it try remove all that data at once? I'm on Nautilus 14.2.22 with all bluestore osds on spinning disks with nvme db. Pools are erasure coded k=4 m=2. I'm thinking it might be best to do rados ls and loop through rados rm the objects to control the speed. There is one mailing list thread from 7 years ago basically saying the same but I was wondering if anyone else had any input around this? Thanks, Rich

4 5

Device missing from "ceph device ls"
by Torkil Svensgaard 29 Mar '25

29 Mar '25

Hi 19.2.1 " [root@franky ~]# ceph device ls | grep franky ATA_HGST_HDN726060ALE614_K1GV9P4B franky:sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P franky:sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ franky:sdg osd.580 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA franky:sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA franky:sdf osd.527 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA franky:sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A franky:sdx osd.583 ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN franky:sdp osd.532 5% now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN franky:sdt osd.531 7% now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH franky:sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH franky:sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH franky:sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH franky:sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH franky:sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 franky:sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 franky:sdr osd.530 ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 franky:sdl osd.528 ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 franky:sdaa osd.529 ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ franky:sdd osd.537 HGST_HUS724040ALS640_PBG458BX franky:sdb osd.568 HGST_HUS724040ALS640_PCJ2VXRX franky:sdj osd.578 HGST_HUS724040ALS640_PCJN3LWX franky:sdi osd.539 HGST_HUS724040ALS640_PCKX3UWX franky:sdo osd.564 HGST_HUS724040ALS640_PCKX4KSX franky:sds osd.575 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 franky:nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 0% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 franky:nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 4% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 franky:nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 4% [root@franky ~]# ceph orch device ls | grep franky franky /dev/md127 ssd 221G No 12m ago Has a FileSystem franky /dev/nvme0n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme1n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme2n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme3n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/sda hdd ATA_HGST_HDN726060ALE614_K1GV9P4B 5589G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdaa hdd ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdb hdd HGST_HUS724040ALS640_PBG458BX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdc hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdd hdd ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sde hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdf hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdg hdd ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdh hdd ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdi hdd HGST_HUS724040ALS640_PCJN3LWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdj hdd HGST_HUS724040ALS640_PCJ2VXRX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdk hdd ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdl hdd ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdm hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdn hdd ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdo hdd HGST_HUS724040ALS640_PCKX3UWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdp ssd ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN 186G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdq hdd ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdr hdd ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sds hdd HGST_HUS724040ALS640_PCKX4KSX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdt ssd ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN 447G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdu hdd ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdv hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdw hdd ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2273861 2794G Yes 12m ago franky /dev/sdx hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected " Any suggestions as to why a device (nvme2n1) might be missing from "ceph device ls"? It is being used as db device for 4 OSDs. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 3

OSD creation from service spec fails to check all db_devices for available space
by Torkil Svensgaard 29 Mar '25

29 Mar '25

Hi I am adding HDDs to a replacement server which will fit 34 HDDs and 2 SATA SSDs, and has 4 NVMe devices for DB/WAL. The orchaetrator now fails to create any more OSDs due to: " /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled " It looks to me like 1 of the 4 NVMe devices is indeed too full to fit another DB/WAL partition but ceph-volume is being passed all 4 devices so it should just pick another? Also, it has been creating DB/WAL partitions across all 4 devices up til now, so it's not like it only looks at the first device passed. Suggestions? " 2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected new or changed devices on franky 2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd " " [root@franky ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2.7T 0 disk └─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd--block--76e673e9--d0c0--46bb--8c3c--e666ac5cef3b 253:2 0 2.7T 0 lvm sdb 8:16 0 223.6G 0 disk ├─sdb1 8:17 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdb2 8:18 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdb3 8:19 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdc2 8:34 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdc3 8:35 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdd 8:48 0 1.8T 0 disk └─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block--b7330837--b986--46d7--9e28--57db65945098 253:4 0 1.8T 0 lvm sde 8:64 0 2.7T 0 disk └─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd--block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3 253:6 0 2.7T 0 lvm sdf 8:80 0 1.8T 0 disk └─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd--block--8a0c4a74--fedc--46b3--b2a8--d60fd18a37c1 253:8 0 1.8T 0 lvm sdg 8:96 0 2.7T 0 disk └─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block--c96afbde--9c71--408f--a961--68c6d14a701f 253:12 0 2.7T 0 lvm sdh 8:112 0 16.4T 0 disk └─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd--block--29747e0e--9c71--44e6--b750--93a7878977ee 253:14 0 16.4T 0 lvm sdi 8:128 0 16.4T 0 disk └─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd--block--618f7582--fce0--41f6--aad8--6d0231ef303a 253:16 0 16.4T 0 lvm sdj 8:144 0 1.8T 0 disk └─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd--block--8e9b21b6--cc39--4c7b--b5f8--9e83e33fa146 253:18 0 1.8T 0 lvm sdk 8:160 0 447.1G 0 disk └─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd--block--14f68253--b370--4300--a319--0c39311a34e1 253:10 0 447.1G 0 lvm sdl 8:176 0 186.3G 0 disk └─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd--block--2b475461--a85d--4e7b--a7e2--8ab1c9d14c6e 253:11 0 186.3G 0 lvm sdm 8:192 0 3.6T 0 disk └─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block--f08c7e71--ddf5--4939--8f0c--42396de2210b 253:26 0 3.6T 0 lvm sdn 8:208 0 2.7T 0 disk └─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd--block--40cd9045--587a--4831--95f7--607c019ef862 253:20 0 2.7T 0 lvm sdo 8:224 0 2.7T 0 disk └─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block--c2087805--441c--422a--b12f--de10a75b7e0b 253:22 0 2.7T 0 lvm sdp 8:240 0 3.6T 0 disk └─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block--c0ef5f16--4bcd--4390--abf8--260c5913cb14 253:24 0 3.6T 0 lvm sdq 65:0 0 3.6T 0 disk └─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd--block--8f783ade--6da6--4967--b87c--fb6d1827460f 253:28 0 3.6T 0 lvm sdr 65:16 0 16.4T 0 disk └─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block--a3dbca80--9cf5--4c98--ad9b--b38803230b1f 253:30 0 16.4T 0 lvm sds 65:32 0 3.6T 0 disk └─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd--block--032fc777--69d9--4f39--8fe6--485e0959ce66 253:32 0 3.6T 0 lvm sdt 65:48 0 16.4T 0 disk └─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block--ff12d109--dfaa--4608--a5e2--36ec39623f36 253:34 0 16.4T 0 lvm sdu 65:64 0 3.6T 0 disk └─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block--b3e38c3d--e3c0--49c2--9e96--93f9e4c909d4 253:36 0 3.6T 0 lvm sdv 65:80 0 3.6T 0 disk sdw 65:96 0 5.5T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 16.4T 0 disk nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533 253:13 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm [root@franky ~]# " Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 4

space size issue
by Mihai Ciubancan 29 Mar '25

29 Mar '25

Hello, My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB): sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000 sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B While on the client side I have this: $ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data I don't know where it's gone all the space that was at the beginning. Someone has any hint? Best regards, Mihai

6 16

Ceph Commvault Experience
by DAN ALBRIGHT 28 Mar '25

28 Mar '25

Hi Ceph Community, looking to understand Ceph implementation as an S3 storage pool in support of the Commvault. Please share your experiences and any important tips to consider. Many Thanks! Dan Albright Americas Sales Leader/Specialist - IBM Object Storage (COS & Ceph)- Systems Sales M: (303) 908-5203

2 1

Re: space size issue
by Eugen Block 28 Mar '25

28 Mar '25

You probably have to change the existing crush rule to use OSD as failure domain instead of host. But if you have only one host, why not set up some other file storage with (software) raid or something? Zitat von "mihai.ciubancan" <mihai.ciubancan(a)eli-np.ro>: > The problem is that I have only one host, and the host doesn't have > a RAID controller...I have set as Anthone suggested, but nothing > changed.Mihai > -------- Original message --------From: Eugen Block <eblock(a)nde.ag> > Date: 3/28/25 3:47 PM (GMT+02:00) To: Mihai Ciubancan > <mihai.ciubancan(a)eli-np.ro> Cc: ceph-users(a)ceph.io Subject: > [ceph-users] Re: space size issue You have the autoscaler enabled, > but it stuck changing the pg_num. The default replicated_rule > (which you are using) requires as many hosts as your pool size is, > so in your case 2. (If you value your data, don't use replicated > pools with size 2.)You could make it work with only one host (as > Anthony suggested with osd_crush_chooseleaf_type 0), but you don't > any real resiliency. I recommend to reconsider your setup.Zitat von > Mihai Ciubancan <mihai.ciubancan(a)eli-np.ro>:> Hi Eugen,>> Thanks for > your answer. Please find below the output of the command:>> ceph osd > pool ls detail> pool 1 '.mgr' replicated size 2 min_size 1 > crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 > autoscale_mode on last_change 323 flags > hashpspool,nearfull > stripe_width 0 pg_num_max 32 pg_num_min 1 > application mgr > read_balance_score 12.50> pool 2 'cephfs.cephfs.meta' replicated > size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 16 > pgp_num 1 pgp_num_target 16 > autoscale_mode on last_change 295 > lfor 0/0/54 flags > hashpspool,nearfull stripe_width 0 > pg_autoscale_bias 4 pg_num_min 16 > recovery_priority 5 application > cephfs read_balance_score 12.50> pool 3 'cephfs.cephfs.data' > replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins > pg_num 129 pgp_num 1 pg_num_target 512 > pgp_num_target 512 > autoscale_mode on last_change 326 lfor 0/0/54 > flags > hashpspool,nearfull,bulk stripe_width 0 application cephfs > > read_balance_score 12.50>>> How can I decrese the number of > pg_num?>> Best,> Mihai>>> On 2025-03-28 13:19, Eugen Block wrote:>> > Do you use size 1 for your data? You know that's bad, right? > Please>> share 'ceph osd pool ls detail'.>> Also, it's recommended > to use a power of 2 for pg_num, so you should>> decrease the pg_num > for the pool cephfs.cephfs.data.>>>> Zitat von Mihai Ciubancan > <mihai.ciubancan(a)eli-np.ro>:>>>>> Hi Anthony,>>> Thanks for the > answer:>>>>>> The output of 'ceph osd df' is:>>>>>> ceph osd df > tree>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA > OMAP >>> META AVAIL %USE VAR PGS STATUS TYPE > NAME>>> -1 167.64825 - 168 TiB 12 TiB 12 TiB > 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - root > default>>> -3 167.64825 - 168 TiB 12 TiB 12 > TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - > host >>> sto-core-hpc01>>> 0 ssd 13.97069 1.00000 14 TiB > 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 > 0 up osd.0>>> 1 ssd 13.97069 1.00000 14 TiB > 12 TiB 12 TiB 6 KiB >>> 26 GiB 2.0 TiB 85.53 12.00 > 129 up osd.1>>> 2 ssd 13.97069 1.00000 14 > TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 > 0 0 up osd.2>>> 3 ssd 13.97069 1.00000 14 > TiB 1.7 GiB 258 MiB 100 MiB >>> 1.3 GiB 14 TiB 0.01 > 0.00 16 up osd.3>>> 4 ssd 13.97069 1.00000 > 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB > 0 0 0 up osd.4>>> 5 ssd 13.97069 > 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 > TiB 0 0 0 up osd.5>>> 6 ssd > 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 > MiB 14 TiB 0 0 0 up osd.6>>> 7 ssd > 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 > MiB 14 TiB 0 0 0 up osd.7>>> 8 ssd > 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 > MiB 14 TiB 0 0 1 up osd.8>>> 9 ssd > 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 > MiB 14 TiB 0 0 0 up osd.9>>> 10 > ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> > 28 MiB 14 TiB 0 0 0 up osd.10>>> 11 > ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> > 63 MiB 14 TiB 0 0 1 up > osd.11>>> TOTAL 168 TiB 12 TiB 12 TiB > 100 MiB >>> 27 GiB 156 TiB 7.13>>>>>> So all the date is on > osd.1>>>>>> But I have checked the balancer and seems active:>>> > ceph balancer status>>> {>>> "active": true,>>> > "last_optimize_duration": "0:00:00.000368",>>> > "last_optimize_started": "Fri Mar 28 10:55:06 2025",>>> "mode": > "upmap",>>> "no_optimization_needed": false,>>> > "optimize_result": "Some objects (0.500000) are degraded; try >>> > again later",>>> "plans": []>>> }>>>>>> But the output of the > commnad 'ceph config dump|grep balancer' >>> gives me > nothing.>>>>>> Best,>>> Mihai>>>>>> On 2025-03-27 23:06, Anthony > D'Atri wrote:>>>> Look at `ceph osd df`. Is the balancer > enabled?>>>>>>>>> On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan > >>>>> <mihai.ciubancan(a)eli-np.ro> wrote:>>>>>>>>>> Hello,>>>>>>>>>> > My name is Mihai, and I have started using CEPH this mount for a > >>>>> HPC cluster.>>>>> When was lunch in the production the > available space shown was >>>>> 80TB now is 16TB and I didn't do > anything, while I'm having 12 >>>>> OSD (SSD of 14TB):>>>>>>>>>> > sudo ceph osd tree>>>>> ID CLASS WEIGHT TYPE > NAME STATUS REWEIGHT PRI-AFF>>>>> -1 > 167.64825 root default>>>>> -3 167.64825 host > sto-core-hpc01>>>>> 0 ssd 13.97069 > osd.0 up 1.00000 1.00000>>>>> 1 ssd > 13.97069 osd.1 up 1.00000 1.00000>>>>> > 2 ssd 13.97069 osd.2 up 1.00000 > 1.00000>>>>> 3 ssd 13.97069 osd.3 up > 1.00000 1.00000>>>>> 4 ssd 13.97069 > osd.4 up 1.00000 1.00000>>>>> 5 ssd > 13.97069 osd.5 up 1.00000 1.00000>>>>> > 6 ssd 13.97069 osd.6 up 1.00000 > 1.00000>>>>> 7 ssd 13.97069 osd.7 up > 1.00000 1.00000>>>>> 8 ssd 13.97069 > osd.8 up 1.00000 1.00000>>>>> 9 ssd > 13.97069 osd.9 up 1.00000 1.00000>>>>> > 10 ssd 13.97069 osd.10 up 1.00000 > 1.00000>>>>> 11 ssd 13.97069 osd.11 up > 1.00000 1.00000>>>>>>>>>> sudo ceph df detail>>>>> --- RAW STORAGE > --->>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED>>>>> > ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>> TOTAL 168 > TiB 156 TiB 12 TiB 12 TiB 7.12>>>>>>>>>> --- POOLS > --->>>>> POOL ID PGS STORED (DATA) (OMAP) > OBJECTS >>>>> USED (DATA) (OMAP) %USED MAX AVAIL QUOTA > OBJECTS QUOTA >>>>> BYTES DIRTY USED COMPR UNDER COMPR>>>>> > .mgr 1 1 705 KiB 705 KiB 0 B 2 > >>>>> 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB > N/A >>>>> N/A N/A 0 B 0 B>>>>> > cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k > >>>>> 270 MiB 270 MiB 0 B 0 16 TiB > N/A >>>>> N/A N/A 0 B 0 B>>>>> > cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M > >>>>> 12 TiB 12 TiB 0 B 42.49 16 TiB > N/A >>>>> N/A N/A 0 B 0 B>>>>>>>>>> > While on the client side I have this:>>>>>>>>>> $ df -h>>>>> > 10.18.31.1:6789:/ 21T 13T 8.1T 61% > /data>>>>>>>>>> I don't know where it's gone all the space that was > at the beginning.>>>>> Someone has any hint?>>>>>>>>>> Best > regards,>>>>> Mihai>>>>> > _______________________________________________>>>>> ceph-users > mailing list -- ceph-users(a)ceph.io>>>>> To unsubscribe send an email > to ceph-users-leave(a)ceph.io>>> > _______________________________________________>>> ceph-users > mailing list -- ceph-users(a)ceph.io>>> To unsubscribe send an email > to ceph-users-leave(a)ceph.io>>>>>> > _______________________________________________>> ceph-users mailing > list -- ceph-users(a)ceph.io>> To unsubscribe send an email to > ceph-users-leave@ceph.io_______________________________________________ceph-users mailing list -- ceph-users(a)ceph.ioTo unsubscribe send an email to > ceph-users-leave(a)ceph.io

1 0

Server not responding to keepalive - cephadm 24.04
by Reid Kelley 28 Mar '25

28 Mar '25

Setting up a new cluster on fresh ubuntu 24.04 hosts using cephadm. The first 5 hosts all added without issue, but the next N hosts all throw the same error when adding through the dashboard or through ceph orch add... All hosts have ssh access, docker and all base requirements confirmed. Command: ceph orch host add foxtrot 10.1.20.16 Error: Error EINVAL: check-host failed: Unable to write foxtrot:/var/lib/ceph/260974b7-fbb6-11ef-911a-107c6152e887/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11: Server not responding to keepalive There is a long stack trace that references asyncssh and the /lib/python3.9/site-packages/asyncssh/sftp.py file. Happy to share but not sure best way in the mailing list. Thanks for any help and advice. -Reid -- Drawbridge Digital is proud to be a certified Veteran-Owned Small Business. The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this email and delete the message and any attachments from your computer.

3 3

Debian Builds for aarch64
by Whitehouse, Dan 28 Mar '25

28 Mar '25

Hi, We are using Debian (bookworm) and have an Ampere based server which we are using for an Openstack compute node alongside some x86_64 machines. If I navigate to the repository here: https://download.ceph.com/debian-squid/pool/main/c/ceph/ I can see that there appear to be x86_64 binaries for Debian and Ubuntu. However, it looks to me like only Ubuntu binaries have been provided for arm64. Can we expect Debian bookworm binaries to be provided too (like they seem to have been for x86_64) or is aarch64 treated differently in this regard? Thanks, Dan Whitehouse

1 0

Production cluster in bad shape after several OSD crashes
by Michel Jouvin 28 Mar '25

28 Mar '25

Hi, We have a production cluster made of 3 mon+mgr, 18 OSD servers and ~500 OSDs and configured with ~50 pools, 1/2 EC (9+6) and 1/2 replica 3. It also has 2 CephFS filesystems with 1 MDS each. 2 days ago, in a period spanning 16 hours, 13 OSD crashed with an OOM. The OSD were first restarted but it was decided to reboot the server with a crashed OSD and "by mistake" (it was at least useless), the OSD of the rebooted server were set noout,norebalance before the reboot. The flags were removed after the reboot. After all of this, 'ceph -s' started to report a lot of misplaced PG and recovery started. All the PGs but one were successfully reactivated. One stayed in the activating+remapped state (located in a pool used for tests). 'ceph health' (I don't put the details here to avoid a too long mail but I can shared them) says: HEALTH_WARN 1 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow metadata IOs; Reduced data availability: 1 pg inactive; 13 daemons have recently crashed and reports about one of the filesystem being degraded despite the only PG reported inactive is not part of a pool related to the FS. The recovery was slow until we realized we should change the mclock profile to high_recovery_ops. Then it completed in a few hours. Unfortunately the degraded filesystem remains degraded without an obvious reason... and the inactive page is still in the activating+remapped state. We have not been able to identify a relevant error in the logs up to now (but we may have missed something...). So far we have avoided restarting too many things until we have a better understanding of what happened and what is the current state. We only restarted the mgr which was using a lot of CPU and the MDS for the degraded FS, without any improvement. We are looking on advices about where to start... It seems we have (at least) 2 independent problems: - A PG that cannot be reactivated with a remap operation that doesn't proceed: would stopping osd.17 help (so that osd.460 is reused)? [root@ijc-mon1 ~]# ceph pg dump_stuck PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 32.7ef activating+remapped [100,154,17] 100 [100,154,460] 100 - 1 degraded filesystem: where to look for a reason? Thanks in advance for any help? Cheers, Michel

3 10

Ceph orch placement - anti affinity
by Kasper Rasmussen 28 Mar '25

28 Mar '25

Let’s say I have 2 cephfs, and three hosts I want to use as MDS hosts. I use ceph orch apply mds to spin up the MDS daemons. Is there a way to ensure that I don’t get two active MDS running on the same host? I mean when using the ceph orch apply mds command, I can specify —placement, but it only seems like I can define the hosts, but not let’s say a primary host or something similar. Anyone with knowledge of how to do this or is it simply not possible?

2 2

Ceph orch placement anti affinity
by Kasper Rasmussen 27 Mar '25

27 Mar '25

Let’s say I have 2 cephfs, and three hosts I want to use as MDS hosts. I use ceph orch apply mds to spin up the MDS daemons. Is there a way to ensure that I don’t get two active MDS running on the same host? I mean when using the ceph orch apply mds command, I can specify —placement, but it only seems like I can define the hosts, but not let’s say a primary host or something similar. Anyone with knowledge of how to do this or is it simply not possible?

2 1

Reef: highly-available NFS with keepalive_only
by Eugen Block 26 Mar '25

26 Mar '25

Hi, I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom. I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments. Thanks, Eugen ---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24 service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip--- [0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with-virtual-ip-but…

5 13

Question about cluster expansion
by Alan Murrell 26 Mar '25

26 Mar '25

Hello, We have a 5-node cluster that each have the following drives: * 4 x 16TB HDD * 4 x 2TB NVME * 1 x 1TB NVME (for the WAL/DB for the HDDs) The nodes don't have any more room to add more NVMEs, but they do have room to add four more HDDs. I know adding more HDDs are able to make the cluster faster due to the additional IOPs. So my question is this: Is it better to: * Add the additional drives/IOPs by adding an additional node * Add the additional drives by adding the the HDDs to the existing nodes Or does it not really matter? I would prefer to add the drives to the existing nodes (ultimately maxing them out), but just wondering if that affects performance as much as expanding by adding additional nodes. Thanks! :-) Sent from my mobile device. Please excuse brevity and ttpos.

3 5

OSD failed: still recovering
by Alan Murrell 25 Mar '25

25 Mar '25

Hello, We had a drive (OSD) failed innour 5 node cluster three days ago (late afternoon of Mar 20). The PGs have sorted themselves out, but the cluster has has been recovering with backfill since then. Every time I run a 'ceph -s' it shows a little over 5% misplaced objects with several jobs of backfill_wait and some scrubbing. What is sort of weird is that if I run the 'ceph -s' as few times in a row, I can see the the percentage of misplaced objects go down a bit but then if I leave it for a while and run 'ceph -s' again, it is still just over 5% misplaced objects but has typically slightly increased. For example, it might be 5.364% when I check it, and then after checking it several times in a row it might go down to 5.276% but then if I check it again after a few hours, it might be something like 5.478% (so still in the 5% range but slightly increased from last check) The cluster is on 10Gbit, and I have increased the max_backfills to 4 while the recovery runs, but it just doesn't seem to be making much progress. I know the failed drive needs to be replaced, but I think it is recommended to wait until the cluster is finished recovering? Your thoughts/advice (as usual) are greatly appreciated. Sent from my mobile device. Please excuse brevity and ttpos.

5 6

OSD_UNREACHABLE After Upgrade to 17.2.8 – Issue with Public Network Detection
by Илья Безруков 25 Mar '25

25 Mar '25

Hello everyone, After upgrading our Ceph cluster from 17.2.7 to 17.2.8 using `cephadm`, all OSDs are reported as unreachable with the following error: ``` HEALTH_ERR 32 osds(s) are not reachable [ERR] OSD_UNREACHABLE: 32 osds(s) are not reachable osd.0's public address is not in '172.20.180.1/32,172.20.180.0/24' subnet osd.1's public address is not in '172.20.180.1/32,172.20.180.0/24' subnet ... osd.31's public address is not in '172.20.180.1/32,172.20.180.0/24' subnet ``` However, all OSDs actually have IP addresses within the `172.20.180.0/24` subnet. The cluster remains functional (CephFS is accessible), and muting the warning with `ceph health mute OSD_UNREACHABLE --sticky` allows normal operation, but the underlying issue persists. ### **Environment Details** - **Ceph Version:** 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) Quincy (stable) - **Container Image:** ` quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f` - **OS:** CentOS Stream 9 - **Deployment Method:** cephadm ### **Network Configuration** The network settings appear correct: ``` ceph config get osd public_network 172.20.180.0/24 ceph config get osd cluster_network 172.20.180.0/24 ceph config get mon public_network 172.20.180.0/24 ceph config get mgr public_network 172.20.180.0/24 ``` OSD metadata confirms that the assigned addresses are in the correct subnet: ``` "front_addr": "[v2: 172.20.180.126:6800/87977614,v1:172.20.180.126:6801/87977614]" ``` ### **Current Cluster Status** - All OSDs are **up** and **in**, and `ceph osd status` reports them as operational. - `ceph orch ps` confirms that OSD services are running. - `ceph mon stat` shows all monitors in quorum. - Cluster services (CephFS, RBD) are working as expected despite the warning. ### **Questions** 1. Has anyone else encountered this issue after upgrading to 17.2.8? 2. Is this a known regression? (This seems similar to issue #67517.) 3. Would upgrading to Ceph 18.x (Reef) resolve the problem? 4. Is there any solution other than muting the health warning? Any insights or recommendations would be greatly appreciated! -- Best regards, Ilya

3 4

Kafka notification, bad certificate
by Malte Stroem 25 Mar '25

25 Mar '25

Hello, configuring bucket notification for Kafka. I have the ca.pem, the user.pem and the user.key file. Running openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem -cert user.pem -key user.key from inside the RGW container everything is fine. I added the three files via - mount_path: ... to the spec file for the RGW service. Inside the container I ran update-ca-trust kafkacat works with the three files as producer. However when uploading objects to the test bucket the RGW shows the following (debug is set to 20): Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down I can only add ca-location to the topic described here: https://docs.ceph.com/en/quincy/radosgw/notifications/ But the notification service needs all infos from all three files. A certificate chain did not work. I browsed rgw_kafka.cc and did not find anything like ca-cert or ca-key. How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker? Best Malte

3 8

Downgrading the osdmap
by Marek Szuba 24 Mar '25

24 Mar '25

Dear fellow Ceph users, I run a Ceph cluster providing CephFS to a medium-sized Linux server farm. Originally we used the kernel driver (which on the distro we use shows itself to the cluster as a Luminous client) to mount the file system, however at the time of the upgrade to Squid we became aware of the data-corruption bug associated with the use of root squash and subsequently switched to (the Squid version of) ceph-fuse. Furthermore, I took advantage of the switch to bump the OSD require-min-compat-client to Reef and subsequently switch the balancer mode to upmap-read. Weeks passed and I became aware that despite all the attempted tuning, certain work loads which performed fine using the kernel driver perform _extremely_ poorly using ceph-fuse. A decision has therefore been made to relocate the servers which absolutely must have root squash to a different network-storage solution, switch root squash off for all CephFS clients and revert to the kernel driver. Unfortunately while removing client_mds_auth_caps from CephFS required_client_features and switching the balancer mode back to upmap went without any problems, "ceph osd set-require-min-compat-client luminous" fails with Error EPERM: osdmap current utilizes features that require reef; cannot set require_min_compat_client below that to luminous Is there a way of making the osdmap Luminous-compatible again without losing any data stored on the cluster ()? Thank you in advance for your help! -- MS

2 3

Reper Ceph Cluster only OSDs intakt
by filip Mutterer 24 Mar '25

24 Mar '25

Playing around with Ceph I did a Big mistake. My root filesystem uses snapshot and only excluding /home & /root. When I switched back to an older Snapshot I noticed Ceph is complaining all OSDs are not working. How can I Fix this? Rebuild cluster and then add the OSDs back in? Is there a guide? Which folders had to be excluded for snapshotting not affecting Ceph?

1 0

Rogue EXDEV errors when hardlinking
by Domhnall McGuigan 24 Mar '25

24 Mar '25

Hi all, we've been seeing persistent problems when trying to create hardlinks on cephfs; it's returning EXDEV in a way that makes no sense given typical POSIX behaviour and ceph documentation. Here's a typical strace of the problem: 78 13:47:26.572435 link("/data/db/hdb/data/2023.08.06/table1.0/column1", "/data/db/hdb/data/2023.08.06/table1.1/column1") = -1 EXDEV (Invalid cross-device link) 78 13:47:26.577661 write(1, "{\"time\":\"2025-03-03T13:47:26.577z\",\"component\":\"MSVC\",\"level\":\"INFO\",\"message\":\"[eoi-78] Retrying in 500 milliseconds\",\"service\":\"eoi\"}\n", 136) = 136 78 13:47:26.577762 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=500000000}, NULL) = 0 78 13:47:27.078037 link("/data/db/hdb/data/2023.08.06/table1.0/column1", "/data/db/hdb/data/2023.08.06/table1.1/column1") = 0 We try creating a link, get EXDEV, wait 500 milliseconds, then try the same operation again and it succeeds. The link and its target are both on the same cephfs mount (/data/db/hdb in this case), so the normal POSIX 'linking between filesystems' explanation doesn't apply. I've looked through the ceph client and server code and from what I've seen EXDEV is only returned in a couple of other situations: linking between snapshots, and linking across quotas. Neither snapshots nor quotas were in use here, and if they were the culprit it seems unlikely the automatic retry would have worked. Web searches on EXDEV errors in ceph have also proven to be a dead end. My best guess, although it's not a very good one, is that stale MDS cache data is somehow involved -- in one case the issue reportedly got much worse after increasing (!) the MDS memory limit. This error has been occurring for a particular client for upwards of 9 months and has proven stubbornly resistant to reproduction elsewhere (we are working on migrating them to a more recent ceph version to see if the error remains), so our technical investigations haven't got particularly far. I was hoping someone here on ceph-users would have seen similar EXDEV errors in the wild or in development and have some insight into what could be causing them. Regards, Domhnall *********************************************************************************************************************************************************************** This email, its contents and any files attached are a confidential communication and are intended only for the named addressees indicated in the message. If you are not the named addressee or if you have received this email in error, you may not, without the consent of KX, copy, use or rely on any information or attachments in any way. Please notify the sender by return email and delete it from your email system. Unless separately agreed, KX does not accept any responsibility for the accuracy or completeness of the contents of this email or its attachments. Please note that any views, opinion or advice contained in this communication are those of the sending individual and not those of KX and KX shall have no liability whatsoever in relation to this communication (or its content) unless separately agreed. ***********************************************************************************************************************************************************************

3 4

Ceph MCP Server
by Anthony Fecarotta 23 Mar '25

23 Mar '25

Has the idea been floated to create a Ceph MCP server? I can't remember if it was the last Ceph Days or Cephlacon, but someone mentioned the concept of a Ceph chatbot trained on Ceph data, which would be awesome, but an MCP server would be a quick/easy way to get up and running in the LLM vertical. Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

1 0

Osd won't restart ceph 17.2.7
by xadhoom76＠gmail.com 22 Mar '25

22 Mar '25

Hi, i try to restart some osds after have enabled compact ceph config set osd osd_compact_on_start true running ceph orch daemon restart osd.4 looking to journaclt of that osd i got this received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 mar 22 03:10:02 ceph03s3.itservicenet.net ceph-aed2e978-85f2-11ed-b955-0cc47aa3fdbc-osd-4[8893]: 2025-03-22T07:10:02.068+0000 7fb232211700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Why does not work ? Best regards

1 1

19.2.1 dashboard OSD column sorts do nothing?
by Harry G Coin 21 Mar '25

21 Mar '25

Has anyone else tried to change the sort order of columns in the cluster/osd display on 19.2.1? While the header changes to indicate 'increasing/descending' and 'selected', the rows stay fixed on an ascending order by ID. ???

2 1

Reef: Dashboard bucket edit fails in get_bucket_versioning
by Eugen Block 21 Mar '25

21 Mar '25

Hi, I just created a new tracker issue [0]. Trying to modify a bucket via dashboard results in a stack trace for the mgr: ---snip--- 2025-03-13T11:23:15.077+0000 7efe02290640 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 357, in get versioning = self._get_versioning(result['owner'], daemon_name, bucket_name) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 246, in _get_versioning return rgw_client.get_bucket_versioning(bucket_name) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 537, in func_wrapper return func( File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 617, in get_bucket_versioning if 'Status' not in result: TypeError: argument of type 'NoneType' is not iterable 2025-03-13T11:23:15.077+0000 7efe02290640 0 [dashboard ERROR request] [::ffff:IP:57676] [GET] [500] [0.034s] [admin] [513.0B] /api/rgw/bucket/bucket2 2025-03-13T11:23:15.077+0000 7efe02290640 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "d574a812-310f-4a51-beba-a677e596c5bf"} ---snip--- I couldn't find anything existing in tracker yet. I don't have versioning enabled. Has anyone seen the same? Thanks! Eugen [0] https://tracker.ceph.com/issues/70449

2 3

One host down osd status error
by Marcus 21 Mar '25

21 Mar '25

Hi all, We are running a ceph cluster with filesystem that contains 5 servers. Ceph version: 19.2.0 squid If I run: ceph osd status when all hosts are online and in the output is the way it should and it prints status for all osds. If just a couple of osds are down status is printed and specific osds are stated as down. One of the servers went down and we ended up with a health warning. If I run: ceph osd stat I get the information that 64 out of 80 osd is in. If I try to run: ceph osd status I get an python error: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1864, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/status/module.py", line 337, in handle_osd_status assert metadata AssertionError I suppose this is some type of bug when one host is down? Thanks! Marcus

3 4

March Ceph Science Virtual User Group
by Enrico Bocchi 20 Mar '25

20 Mar '25

Dear All, We will be having a Ceph science/research/big cluster call on Tuesday March 25th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We are trying to start each meeting with a brief 10-15 minutes talk given by a community member. While there is none schedule for the upcoming meeting, we welcome contributions. If you are interested in signing up to give a short talk for a meeting please let me know! Virtual event details: March 25th, 2025 10am Central US 15:00 UTC 4pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ Go to Mar 25 and you can quickly add it to your calendar as well. See you there! Enrico

1 1

Subject: Assistance Required: Vault Integration with RADOS Gateway for SSE-S3 Encryption
by Dhivya G 20 Mar '25

20 Mar '25

Dear Ceph Community, We have been researching encryption mechanisms for a while and have successfully deployed Ceph using Kubernetes. Currently, we are working on integrating HashiCorp Vault with RADOS Gateway to enable server-side encryption (SSE-S3) for object uploads. Despite configuring Vault with RADOS Gateway, we are encountering an "INVALID REQUEST" error when attempting to upload objects into an encrypted bucket. Additionally, we are unable to locate any relevant Vault logs that provide insights into this specific error. We would greatly appreciate any insights from those who have faced a similar issue or have successfully configured Vault transit with RADOS Gateway for SSE-S3 encryption. Any guidance or shared experiences would be immensely helpful in resolving this issue. Looking forward to your expert advice. Thanks & Regards, Dhivya G|Associate Software Engineer Ph No: (+91) 9894944910 email: dhivya.g(a)zybisys.com Zybisys Consulting LLP | Srinivasa Tower' No.29, 39th E Cross, 4th T Block, Jayanagar , Bangalore India - 560041http://zybisys.com/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 3

Attention: Documentation
by Joel Davidow 20 Mar '25

20 Mar '25

While reading about mons, I've come across two documentation issues. 1. The 2013 blog "Monitors and Paxos, a chat with Joao" has a broken link. - The broken link returns a 404 - The bad URL is http://ceph.com/dev-notes/cephs-new-monitor-changes/ - That bad URL is on line 8 at https://github.com/ceph/ceph.io/blob/main/src/en/news/blog/2013/monitors-an… - The working URL is https://ceph.io/en/news/blog/2013/cephs-new-monitor-changes/ 2. The 2013 blog "Ceph’s New Monitor Changes" has 3 broken links. - Figures 1, 2, and 3 each have URLs that return 404 - http://ceph.com/wp-content/uploads/2013/03/2013-03-07_17-57-12.png - http://ceph.com/wp-content/uploads/2013/03/2013-03-07_17-58-59.png - http://ceph.com/wp-content/uploads/2013/03/2013-03-07_17-59-49.png - I was unable to locate any of those three png files in Ceph's Github org - It'd be great if those three png files could be found and links updated so I can continue learning about how mons work. Thanks, Joel

4 5

Division by zero while upgrading
by Alex 20 Mar '25

20 Mar '25

Upgrading from Pacific to Quincy, anyone ever see this? This one looks FUN! I upgraded mgrs, mons, and 1/2 of the osd nodes. Hit this when upgrading the last OSD node. It's a test cluster. Running the command below: ceph orch upgrade start ... --hosts <one_host> --services osd.all-available-devices $ ceph health detail HEALTH_ERR Upgrade: failed due to an unexpected exception [ERR] UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception Unexpected exception occurred during upgrade process: division by zero

1 1

ceph-osd/bluestore using page cache
by Brian Marcotte 19 Mar '25

19 Mar '25

Some years ago when first switching to Bluestore, I could see that ceph-osd wasn't using the host page cache anymore. Some time later after a Ceph upgrade, I found that ceph-osd was now filling the page cache. I'm sorry I don't remember which upgrade that was. Currently I'm running pacific and reef clusters. Should ceph-osd (Bluestore) be going through the page cache? Can ceph-osd be configured to go direct? Thanks. -- - Brian

5 5

Issues with Ceph Cluster Behavior After Migration from ceph-ansible to cephadm and Upgrade to Quincy/Reef
by Jeremi-Ernst Avenant 19 Mar '25

19 Mar '25

Hi all, I recently migrated my Ceph cluster from *ceph-ansible* to *cephadm* (about five months ago) and upgraded from *Pacific 16.2.11* to *Quincy (latest at the time)*, followed by an upgrade to *Reef 18.2.4* two months later - due to running an unsupported version of Ceph. Since this migration and upgrade, I’ve noticed unexpected behavior in the cluster, particularly related to OSD state awareness and balancer efficiency. *1. OSD Nearfull Not Reported Until Restart* I had an OSD exceed its configured nearfull threshold, but *Ceph did not detect or report it* via ceph status. As a result, the cluster entered a degraded state without any warnings. Only after manually restarting the affected OSD did Ceph recognize the nearfull state and update the corresponding pools accordingly. This behavior did not occur in *Pacific/ceph-ansible*—Ceph would previously detect and act on the nearfull condition without requiring a restart. This has been a common recurrence since the migration/upgrade. *2. injectargs Not Taking Effect Until OSD Restart* I've also observed that ceph tell osd.X injectargs --command ... often has no effect. The OSD does not seem to apply the new arguments until it is *manually restarted*, at which point I can modify values via injectargs as expected. However, after a few hours or days, the issue reappears, requiring another restart to modify runtime settings. *3. Ceph Balancer and PG Remapping Issues* The Ceph balancer appears to be operating, but its behavior seems inefficient compared to what we experienced on *Pacific*. It often fails to optimize data distribution effectively, and I have to rely on the *pgremapper* tool to manually intervene. Restarting OSDs seems to improve the balancer’s effectiveness temporarily, suggesting that stale OSD state information may be contributing to the issue. Since this is a *high-performance computing (HPC) environment*, manually restarting OSDs on a regular basis is not a viable solution. These issues did not occur when we were running *Pacific with ceph-ansible*, and I’m wondering if others have experienced similar problems after migrating to *cephadm* and/or upgrading to *Quincy/Reef*. I noticed people on reddit with the same issue, but their resolution was that they "I switch off my whole ceph cluster & switch it back on - to get it working 100% again, - DAILY" Has anyone else encountered these behaviors? Are there any known bugs or workarounds that could help restore expected OSD state tracking and balancer efficiency? Any insights would be greatly appreciated! Thanks, -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

3 3

Is it safe to set multiple OSD out across multiple failure domain?
by Kai Stian Olstad 19 Mar '25

19 Mar '25

Say we have 10 host with 10 OSDs and the failure domain is host. host0 osd 0 to 9 host1 osd 10 to 19 host2 osd 20 to 29 host3 osd 30 to 39 host4 osd 40 to 49 host5 osd 50 to 59 host6 osd 60 to 69 host7 osd 70 to 79 host8 osd 80 to 89 host9 osd 90 to 99 A pool have EC 4+2 and PG 2.1 has UP and ACTING the following OSDs 0,10,20,30,40 and 50. I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out? I would think and hope that Ceph do the right thing and the PG remain accessible since all OSD is available and data is being copied to other OSDs. -- Kai Stian Olstad

6 10

All Github Actions immediately blocked, except GH-official and Ceph-hosted ones
by Ernesto Puerta 18 Mar '25

18 Mar '25

Hi Cephers, Due to the escalating situation in which leaked secrets from a previously compromised GH Action (tj-actions) [1] are used to compromise more popular GH Actions (reviewdog)[2], *we have decided to immediately disable all Github Actions in all repositories, except the official GH ones, and the ones under the Ceph organization*. *From a preliminary analysis, none of our repos has been impacted*. However, given that the Ceph org hosts more that 200 repositories (including forks), it's safer to take a conservative approach while a deeper analysis is undertaken (and the situation settles down). We know that this might have an immediate impact on Ceph teams, as it will break CI using unofficial Github Actions. The recommendation for *each team is to review their Github workflows in their repos* (.github/workflows/*.yaml) and ensure that: - *Version tag pinning is replaced with the corresponding SHA-1 hash*, even for GH official actions. Example <https://github.com/ceph/ceph/blob/8890c0f0df343897195ab97104be0a551d899361/…> . - Once the previous requirement is met, *GH Actions from unofficial sources can be requested* (via mail to security(a)ceph.io) to be included in the allow-list. - Except for actions from the "reviewdog" org <https://github.com/reviewdog>: their team is still analyzing the scope of the attack. In this case, we shouldn't yet trust SHA-1 hashes obtained after the attack until the analysis concludes. - Alternatively, in the long run, we are considering to start forking 3rd party GH Actions under the Ceph org, but carries extra overhead for the Ceph maintainers. We apologize for the inconvenience, but we did this to reduce the exposure of the Ceph community to a series of attacks that may continue to escalate. Kind Regards, Ernesto [1] https://www.stepsecurity.io/blog/harden-runner-detection-tj-actions-changed… [2] https://www.stepsecurity.io/blog/reviewdog-github-actions-are-compromised

1 0

Ceph User + Developer March Meetup happening tomorrow!
by Laura Flores 18 Mar '25

18 Mar '25

Hi all, Join us at the User + Dev meeting tomorrow, where we'll be presenting some highlights from the Ceph User Stories survey and discussing next steps! RSVP here: https://www.meetup.com/ceph-user-group/events/306792345/ Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Remove ... something
by Albert Shih 17 Mar '25

17 Mar '25

Hi, I got two pool one replica 3 and another with EC 4+2 for rbd. During my test to access from a KVM hyperviseur I create a image with meta data in the replica 3 pool and the data in the EC 4+2. After I end the test (where everything works) I can't see anything in erasure42 pool. rbd list --pool erasure42 return a empty list. But root@alopod:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 147 TiB 146 TiB 564 GiB 564 GiB 0.38 TOTAL 147 TiB 146 TiB 564 GiB 564 GiB 0.38 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 23 MiB 7 68 MiB 0 46 TiB erasure42 14 32 336 GiB 86.46k 504 GiB 0.36 92 TiB it still use 336GiB. How can I find where those 336Gib are and delete the image or whatever. Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: lun. 17 mars 2025 08:24:32 CET

2 2

My new osd is not normally ?
by Yunus Emre Sarıpınar 17 Mar '25

17 Mar '25

I have 6 ssd sata and 12 osd per server in a 24 server cluster. This environment was created when it was in the natilus version. I switched this environment to the Octopus version 6 months ago. The cluster is working healthily. I added 8 new servers, I created 6 ssd sata and 12 osd on these servers in the same way. I did not change the number of PGs in the environment, I have 8192 PGs. The problem is that in my ceph -s output, the remapped pg and missplaced object states are gone, but there is a warning of 6nearfull osds and 4pools nearfull. I saw in the ceph df output that my pools are also full above normal. In the output of the ceph osd df tree command, I observed that the occupancy percentages of the newly added osds were around 80%, while the percentages of the old osds were around 30%. How do I equalize this situation? Note: I am sharing the output of crushmap and osd df tree with you in the attachment. My new osds between 288-384. My new servers are ekuark13,14,15,16 and bkuark13,14,15,16.

2 1

Slow benchmarks for rbd vs. rados bench
by Daniel Vogelbacher 16 Mar '25

16 Mar '25

Hi, during deployment of a small testcluster on two existing servers I got multiple benchmark results I don't understand. In summary, I think the rados benchmark values are okay for my limited setup - I got ~80 MiB/s for write, ~300 MiB/s for reads. But for rbd benchmarks, the values dropped down to ~46 MiB/s for writes and 71 MiB/s for reads. Have I missed something for rbd bench? I don't understand why rbd is so much slower than rados. Hint: the productive cluster should get 3 servers total and each server 6x 12TB SAS3 disks and maybe 2 additional servers for up to 5 monitors. I know Ceph makes more fun with > 3, but budget ist limited :( Here are the results of all benches: # rados bench -p testbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_virt-master4_1711455 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 31 15 59.9956 60 0.430283 0.622336 2 16 51 35 69.9923 80 0.565152 0.635417 3 16 74 58 77.3241 92 0.746055 0.663046 4 16 97 81 80.99 92 0.558241 0.646831 5 16 122 106 84.7893 100 0.705829 0.686034 6 16 149 133 88.6553 108 0.496865 0.674404 7 16 170 154 87.9886 84 0.916755 0.674053 8 16 189 173 86.4887 76 4.24615 0.700781 9 16 215 199 88.4327 104 0.60869 0.702643 10 16 235 219 87.588 80 0.882353 0.696558 11 1 235 234 85.0792 60 0.666018 0.692935 Total time run: 11.8131 Total writes made: 235 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 79.5726 Stddev Bandwidth: 16.1087 Max bandwidth (MB/sec): 108 Min bandwidth (MB/sec): 60 Average IOPS: 19 Stddev IOPS: 4.02718 Max IOPS: 27 Min IOPS: 15 Average Latency(s): 0.708564 Stddev Latency(s): 0.440743 Max latency(s): 4.36566 Min latency(s): 0.318937 # rados bench -p testbench 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 87 71 283.925 284 0.354899 0.187907 2 16 161 145 289.941 296 0.122009 0.194255 3 5 235 230 306.611 340 0.231543 0.198982 Total time run: 3.08835 Total reads made: 235 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 304.37 Average IOPS: 76 Stddev IOPS: 7.37111 Max IOPS: 85 Min IOPS: 71 Average Latency(s): 0.200212 Max latency(s): 1.87814 Min latency(s): 0.017529 # rbd bench --io-type write image01 --pool=testbench bench type write io_size 4096 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 11520 11536.2 45 MiB/s 2 23984 12000.2 47 MiB/s 3 38304 12739.6 50 MiB/s 4 52672 13028.9 51 MiB/s 5 63616 12575.7 49 MiB/s 6 71456 11987.4 47 MiB/s 7 82464 11677.5 46 MiB/s 8 96080 11573.9 45 MiB/s 9 110080 11436.1 45 MiB/s 10 124512 12327.4 48 MiB/s 11 136496 13008.2 51 MiB/s 12 148160 13118.5 51 MiB/s 13 163216 13374 52 MiB/s 14 176848 13527 53 MiB/s 15 191120 13132.7 51 MiB/s 16 203376 13376.3 52 MiB/s 17 212576 12924.8 50 MiB/s 18 221120 11534.9 45 MiB/s 19 228032 10196.2 40 MiB/s 20 241584 10240.5 40 MiB/s 21 254176 10095.6 39 MiB/s elapsed: 22 ops: 262144 ops/sec: 11896.4 bytes/sec: 46 MiB/s ~# rbd bench --io-type read image01 --pool=testbench bench type read io_size 4096 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 18368 18384.4 72 MiB/s 2 36032 18024.3 70 MiB/s 3 56768 18928.4 74 MiB/s 4 75472 18872.4 74 MiB/s 5 93264 18656.4 73 MiB/s 6 109232 18173.1 71 MiB/s 7 128320 18458 72 MiB/s 8 146336 17913.9 70 MiB/s 9 164176 17741.1 69 MiB/s 10 184720 18291.6 71 MiB/s 11 202416 18637.2 73 MiB/s 12 216688 17673.9 69 MiB/s 13 236464 18025.9 70 MiB/s 14 255776 18320.4 72 MiB/s elapsed: 14 ops: 262144 ops/sec: 18255.5 bytes/sec: 71 MiB/s Some cluster details (per server): * SAS3 SEAGATE ST12000NM004J * Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz * Dual NIC Intel E810 (LACP, 2x10GBit Link) * LSI SAS3004 PCI-Express Fusion-MPT SAS-3 * 192 GB ECC RAM Ceph Squid deployed via cephadm. ~# ceph -s cluster: id: 7644057a-00f6-11f0-9a0c-eac00fed9338 health: HEALTH_WARN 1 pool(s) do not have an application enabled services: mon: 2 daemons, quorum virt-master4,virt-master3 (age 24h) mgr: virt-master4.lddrxr(active, since 24h), standbys: virt-master3.akkopo osd: 2 osds: 2 up (since 3h), 2 in (since 3h) data: pools: 3 pools, 65 pgs objects: 3.43k objects, 13 GiB usage: 27 GiB used, 22 TiB / 22 TiB avail pgs: 65 active+clean ~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 21.82808 root default -5 10.91399 host virt-master3 1 hdd 10.91399 osd.1 up 1.00000 1.00000 -3 10.91409 host virt-master4 0 hdd 10.91409 osd.0 up 1.00000 1.00000 # ceph osd pool ls detail pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 2.00 pool 2 'kvm' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 33 lfor 0/0/28 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm snappy compression_mode aggressive application rbd read_balance_score 1.06 pool 3 'testbench' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 446 lfor 0/443/441 flags hashpspool,selfmanaged_snaps stripe_width 0 read_balance_score 1.06 osd.0 direct benchmark with 4M and 4K blocksize: root@virt-master4:~# fio --ioengine=sync --filename=/dev/disk/by-id/scsi-35000c500f8bfda83 --direct=1 --sync=1 --rw=write --bs=4M --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=sync, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=172MiB/s][w=43 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=1585455: Sat Mar 15 14:29:21 2025 write: IOPS=42, BW=170MiB/s (178MB/s)(9.97GiB/60013msec); 0 zone resets clat (usec): min=21993, max=40219, avg=23274.37, stdev=963.46 lat (usec): min=22341, max=40453, avg=23511.77, stdev=961.26 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22414], 20.00th=[22676], | 30.00th=[22938], 40.00th=[22938], 50.00th=[23200], 60.00th=[23200], | 70.00th=[23462], 80.00th=[23462], 90.00th=[24249], 95.00th=[24511], | 99.00th=[24773], 99.50th=[31327], 99.90th=[32637], 99.95th=[32637], | 99.99th=[40109] bw ( KiB/s): min=163840, max=180224, per=99.99%, avg=174166.05, stdev=3764.97, samples=119 iops : min= 40, max= 44, avg=42.52, stdev= 0.92, samples=119 lat (msec) : 50=100.00% cpu : usr=1.00%, sys=2.05%, ctx=7648, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2552,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=9.97GiB (10.7GB), run=60013-60013msec root@virt-master4:~# fio --ioengine=sync --filename=/dev/disk/by-id/scsi-35000c500f8bfda83 --direct=1 --sync=1 --rw=randwrite --bs=4M --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=randwrite, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=sync, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=124MiB/s][w=31 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=1588408: Sat Mar 15 14:37:13 2025 write: IOPS=30, BW=123MiB/s (129MB/s)(7360MiB/60028msec); 0 zone resets clat (usec): min=19173, max=77481, avg=32327.62, stdev=6604.03 lat (usec): min=19453, max=77684, avg=32615.39, stdev=6605.82 clat percentiles (usec): | 1.00th=[21103], 5.00th=[23462], 10.00th=[25035], 20.00th=[26870], | 30.00th=[28443], 40.00th=[30016], 50.00th=[31327], 60.00th=[32637], | 70.00th=[34341], 80.00th=[36439], 90.00th=[41157], 95.00th=[45876], | 99.00th=[51119], 99.50th=[53216], 99.90th=[69731], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min=65536, max=147456, per=99.99%, avg=125542.40, stdev=9567.74, samples=120 iops : min= 16, max= 36, avg=30.65, stdev= 2.34, samples=120 lat (msec) : 20=0.22%, 50=98.32%, 100=1.47% cpu : usr=0.90%, sys=1.53%, ctx=5513, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1840,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=7360MiB (7718MB), run=60028-60028msec root@virt-master4:~# fio --ioengine=io_uring --filename=/dev/disk/by-id/scsi-35000c500f8bfda83 --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=37.6MiB/s][w=9628 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=1591836: Sat Mar 15 14:48:18 2025 write: IOPS=9560, BW=37.3MiB/s (39.2MB/s)(2241MiB/60001msec); 0 zone resets slat (usec): min=3, max=242, avg= 5.75, stdev= 3.14 clat (usec): min=3, max=37068, avg=97.67, stdev=81.74 lat (usec): min=84, max=37086, avg=103.42, stdev=81.88 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 89], 10.00th=[ 90], 20.00th=[ 92], | 30.00th=[ 93], 40.00th=[ 94], 50.00th=[ 95], 60.00th=[ 97], | 70.00th=[ 98], 80.00th=[ 100], 90.00th=[ 104], 95.00th=[ 109], | 99.00th=[ 125], 99.50th=[ 159], 99.90th=[ 416], 99.95th=[ 676], | 99.99th=[ 889] bw ( KiB/s): min=13600, max=39344, per=100.00%, avg=38312.13, stdev=2316.38, samples=119 iops : min= 3400, max= 9836, avg=9578.03, stdev=579.09, samples=119 lat (usec) : 4=0.01%, 20=0.01%, 50=0.01%, 100=80.54%, 250=19.17% lat (usec) : 500=0.21%, 750=0.04%, 1000=0.03% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=3.54%, sys=8.67%, ctx=573655, majf=0, minf=52 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,573639,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=37.3MiB/s (39.2MB/s), 37.3MiB/s-37.3MiB/s (39.2MB/s-39.2MB/s), io=2241MiB (2350MB), run=60001-60001msec -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher

2 2

Adding device class to CRUSH rule without data movement
by Hector Martin 15 Mar '25

15 Mar '25

Hi, I have an old Mimic cluster that I'm doing some cleanup work on and adding SSDs, before upgrading to a newer version. As part of adding SSDs, I need to switch the existing CRUSH rules to only use the HDD device class first. Is there some way of doing this that doesn't result in 100% data movement? Simply replacing `step take default` with `step take default class hdd` for every CRUSH rule seems to completely shuffle the cluster data. I tried manually specifying the bucket IDs for the hdd hierarchy to at least be in the same order as the bucket IDs for the primary hierarchy, hoping having them sort the same would end up with the same data distribution, but that didn't work either. Is there some magic incantation to swap around the CRUSH rules/tree so that it results in exactly the same data distribution after adding the hdd class constraint? The set of potential OSDs should be identical (there are no SSDs yet), so the data movement seems to be some technicality of the CRUSH implementation... perhaps completely switching around the main id and hdd-class id of all the buckets would do it? (I'm a little afraid to mess with the main ids in a production cluster...). This cluster is already having I/O load issues (that's part of why I'm adding SSDs), so I'd really like to avoid a total data shuffle if possible. Thanks, - Hector

3 4

Massive performance issues
by Thomas Schneider 15 Mar '25

15 Mar '25

Hi, we are having massive performance issues with our Ceph cluster, and by now I have no idea how and where to debug further. See attachments for full benchmark logs, relevant excerpts: HDD pool: > read: IOPS=7591, BW=119MiB/s (124MB/s)(34.9GiB/300892msec) SSD pool: > read: IOPS=2007, BW=31.4MiB/s (32.9MB/s)(9422MiB/300334msec) Yes, the SSD pool is in fact slower in both IOPS and data rate than the HDD pool. Of course this is only one particular benchmark scenario, but at least it has some numbers and not just “everything feels slow”. I’m happy to run different benchmarks if required. Cluster information: * 2x10G ethernet to a Cisco Catalyst 3850 on each node * Debian 11 (bullseye), Linux 5.10 * Ceph 18.2.4 (reef), managed with cephadm * All but the oldest node serve as VM hypervisors (KVM) as well * 6 nodes (hostnames appended to compare with OSD information later) - 2x AMD EPYC 7232P, 64G RAM (cirrus, nimbus) - 1x Intel Xeon E3-1230 v3, 32G RAM (pileus, no VMs) - 2x Intel Xeon E5-1620 v4, 64G RAM (lenticular, nacreous) - 1x Intel Xeon Silver 4110, 96G RAM (stratus) * Cluster usage: mostly RBD, some RGW and CephFS * OSDs: - 19 Hitachi Ultrastar 7K4000 2TB - 1 WD Ultrastar DC HC310 4TB - 3 Hitachi Deskstar 7k2000 2TB - 13 Hitachi Ultrastar A7K3000 2TB - 8 Micron 5210 ION 2TB - 4 Micron 5300 MAX - Mixed Use 2TB - 5 Samsung Datacenter PM893 2TB You can find various information that I deemed useful below. Please ask if you would like further information. As I said, I don’t really know where to look and what to do, so I would really appreciate any pointers on how to debug this and improve performance. Of course I have already seen the pg warnings, but I am really not sure what to adjust in which direction, especially the contradiction between the MANY_OBJECTS_PER_PG and POOL_TOO_MANY_PGS warnings for the default.rgw.ec.data pool (there are too many objects per pg, so I should probably increase pg_num, however it recommends to scale down from 128 to 32?!). Thanks, Thomas # ceph -s cluster: id: 91688ac0-1b4f-43e3-913d-a844338d9325 health: HEALTH_WARN 1 pools have many more objects per pg than average 3 pools have too many placement groups services: mon: 5 daemons, quorum nimbus,lenticular,stratus,cirrus,nacreous (age 5d) mgr: lenticular(active, since 3M), standbys: nimbus, stratus, nacreous, cirrus mds: 1/1 daemons up, 1 standby osd: 53 osds: 53 up (since 2w), 53 in (since 18M) rgw: 5 daemons active (5 hosts, 1 zones) data: volumes: 1/1 healthy pools: 15 pools, 2977 pgs objects: 7.20M objects, 15 TiB usage: 42 TiB used, 55 TiB / 97 TiB avail pgs: 2975 active+clean 1 active+clean+scrubbing 1 active+clean+scrubbing+deep io: client: 6.5 MiB/s rd, 1.3 MiB/s wr, 158 op/s rd, 63 op/s wr # ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average; 3 pools have too many placement groups [WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than average pool default.rgw.ec.data objects per pg (27197) is more than 11.2524 times cluster average (2417) [WRN] POOL_TOO_MANY_PGS: 3 pools have too many placement groups Pool default.rgw.ec.data has 128 placement groups, should have 32 Pool templates has 512 placement groups, should have 256 Pool rbd.ec has 1024 placement groups, should have 256 # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 67 TiB 29 TiB 39 TiB 39 TiB 57.58 ssd 30 TiB 26 TiB 3.7 TiB 3.7 TiB 12.37 TOTAL 97 TiB 55 TiB 42 TiB 42 TiB 43.74 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL templates 1 512 4.4 TiB 1.27M 13 TiB 37.46 7.4 TiB .rgw.root 9 32 1.4 KiB 4 768 KiB 0 7.4 TiB default.rgw.control 10 32 0 B 8 0 B 0 7.4 TiB default.rgw.meta 11 32 7.0 KiB 38 5.1 MiB 0 7.4 TiB default.rgw.log 12 32 3.5 KiB 208 5.7 MiB 0 7.4 TiB default.rgw.ec.data 14 128 1.3 TiB 3.48M 2.7 TiB 10.95 13 TiB default.rgw.buckets.index 15 32 57 MiB 53 171 MiB 0 7.4 TiB default.rgw.buckets.data 16 32 2.5 GiB 17.85k 8.3 GiB 0.04 7.4 TiB default.rgw.buckets.non-ec 17 32 1.6 KiB 1 197 KiB 0 7.4 TiB rbd.ssd 19 512 872 GiB 259.98k 2.6 TiB 9.46 8.1 TiB .mgr 20 1 640 MiB 161 1.9 GiB 0 7.4 TiB cephfs_metadata 21 32 404 MiB 35.40k 1.2 GiB 0 8.1 TiB cephfs_data 22 32 42 GiB 28.09k 128 GiB 0.56 7.4 TiB rbd 23 512 12 GiB 6.57k 39 GiB 0.17 7.4 TiB rbd.ec 24 1024 8 TiB 2.10M 16 TiB 42.48 11 TiB The pool "templates" is named as such for historical reasons. It is used as RBD VM image storage. # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .rgw.root 256.0k 3.0 68931G 0.0000 1.0 32 off False default.rgw.control 0 3.0 68931G 0.0000 1.0 32 off False default.rgw.meta 1740k 3.0 68931G 0.0000 1.0 32 off False default.rgw.log 1932k 3.0 68931G 0.0000 1.0 32 off False default.rgw.ec.data 1678G 1.6666666269302368 68931G 0.0406 1.0 128 warn False default.rgw.buckets.index 58292k 3.0 68931G 0.0000 1.0 32 off False default.rgw.buckets.data 2828M 3.0 68931G 0.0001 1.0 32 off False default.rgw.buckets.non-ec 67159 3.0 68931G 0.0000 1.0 32 off False .mgr 639.7M 3.0 68931G 0.0000 1.0 1 off False cephfs_metadata 405.4M 3.0 30404G 0.0000 4.0 32 off False cephfs_data 43733M 3.0 68931G 0.0019 1.0 32 off False templates 4541G 3.0 68931G 0.1977 1.0 512 warn True rbd.ssd 870.4G 3.0 30404G 0.0859 1.0 512 warn True rbd 13336M 3.0 68931G 0.0006 1.0 512 off True rbd.ec 8401G 2.0 68931G 0.2438 1.0 1024 warn True # ceph osd pool ls detail pool 1 'templates' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode warn last_change 7495052 lfor 0/2976944/7469772 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application rbd read_balance_score 1.48 pool 9 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467477 flags hashpspool stripe_width 0 application rgw read_balance_score 3.38 pool 10 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467478 flags hashpspool stripe_width 0 application rgw read_balance_score 3.37 pool 11 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467479 flags hashpspool stripe_width 0 application rgw read_balance_score 3.36 pool 12 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467480 flags hashpspool stripe_width 0 application rgw read_balance_score 4.52 pool 14 'default.rgw.ec.data' erasure profile rgw-video size 5 min_size 4 crush_rule 3 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 7494999 lfor 0/5467738/7467533 flags hashpspool stripe_width 12288 application rgw pool 15 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467482 flags hashpspool stripe_width 0 application rgw read_balance_score 4.52 pool 16 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467483 flags hashpspool stripe_width 0 application rgw read_balance_score 3.39 pool 17 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467484 flags hashpspool stripe_width 0 application rgw read_balance_score 3.38 pool 19 'rbd.ssd' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode warn last_change 7495053 lfor 0/141997/7467030 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application rbd read_balance_score 1.43 pool 20 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 7480977 flags hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth read_balance_score 37.50 pool 21 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467487 lfor 0/0/96365 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.59 pool 22 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467488 lfor 0/0/96365 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs read_balance_score 4.52 pool 23 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode off last_change 7467489 lfor 0/0/884114 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application rbd read_balance_score 1.41 pool 24 'rbd.ec' erasure profile rbd size 4 min_size 3 crush_rule 5 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 7495054 lfor 0/2366959/2366961 flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 8192 application rbd # ceph osd utilization avg 192.66 stddev 75.9851 (expected baseline 13.7486) min osd.48 with 88 pgs (0.456762 * mean) max osd.10 with 500 pgs (2.59524 * mean) # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -13 95.71759 - 97 TiB 42 TiB 42 TiB 4.2 GiB 204 GiB 55 TiB 43.74 1.00 - root default -16 14.15759 - 15 TiB 8.2 TiB 8.2 TiB 713 MiB 29 GiB 6.4 TiB 56.36 1.29 - host cirrus 2 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.0 TiB 85 MiB 4.9 GiB 784 GiB 57.94 1.32 223 up osd.2 5 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 50 MiB 3.2 GiB 821 GiB 55.95 1.28 223 up osd.5 8 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 88 MiB 2.9 GiB 804 GiB 56.86 1.30 229 up osd.8 11 hdd 1.76970 1.00000 1.8 TiB 963 GiB 959 GiB 236 MiB 3.6 GiB 900 GiB 51.66 1.18 240 up osd.11 13 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 61 MiB 4.7 GiB 823 GiB 55.81 1.28 220 up osd.13 15 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 52 MiB 3.1 GiB 762 GiB 59.13 1.35 229 up osd.15 18 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.0 TiB 85 MiB 2.7 GiB 787 GiB 57.76 1.32 228 up osd.18 19 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 54 MiB 3.4 GiB 824 GiB 55.79 1.28 219 up osd.19 -18 15.93417 - 16 TiB 6.0 TiB 6.0 TiB 663 MiB 23 GiB 10 TiB 37.15 0.85 - host lenticular 0 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 88 MiB 3.2 GiB 796 GiB 57.26 1.31 229 up osd.0 4 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 42 MiB 6.0 GiB 753 GiB 59.57 1.36 231 up osd.4 6 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 3.5 GiB 758 GiB 59.31 1.36 237 up osd.6 10 hdd 3.63869 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 127 MiB 5.9 GiB 1.7 TiB 52.08 1.19 500 up osd.10 36 ssd 1.74660 1.00000 1.7 TiB 229 GiB 228 GiB 92 MiB 959 MiB 1.5 TiB 12.79 0.29 103 up osd.36 37 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 97 MiB 1006 MiB 1.5 TiB 12.49 0.29 98 up osd.37 38 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 61 MiB 1.0 GiB 1.5 TiB 12.46 0.28 98 up osd.38 39 ssd 1.74660 1.00000 1.7 TiB 221 GiB 219 GiB 102 MiB 1.4 GiB 1.5 TiB 12.35 0.28 98 up osd.39 -17 22.89058 - 23 TiB 9.5 TiB 9.4 TiB 1.1 GiB 45 GiB 14 TiB 40.59 0.93 - host nacreous 1 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 124 MiB 6.0 GiB 818 GiB 56.07 1.28 229 up osd.1 3 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 3.1 GiB 780 GiB 58.11 1.33 222 up osd.3 7 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 48 MiB 3.1 GiB 773 GiB 58.48 1.34 237 up osd.7 9 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 50 MiB 4.9 GiB 766 GiB 58.88 1.35 223 up osd.9 14 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 93 MiB 3.3 GiB 812 GiB 56.43 1.29 224 up osd.14 16 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 43 MiB 5.0 GiB 792 GiB 57.51 1.31 235 up osd.16 17 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 68 MiB 4.2 GiB 777 GiB 58.31 1.33 227 up osd.17 20 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 66 MiB 5.9 GiB 780 GiB 58.15 1.33 231 up osd.20 48 ssd 1.74660 1.00000 1.7 TiB 213 GiB 212 GiB 94 MiB 1.3 GiB 1.5 TiB 11.92 0.27 88 up osd.48 49 ssd 1.74660 1.00000 1.7 TiB 215 GiB 212 GiB 94 MiB 2.5 GiB 1.5 TiB 12.01 0.27 90 up osd.49 50 ssd 1.74660 1.00000 1.7 TiB 214 GiB 212 GiB 95 MiB 2.1 GiB 1.5 TiB 11.99 0.27 92 up osd.50 51 ssd 1.74660 1.00000 1.7 TiB 217 GiB 215 GiB 144 MiB 1.9 GiB 1.5 TiB 12.13 0.28 94 up osd.51 52 ssd 1.74660 1.00000 1.7 TiB 214 GiB 212 GiB 144 MiB 2.0 GiB 1.5 TiB 11.95 0.27 94 up osd.52 -19 6.98639 - 7.0 TiB 895 GiB 889 GiB 345 MiB 5.6 GiB 6.1 TiB 12.51 0.29 - host nimbus 44 ssd 1.74660 1.00000 1.7 TiB 227 GiB 224 GiB 148 MiB 2.3 GiB 1.5 TiB 12.67 0.29 98 up osd.44 45 ssd 1.74660 1.00000 1.7 TiB 222 GiB 221 GiB 68 MiB 1021 MiB 1.5 TiB 12.39 0.28 93 up osd.45 46 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 64 MiB 972 MiB 1.5 TiB 12.48 0.29 97 up osd.46 47 ssd 1.74660 1.00000 1.7 TiB 224 GiB 222 GiB 64 MiB 1.3 GiB 1.5 TiB 12.52 0.29 95 up osd.47 -15 21.19368 - 22 TiB 9.3 TiB 9.3 TiB 848 MiB 56 GiB 12 TiB 43.35 0.99 - host pileus 12 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 54 MiB 8.7 GiB 790 GiB 57.61 1.32 231 up osd.12 21 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 46 MiB 6.1 GiB 765 GiB 58.94 1.35 236 up osd.21 22 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 60 MiB 6.2 GiB 810 GiB 56.52 1.29 228 up osd.22 23 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 83 MiB 8.0 GiB 779 GiB 58.16 1.33 230 up osd.23 24 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 44 MiB 5.5 GiB 777 GiB 58.29 1.33 229 up osd.24 25 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 73 MiB 5.6 GiB 799 GiB 57.09 1.31 230 up osd.25 26 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 63 MiB 5.4 GiB 797 GiB 57.22 1.31 239 up osd.26 27 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 99 MiB 7.1 GiB 720 GiB 61.36 1.40 247 up osd.27 40 ssd 1.74660 1.00000 1.7 TiB 220 GiB 219 GiB 66 MiB 1013 MiB 1.5 TiB 12.31 0.28 96 up osd.40 41 ssd 1.74660 1.00000 1.7 TiB 221 GiB 220 GiB 63 MiB 981 MiB 1.5 TiB 12.36 0.28 97 up osd.41 42 ssd 1.74660 1.00000 1.7 TiB 228 GiB 227 GiB 98 MiB 999 MiB 1.5 TiB 12.73 0.29 101 up osd.42 43 ssd 1.74660 1.00000 1.7 TiB 227 GiB 226 GiB 99 MiB 866 MiB 1.5 TiB 12.67 0.29 100 up osd.43 -14 14.55518 - 15 TiB 8.6 TiB 8.5 TiB 582 MiB 45 GiB 6.0 TiB 59.02 1.35 - host stratus 28 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 5.5 GiB 756 GiB 59.41 1.36 239 up osd.28 29 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 44 MiB 4.7 GiB 759 GiB 59.27 1.35 230 up osd.29 30 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 68 MiB 5.0 GiB 753 GiB 59.57 1.36 232 up osd.30 31 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 64 MiB 6.2 GiB 778 GiB 58.26 1.33 229 up osd.31 32 hdd 1.81940 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 85 MiB 5.0 GiB 797 GiB 57.24 1.31 226 up osd.32 33 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 112 MiB 4.1 GiB 749 GiB 59.81 1.37 235 up osd.33 34 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 46 MiB 8.5 GiB 770 GiB 58.66 1.34 238 up osd.34 35 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 109 MiB 6.2 GiB 747 GiB 59.92 1.37 244 up osd.35 TOTAL 97 TiB 42 TiB 42 TiB 4.2 GiB 204 GiB 55 TiB 43.74 MIN/MAX VAR: 0.27/1.40 STDDEV: 21.24 # ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.040070", "last_optimize_started": "Thu Mar 13 14:59:05 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } -- Fachschaft I/1 Mathematik/Physik/Informatik der RWTH Aachen Thomas Schneider Campus Mitte: Augustinerbach 2a, 52062 Aachen Telefon: +49 241 80 94506 Informatikzentrum: Ahornstraße 55, Raum 4U17, 52074 Aachen Telefon: +49 241 80 26741 https://www.fsmpi.rwth-aachen.de # fio --ioengine=libaio --direct=1 --bs=16384 --iodepth=128 --rw=randread --norandommap --size=20G --numjobs=1 --runtime=300 --time_based --name=/dev/rbd/rbd/test /dev/rbd/rbd/test: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128 fio-3.25 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=35.0MiB/s][r=2241 IOPS][eta 00m:00s] /dev/rbd/rbd/test: (groupid=0, jobs=1): err= 0: pid=3832471: Thu Jan 30 19:55:46 2025 read: IOPS=7591, BW=119MiB/s (124MB/s)(34.9GiB/300892msec) slat (usec): min=2, max=665, avg= 8.93, stdev= 5.57 clat (usec): min=65, max=5330.5k, avg=16850.13, stdev=184056.88 lat (usec): min=82, max=5330.5k, avg=16859.23, stdev=184056.97 clat percentiles (usec): | 1.00th=[ 147], 5.00th=[ 180], 10.00th=[ 202], | 20.00th=[ 237], 30.00th=[ 269], 40.00th=[ 297], | 50.00th=[ 330], 60.00th=[ 367], 70.00th=[ 420], | 80.00th=[ 523], 90.00th=[ 1876], 95.00th=[ 6128], | 99.00th=[ 362808], 99.50th=[1098908], 99.90th=[3506439], | 99.95th=[4143973], 99.99th=[4731175] bw ( KiB/s): min= 768, max=478432, per=100.00%, avg=121819.01, stdev=95962.09, samples=600 iops : min= 48, max=29902, avg=7613.70, stdev=5997.62, samples=600 lat (usec) : 100=0.01%, 250=24.52%, 500=53.85%, 750=8.61%, 1000=1.56% lat (msec) : 2=1.84%, 4=3.46%, 10=2.34%, 20=1.12%, 50=0.91% lat (msec) : 100=0.40%, 250=0.31%, 500=0.13%, 750=0.18%, 1000=0.21% lat (msec) : 2000=0.34%, >=2000=0.20% cpu : usr=2.33%, sys=10.06%, ctx=1687235, majf=0, minf=523 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=2284212,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=119MiB/s (124MB/s), 119MiB/s-119MiB/s (124MB/s-124MB/s), io=34.9GiB (37.4GB), run=300892-300892msec

4 4

[RGW] Full replication gives stale recovering shard
by Gilles Mocellin 14 Mar '25

14 Mar '25

Hello Cephers, Since I didn't have any progress on my side, I share again my problem, hoping for some clues. --- Since I was not confident in my replication status, I've done a radosgw sync init one after the other, in both of my zones. Since then, there art stale recovering shards. Incremental replication seems OK. Cluster A sync status: realm 421c9bbd-83cc-4d85-a25a-ca847f225bfe (R) zonegroup 2c5ccf52-88be-4b2f-b4b9-c55f73bbacd1 (ZG) zone 3e3b0a2b-1864-4923-93a6-237a13b51594 (Z1) current time 2025-02-06T19:03:10Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: aefd4003-1866-4b16-b1b3-2f308075cd1c (Z2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [65,66,77] oldest incremental change not applied: 2025-02-06T20:02:53.441838+0100 [66] 111 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101, 102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] Cluster B sync status: realm 421c9bbd-83cc-4d85-a25a-ca847f225bfe (R) zonegroup 2c5ccf52-88be-4b2f-b4b9-c55f73bbacd1 (ZG) zone aefd4003-1866-4b16-b1b3-2f308075cd1c (Z2) current time 2025-02-06T19:03:37Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 3e3b0a2b-1864-4923-93a6-237a13b51594 (Z1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 7 shards behind shards: [58,59,60,62,64,66,67] oldest incremental change not applied: 2025-02-06T20:03:27.831651+0100 [60] 81 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,57,58,59,60,61,62,63,64,65,66,67,68,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123 ,124,126,127] Restarting full sync does nothing, except eventually adding recovering shards ! I also have some large omap objects in Z1.rgw.log pool. Where I see a lot of objects with names beginning by bucket.sync-status.*. What can I do to remove those recovering shards ? Some trimming (bilog ?), deleting objects in Z1.rgw.log pool ? I have some errors in radosgw-admin sync error list, but not really usefull : failed to sync bucket instance: (125) Operation canceled => A racing condition exists between writes to the same RADOS object. I also tried to rewrite all objects in buckets listed in thoses errors, with minimal size=0. Nothing changed. Thank you for reading. Regards, -- Gilles

1 1

Archive Sync Module does not add a Delete Marker when object is deleted
by motaharesdq＠gmail.com 14 Mar '25

14 Mar '25

Hello everyone, I was testing out the Archive sync module with 2 zones, one as the master zone and another one as a read-only archive tier zone. The sync policy is set with a standard hierarchy of Group/Flow/Pipe. Came across this scenario: overwriting an object in master zone does result in adding a version and changing latest on the archive zone, but deleting the object in the master zone does not add a delete marker in secondary, and the object remains as latest in the archive zone. This way even setting lifecycle wouldn't help deleting that, since it's not a non-current version. Is this the standard behaviour of archive module? And how should you delete something permanently? Kind Regards

1 0

How to (permanently) disable msgr v1 on Ceph?
by Stefan Kooman 14 Mar '25

14 Mar '25

Hi, For new clusters one of the first things I do is to disable messenger v1: ceph config set mon ms_bind_msgr1 false However, that is not enough, as a restart of the monitors will leave v1 enabled in the monmap. Why is that? That looks like a bug to me. Therefore I explicitly set the addrs and port of the monitors, i.e.: ceph mon set-addrs stefan-ceph-test01 $ip:3300 Be careful not to make a typo, that will ruin your day, see [1]. This immediately disables v1 and a new monmap is created. All good, however ... When a monitor is removed, say mon1, and added again, the new mon is added with both a v1 and a v2 address. I don't want that, I want to disable legacy v1 permanently. How do I do that? Adding: [mon] ms_bind_msgr1 = false to /etc/ceph/ceph.conf does not help either. When I performed a test this morning on a fresh 18.2.4 cluster (remove a mon, add it back again) to reproduce this issue the v1 was enabled again on two monitors (a third was left with only v2). Repeating the "ceph mon set-addrs" command removes the v1 binding again. But I would like to avoid messing around with this, triggering monmaps / elections, especially when making an error with this brings your cluster to a halt [1]. In what Ceph release will v1 be disabled by default on new clusters? Gr. Stefan [1] https://tracker.ceph.com/issues/54744

4 8

Re: ceph orch command not working anymore on squid (19.2.1)
by Eugen Block 13 Mar '25

13 Mar '25

(Don't drop the list in your responses) There's a lot going on in the cephadm.log, it's not clear which of these are relevant to the issue. This one looks bad though: ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not appear to be an IPv4 or IPv6 address Did you build this cluster with 19.2.1 or did you upgrade? Do you have the output from 'ceph orch host ls' lying around (I know, currently it's not working)? Are you using IPv6? I haven't seen this issue yet. Zitat von Moritz Baumann <mo(a)mo.homeip.net>: > Am 09.03.2025 um 15:22 schrieb Moritz Baumann: >> I dont want to sound stupid, but how do I get the log? >> >> Am 09.03.2025 um 11:11 schrieb Eugen Block: >>> ceph versions >> > > > ceph01[22]:~# cephadm shell -- ceph log last cephadm > Inferring fsid 4cfb1138-d2a5-11ef-87eb-1c34da408e52 > Inferring config > /var/lib/ceph/4cfb1138-d2a5-11ef-87eb-1c34da408e52/mon.ceph01/config > Not using image > 'f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d' > as it's not in list of non-dangling images with ceph=True label > 2025-02-25T08:42:54.590667+0000 mgr.ceph01.ombogc (mgr.180813) 1706 > : cephadm [INF] maintenance mode request for ceph05.inf.ethz.ch has > SET the noout group > 2025-02-25T08:43:38.380838+0000 mgr.ceph01.ombogc (mgr.180813) 1729 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:44:40.343420+0000 mgr.ceph01.ombogc (mgr.180813) 1762 > : cephadm [INF] Adjusting osd_memory_target on ceph04.inf.ethz.ch to > 33423M > 2025-02-25T08:44:40.860933+0000 mgr.ceph01.ombogc (mgr.180813) 1764 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:45:41.993564+0000 mgr.ceph01.ombogc (mgr.180813) 1797 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:46:42.758717+0000 mgr.ceph01.ombogc (mgr.180813) 1830 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:47:44.190946+0000 mgr.ceph01.ombogc (mgr.180813) 1864 > : cephadm [ERR] Unable to reach remote host ceph05.inf.ethz.ch. SSH > connection closed > Traceback (most recent call last): > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 245, in _execute_command > r = await conn.run(str(rcmd), input=stdin) > File "/lib/python3.9/site-packages/asyncssh/connection.py", line > 4250, in run > process = await self.create_process(*args, **kwargs) # type: ignore > File "/lib/python3.9/site-packages/asyncssh/connection.py", line > 4128, in create_process > chan, process = await self.create_session( > File "/lib/python3.9/site-packages/asyncssh/connection.py", line > 4018, in create_session > chan = SSHClientChannel(self, self._loop, encoding, errors, > File "/lib/python3.9/site-packages/asyncssh/channel.py", line > 1085, in __init__ > super().__init__(conn, loop, encoding, errors, window, max_pktsize) > File "/lib/python3.9/site-packages/asyncssh/channel.py", line 140, > in __init__ > self._recv_chan: Optional[int] = conn.add_channel(self) > File "/lib/python3.9/site-packages/asyncssh/connection.py", line > 1307, in add_channel > raise ChannelOpenError(OPEN_CONNECT_FAILED, > asyncssh.misc.ChannelOpenError: SSH connection closed > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper > return OrchResult(f(*args, **kwargs)) > File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper > return func(*args, **kwargs) > File "/usr/share/ceph/mgr/cephadm/module.py", line 2224, in > exit_host_maintenance > outs, errs, _code = self.wait_async( > File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async > return self.event_loop.get_result(coro, timeout) > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result > return future.result(timeout) > File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result > return self.__get_result() > File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in > __get_result > raise self._exception > File "/usr/share/ceph/mgr/cephadm/serve.py", line 1669, in _run_cephadm > python = await self.mgr.ssh._check_execute_command(host, cmd, addr=addr) > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 300, in > _check_execute_command > out, err, code = await self._execute_command(host, cmd, stdin, > addr, log_command) > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 253, in _execute_command > raise HostConnectionError(f'Unable to reach remote host {host}. > {str(e)}', host, address) > cephadm.ssh.HostConnectionError: Unable to reach remote host > ceph05.inf.ethz.ch. SSH connection closed > 2025-02-25T08:47:44.762770+0000 mgr.ceph01.ombogc (mgr.180813) 1865 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:47:55.631018+0000 mgr.ceph01.ombogc (mgr.180813) 1874 > : cephadm [INF] exit maintenance request has UNSET for the noout > group on host ceph05.inf.ethz.ch > 2025-02-25T08:48:47.623644+0000 mgr.ceph01.ombogc (mgr.180813) 1901 > : cephadm [INF] Adjusting osd_memory_target on ceph05.inf.ethz.ch to > 32911M > 2025-02-25T08:48:47.827026+0000 mgr.ceph01.ombogc (mgr.180813) 1903 > : cephadm [ERR] Detected invalid grafana certificate on host > ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '', > 'no start line')] > 2025-02-25T08:49:21.108932+0000 mgr.ceph01.ombogc (mgr.180813) 1925 > : cephadm [INF] Schedule redeploy daemon grafana.ceph05 > 2025-02-25T08:49:21.292879+0000 mgr.ceph01.ombogc (mgr.180813) 1926 > : cephadm [INF] Regenerating cephadm self-signed grafana TLS > certificates > 2025-02-25T08:49:21.328348+0000 mgr.ceph01.ombogc (mgr.180813) 1929 > : cephadm [INF] Deploying daemon grafana.ceph05 on ceph05.inf.ethz.ch > 2025-02-25T09:07:00.824841+0000 mgr.ceph01.ombogc (mgr.180813) 2461 > : cephadm [INF] Detected new or changed devices on ceph05.inf.ethz.ch > 2025-02-25T09:07:01.376245+0000 mgr.ceph01.ombogc (mgr.180813) 2462 > : cephadm [INF] Detected new or changed devices on ceph04.inf.ethz.ch > 2025-02-28T06:18:29.439683+0000 mgr.ceph01.ombogc (mgr.180813) > 137418 : cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE > ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not > appear to be an IPv4 or IPv6 address') > Traceback (most recent call last): > File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, > in communicate > req.parse_request() > File "/lib/python3.9/site-packages/cheroot/server.py", line 714, > in parse_request > success = self.read_request_line() > File "/lib/python3.9/site-packages/cheroot/server.py", line 815, > in read_request_line > scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) > File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit > _check_bracketed_host(bracketed_host) > File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host > ip = ipaddress.ip_address(hostname) # Throws Value Error if not > IPv6 or IPv4 > File "/lib64/python3.9/ipaddress.py", line 53, in ip_address > raise ValueError(f'{address!r} does not appear to be an IPv4 or > IPv6 address') > ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not > appear to be an IPv4 or IPv6 address > > 2025-02-28T06:18:29.441452+0000 mgr.ceph01.ombogc (mgr.180813) > 137419 : cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE > ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not > appear to be an IPv4 or IPv6 address') > Traceback (most recent call last): > File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, > in communicate > req.parse_request() > File "/lib/python3.9/site-packages/cheroot/server.py", line 714, > in parse_request > success = self.read_request_line() > File "/lib/python3.9/site-packages/cheroot/server.py", line 815, > in read_request_line > scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) > File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit > _check_bracketed_host(bracketed_host) > File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host > ip = ipaddress.ip_address(hostname) # Throws Value Error if not > IPv6 or IPv4 > File "/lib64/python3.9/ipaddress.py", line 53, in ip_address > raise ValueError(f'{address!r} does not appear to be an IPv4 or > IPv6 address') > ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not > appear to be an IPv4 or IPv6 address > > 2025-03-07T06:17:28.282574+0000 mgr.ceph01.ombogc (mgr.180813) > 447472 : cephadm [ERR] [07/Mar/2025:06:17:28] ENGINE > ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not > appear to be an IPv4 or IPv6 address') > Traceback (most recent call last): > File "/lib/python3.9/site-packages/cheroot/server.py", line 1281, > in communicate > req.parse_request() > File "/lib/python3.9/site-packages/cheroot/server.py", line 714, > in parse_request > success = self.read_request_line() > File "/lib/python3.9/site-packages/cheroot/server.py", line 815, > in read_request_line > scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri) > File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit > _check_bracketed_host(bracketed_host) > File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host > ip = ipaddress.ip_address(hostname) # Throws Value Error if not > IPv6 or IPv4 > File "/lib64/python3.9/ipaddress.py", line 53, in ip_address > raise ValueError(f'{address!r} does not appear to be an IPv4 or > IPv6 address') > ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not > appear to be an IPv4 or IPv6 address > > 2025-03-09T07:36:00.522250+0000 mgr.ceph01.ombogc (mgr.180813) > 536135 : cephadm [ERR] ALERT: Cannot stop active Mgr daemon, Please > switch active Mgrs with 'ceph mgr fail ceph01.ombogc' > Note: Warnings can be bypassed with the --force flag > Traceback (most recent call last): > File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper > return OrchResult(f(*args, **kwargs)) > File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper > return func(*args, **kwargs) > File "/usr/share/ceph/mgr/cephadm/module.py", line 2165, in > enter_host_maintenance > raise OrchestratorError( > orchestrator._interface.OrchestratorError: ALERT: Cannot stop active > Mgr daemon, Please switch active Mgrs with 'ceph mgr fail > ceph01.ombogc' > Note: Warnings can be bypassed with the --force flag

3 10

ceph orch command not working anymore on squid (19.2.1)
by Moritz Baumann 13 Mar '25

13 Mar '25

Hi, we are running a Ceph cluster (on almalinux-9 installed with cephadm) and after I failed the active ceph mgr with "ceph mgr fail "ceph01.xxxxx" to do a "ceph orch host maintenance enter ceph01" I got a: Error ENOENT: Module not found I then did a: ceph01[0]:~# ceph orch set backend "" ceph01[0]:~# ceph orch set backend "cephadm" Error ENOENT: Module not found ceph01[2]:~# ceph orch host ls Error ENOENT: No orchestrator configured (try `ceph orch set backend`) I do I get my orchestrator back to up and running? Best Moritz

3 6

Ceph with 3 nodes and hybrid storage policy: how to configure OSDs with different HDD and SSD sizes
by Daniel Vogelbacher 12 Mar '25

12 Mar '25

Hi, I want to setup a 3-node Ceph cluster with fault domain configured to "host". Each node should be equipped with: 6x SAS3 HDD 12TB 1x SAS3 SSD 7TB (should be extended to 2x7 later) The ceph configuration should be size=3, min_size=2. All nodes are connected with 2x10Gbit (LACP). I want to use different CRUSH rules for different pools. CephFS and low priority/IO VMs stored on RBD should use only HDD drives with default replication CRUSH rule. For high priority VMs, I want to create another RBD data pool which uses a modified CRUSH replication rule: |# Hybrid storage policy rule hybrid { ruleset 2 type replicated step take ssd step chooseleaf firstn 1 type host step emit step take hdd step chooseleaf firstn -1 type host step emit } | |For pools using this hybrid rule, PGs are stored on one SSD (primary) and two HDD (secondary) devices. But these have different sizes in my hardware setup. What happens with the remaining disk space (12-7=5) 5GB on the secondary devices? Is it just unusable, or will ceph use it for other pools with default replication? In any case, I don't bother about these 5GB, just want to know how it works. For the above setup, can you recommend any important configuration settings and should I modify the OSD weighting? Thanks. |-- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher

3 7

RGW multisite metadata sync issue
by Vahideh Alinouri 11 Mar '25

11 Mar '25

Hi guys, My Ceph release is Quincy 17.2.5. I need to change the master zone to decommission the old one and upgrade all zones. I have separated the client traffic and sync traffic in RGWs, meaning there are separate RGW daemons handling the sync process. I encountered an issue when trying to sync one of the zones in the zonegroup. The data sync is proceeding fine, but I have an issue with the metadata sync. It gets stuck behind on a shard. Here is the output from radosgw-admin sync status: metadata sync syncing full sync: 1/64 shards full sync: 135 entries to sync incremental sync: 63/64 shards metadata is behind on 1 shard behind shards: [0] In the RGW log, I see this error: 2024-12-15T21:30:59.641+0000 7f6dff472700 1 beast: 0x7f6d2f1cf730: 172.19.66.112 - s3-cdn-user [15/Dec/2024:21:30:59.641 +0000] "GET /admin/log/?type=data&id=56&marker=00000000000000000000%3A00000000000000204086&extra-info=true&rgwx-zonegroup=7c01d60f-88c6-4192-baf7-d725260bf05d HTTP/1.1" 200 44 - - - latency=0.000000000s 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 I’ve tried the following steps: - Changed the PG number of the metadata pool to force a rebalance, but everything was fine. - Ran metadata sync init and tried to run it again. - Restarted RGW services in both the zone and the master zone. - Created a user in the master zone to ensure metadata sync works, which was successful. - Checked OSD logs but didn’t see any specific errors. - Attempted to list metadata in the pool using rados ls -p s3-cdn-dc07.rgw.meta, but got an empty result. - Compared the code for listing OMAP keys between Quincy and Squid versions; there were no specific changes. I’m looking for any advice or suggestions to resolve this issue.

3 6

Sometimes PGs inconsistent (although there is no load on them)
by Marianne Spiller 11 Mar '25

11 Mar '25

Dear list, I'm currently maintaining several Ceph (prod) installations. One of them consists of 3 MON hosts and 6 OSD hosts hosting 40 OSDs in total. And there are 5 separate Proxmox-Hosts - they only host the VMs and use the storage provided by Ceph, but they are not part of Ceph. The worst case happened: due to an outage, all these hosts crashed hardly the same time. Last week, I began to restart (only the Ceph hosts; Proxmox servers are still down). Ceph was very unhappy with the situation as a whole - one OSD host (and its 6 OSDs) is completely gone, some hardware issues (33 OSDs left, networking, PSU, I'm working on it) and 73 out of 129 PGs inconsistent. Meanwhile, the overall status of the cluster is "HEALTHY" again. But nearly every day, one or two PGs get damaged. Never on the same OSDs. And there is no traffic on the storage as the virtualization hosts are not running. I see no further reason in the logs: everything is fine, scrub starts and leaves one or more PGs damaged. Repairing them is successful, but maybe next night, another PG is stuck. Do you have hints to investigate this any further? I would love to understand more before starting the Proxmox cluster again. Using Ceph 18.2.4 (Proxmox packages). Thanks a lot, Marianne

4 3

Move block.db to new ssd
by Roland Giesler 11 Mar '25

11 Mar '25

I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions. The new 4TB SSD's I have split into volumes with: # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 & osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this: # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami I then take osd4 out of the cluster and stop it. Then I do this: # ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami Thereafter I do this to load the changes: # ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 --path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami After that the configuration change has reverted to what it was before. I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 What am I missing to get these changes to be permanent? regards Roland

6 13

Submitting proposals for Ceph Day London 2025
by Ivan Clayson 10 Mar '25

10 Mar '25

Hi, I was interested in applying to give a talk at the upcoming Ceph day in London this June (04/06/25) but I can't find an email or form to submit my proposal to. Is there an email I can send my proposal to? Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N269 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

2 2

Ceph cluster unable to read/write data properly and cannot recover normally.
by 小小 10 Mar '25

10 Mar '25

Hi, I'm facing a critical issue with my Ceph cluster. It has become unable to read/write data properly and cannot recover normally. What steps should I take to resolve this? [root@ceph-node1 ~]# ceph -s cluster: id: 76956086-25f5-445d-a49e-b7824393c17b health: HEALTH_WARN 1 pools have many more objects per pg than average 102131233/124848552 objects misplaced (81.804%) Reduced data availability: 40 pgs inactive Degraded data redundancy: 6402821/124848552 objects degraded (5.128%), 11 pgs degraded, 31 pgs undersized services: mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 mgr: ceph-node2(active), standbys: ceph-node3, ceph-node1 osd: 6 osds: 6 up, 6 in; 169 remapped pgs data: pools: 7 pools, 216 pgs objects: 41.62 M objects, 2.6 TiB usage: 18 TiB used, 12 TiB / 30 TiB avail pgs: 12.500% pgs unknown 6.019% pgs not active 6402821/124848552 objects degraded (5.128%) 102131233/124848552 objects misplaced (81.804%) 143 active+clean+remapped 27 unknown 15 active+undersized+remapped 13 active+clean 9 undersized+degraded+peered 4 undersized+peered 2 active+undersized+degraded 1 active+undersized 1 active+remapped+backfilling 1 active+clean+remapped+scrubbing+deep io: recovery: 1.3 MiB/s, 20 objects/s， [root@ceph-node1 ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 30.00000 - 30 TiB 18 TiB 17 TiB 125 GiB 302 GiB 12 TiB 58.88 1.00 - root default -3 10.00000 - 10 TiB 5.9 TiB 5.8 TiB 38 GiB 101 GiB 4.1 TiB 59.11 1.00 - host ceph-node1 0 hdd 5.00000 0.21257 5.0 TiB 2.3 TiB 2.2 TiB 25 GiB 41 GiB 2.7 TiB 46.21 0.78 61 osd.0 3 hdd 5.00000 0.13638 5.0 TiB 3.6 TiB 3.5 TiB 13 GiB 60 GiB 1.4 TiB 72.02 1.22 84 osd.3 -5 10.00000 - 10 TiB 6.2 TiB 6.1 TiB 42 GiB 105 GiB 3.8 TiB 62.40 1.06 - host ceph-node2 1 hdd 5.00000 0.13644 5.0 TiB 3.6 TiB 3.5 TiB 11 GiB 61 GiB 1.4 TiB 71.99 1.22 95 osd.1 4 hdd 5.00000 0.16666 5.0 TiB 2.6 TiB 2.6 TiB 31 GiB 44 GiB 2.4 TiB 52.81 0.90 104 osd.4 -7 10.00000 - 10 TiB 5.5 TiB 5.4 TiB 45 GiB 96 GiB 4.5 TiB 55.11 0.94 - host ceph-node3 2 hdd 5.00000 0.16666 5.0 TiB 3.5 TiB 3.4 TiB 22 GiB 62 GiB 1.5 TiB 70.19 1.19 79 osd.2 5 hdd 5.00000 0.21664 5.0 TiB 2.0 TiB 1.9 TiB 23 GiB 34 GiB 3.0 TiB 40.04 0.68 100 osd.5 TOTAL 30 TiB 18 TiB 17 TiB 125 GiB 302 GiB 12 TiB 58.88 MIN/MAX VAR: 0.68/1.22 STDDEV: 13.38 Thanks~

4 4

Created no osd(s) on host, already created?
by Gustavo Garcia Rondina 10 Mar '25

10 Mar '25

Hello list, We have a Ceph cluster with two management nodes and six data nodes. Each data node has 28 HDD disks. One disk recently failed in one of the nodes, corresponding to osd.2. To replace the disk, we took the osd.2 out, stopped it, and after a few days removed it, basically: ceph osd out osd.2 ceph osd ok-to-stop osd.2 Once Ok to stop, then: ceph orch daemon stop osd.2 Once stopped: ceph osd crush remove osd.2 ceph auth del osd.2 ceph osd rm osd.2 Then checked `ceph osd tree`, `ceph orch ps`, `ceph -s`, and all confirmed that the OSD was gone. We then proceeded to physically replace the failed drive on the node. It was /dev/sdac before, and after replacing it (hot swap) the system identified it as /dev/sdah. I zapped it on the node with `sgdisk --zap-all /dev/sdah` and, back on the management node, I could see that the disk was now showing up and marked as available with `ceph orch device ls`. Then I proceeded to add a new OSD to that disk with: `ceph orch daemon add osd node-osd1:/dev/sdah`, which then failed with: Created no osd(s) on host server-osd1; already created? Which was rather strange, because total OSDs was still showing as 1 less, no new OSD was showing up in `ceph osd tree`. What was even odder is that the disk that was showing as available in `ceph orch device ls` now shows as *not* available, and looking at lsblk's output in the node it seems that it was populated by Ceph: # lsblk /dev/sdah NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdah 66:16 0 16.4T 0 disk └─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0 lvm Any hints on how to proceed to get the OSD added back with this disk? Thank you for any suggestions! - Gustavo

4 10

DC 4 EC 4+5 with 4 servers make sense?
by Torkil Svensgaard 10 Mar '25

10 Mar '25

Hi We currently have a 3 DC setup with 6 HDD servers in each of 3 DCs, running replica 3 and EC 4+5 pools. This works fine for mostly dormant data but of course not so well for other stuff requiring low latency etc. Mostly big RBDs shared by kernel NFS with a bit of CephFS. We want to start migrating some of these big RBD backed NFS shares to NVMe. We currently have 3 E3.S servers, one in each DC, with an EC 4+5 pool with DC->OSD crush selection. We are wondering how to cheapest increase redundancy short term since we do not have enterprise money. With what we have we can lose 1 DC (aka host in this case) and 1 OSD and still be online, but there's nowhere to backfill and some hardware problems, like a faulty mainboard or CPU can take down an entire "DC". Ideally we would have several more hosts in each DC so we had DC->Host->OSD but that's not an option. I am thinking it could make sense to add just 1 more server in a 4th DC and keep the 4+5 rule as is, simply giving us a buffer of 1 more failed host before we hit problems. Thoughts? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 0

cephadm bootstrap failed with docker
by farhad kh 08 Mar '25

08 Mar '25

hi , for trying deploy cluster with cephadm version 19.2.1 and using docker version 28.0.1 i get this error : ------- # cephadm --image opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 bootstrap --mon-ip 10.248.35.143 --registry-json /root/reg.json --allow-fqdn-hostname --initial-dashboard-user admin --initial-dashboard-password P@ssw0rd1404 --dashboard-password-noupdate --ssh-user cephadmin Verifying ssh connectivity using standard pubkey authentication ... Adding key to cephadmin@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... docker (/bin/docker) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 15d9eaee-fbe0-11ef-ad63-005056a83619 Verifying IP 10.248.35.143 port 3300 ... Verifying IP 10.248.35.143 port 6789 ... Mon IP `10.248.35.143` is in CIDR network `10.248.35.0/24` Mon IP `10.248.35.143` is in CIDR network `10.248.35.0/24` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Logging into custom registry. Pulling custom registry login info from /root/reg.json. Pulling container image opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1... Non-zero exit code 125 from /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 -e NODE_NAME=opcpmfpsksa0403 opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 --version ceph: stderr docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown ceph: stderr ceph: stderr Run 'docker run --help' for more information RuntimeError: Failed command: /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 -e NODE_NAME=opcpmfpsksa0403 opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 --version *************** Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically. To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous broken installation, users must use the following command to completely delete the broken cluster: > cephadm rm-cluster --force --zap-osds --fsid <fsid> for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster *************** Deleting cluster with fsid: 15d9eaee-fbe0-11ef-ad63-005056a83619 Traceback (most recent call last): File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 2628, in _rollback File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 446, in _default_image File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 2763, in command_bootstrap File "/tmp/tmpfe1bt8s9.cephadm.build/app/cephadmlib/container_types.py", line 429, in run File "/tmp/tmpfe1bt8s9.cephadm.build/app/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 -e NODE_NAME=opcpmfpsksa0403 opkbhfpsksp0101.p.fnst/ceph/ceph:v19.2.1 --version During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 5581, in <module> File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 5569, in main File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 2657, in _rollback File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 4391, in _rm_cluster File "/tmp/tmpfe1bt8s9.cephadm.build/app/__main__.py", line 4317, in get_ceph_cluster_count FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/ceph' ----- for that i checked everything lik docker version also containerd and runc and just find this log in journalctl -u docker : ------- Mar 02 06:23:00 opcpmfpsksa0403 systemd[1]: Starting Docker Application Container Engine... Mar 02 06:23:00 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:00.533468075Z" level=info msg="Starting up" Mar 02 06:23:00 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:00.535018237Z" level=info msg="OTEL tracing is not configured, using no-op tracer provider" Mar 02 06:23:00 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:00.566901059Z" level=info msg="[graphdriver] using prior storage driver: overlay2" Mar 02 06:23:00 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:00.567653709Z" level=info msg="Loading containers: start." Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.325500958Z" level=info msg="Loading containers: done." Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.344752401Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2 Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.344892780Z" level=info msg="Docker daemon" commit=bbd0a17 containerd-snapshotter=false storage-driver=overlay2 version=28.0.1 Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.344968676Z" level=info msg="Initializing buildkit" Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.393298555Z" level=info msg="Completed buildkit initialization" Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.401141531Z" level=info msg="Daemon has completed initialization" Mar 02 06:23:01 opcpmfpsksa0403 dockerd[18363]: time="2025-03-02T06:23:01.401227515Z" level=info msg="API listen on /run/docker.sock" Mar 02 06:23:01 opcpmfpsksa0403 systemd[1]: Started Docker Application Container Engine. Mar 08 05:42:50 opcpmfpsksa0403 dockerd[18363]: time="2025-03-08T05:42:50.468402014Z" level=error msg="copy stream failed" error="reading from a closed fifo" stream=stderr Mar 08 05:42:50 opcpmfpsksa0403 dockerd[18363]: time="2025-03-08T05:42:50.468433232Z" level=error msg="copy stream failed" error="reading from a closed fifo" stream=stdout Mar 08 05:42:51 opcpmfpsksa0403 dockerd[18363]: time="2025-03-08T05:42:51.103965419Z" level=error msg="Handler for POST /v1.48/containers/40b1295b9067eb570d01b1509c59593f29e7ad61fb61e8ed4a82166441d52d53/start returned error: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown" -------- we use linux oracle 9.5 with kernel 5.15.0-305.176.4.el9uek.x86_64, we searched anywhere but we can't understand that what happen, anybody know how to can resolve that ? or what's happening?

2 1

BlueStore and BlueFS warnings after upgrade to 19.2.1
by Nicola Mori 07 Mar '25

07 Mar '25

Dear Ceph users, after upgrading from 19.2.0 to 19.2.1 (via cephadm) my cluster started showing some warnings never seen before: 29 OSD(s) experiencing slow operations in BlueStore 13 OSD(s) experiencing stalled read in db device of BlueFS I searched for these messages but didn't find much. I also noticed that when browsing the CephFS folders (using the kernel module for the client) sometimes the client gets stuck for a long time before showing the folder content; however I don't know if this can be related to the above warnings. I'd need help to understand what's going on and eventually how to troubleshoot it. Thanks, Nicola

4 8

ceph orch is not working
by xadhoom76＠gmail.com 06 Mar '25

06 Mar '25

Hi , we have issue using ceph orch , it doesn't send any command for real. We try to restart or redeploy but nothing works This happened downgrading from 17.2.8 to 17.2.6 Thanks

3 3

RGW Cloud-Sync Configuration Help
by Mark Selby 06 Mar '25

06 Mar '25

I am trying to use the RGW Cloud-Sync module to replicate data from an on-prem RGW cluster to AWS. This is a multi-tenant cluster with many tenants each who have multiple buckets. I only want to replicate a single bucket from a single tenant and am having problems creating a tier config that does this. It seems that the tier config requires a default connection and will try and replicate all buckets that do not have a specific profile. I have used sync groups, flows, and pipes with “standard” multisite replication but it does not seem to be applicable to cloud sync. Does anyone know how to configure cloud-sync to only replicate a single bucket from a single tenant. Here are some details on my current tier-config. I want to replicate the bucket roadrunner from the tenant wile to the beepbeep bucket in aws. This the only bucket I want cloud-sync to replicate. Any and all help is greatly appreciated. "tier_config": { "connection_id": "aws-default", "connections": [ { "id": "aws-default", "endpoint": https://s3.amazonaws.com, "access_key": "XXXXXXXXXXXXXXXXXXXX", "secret": "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY" }, { "id": "aws-cloud-sync", "endpoint": https://s3.amazonaws.com, "access_key": "XXXXXXXXXXXXXXXXXXXX", "secret": "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY" }, ], "profiles": [ { "connection_id": "aws-cloud-sync", "source_bucket": "wile:roadrunner", "target_path": "beepbeep" } ] } -- Mark Selby Sr Linux Administrator, The Voleon Group mselby(a)voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/.

1 0

Module 'devicehealth' has failed
by Alex from North 06 Mar '25

06 Mar '25

Hello everybody! Running 19.2.0 faced an issued still cannot struggle. And this is Module 'devicehealth' has failed: Expecting value: line 1 column 2378 (char 2377) in MGR log I see Mar 04 12:48:07 node2.ec.mts ceph-mgr[3821449]: Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in serve self._do_serve() File "/usr/share/ceph/mgr/mgr_module.py", line 549, in check return func(self, *args, **kwargs) File "/usr/share/ceph/mgr/devicehealth/module.py", line 355, in _do_serve if self.db_ready() and self.enable_monitoring: File "/usr/share/ceph/mgr/mgr_module.py", line 1286, in db_ready return self.db is not None File "/usr/share/ceph/mgr/mgr_module.py", line 1298, in db self._db = self.open_db() File "/usr/share/ceph/mgr/mgr_module.py", line 1265, in open_db if not self.pool_exists(self.MGR_POOL_NAME): File "/usr/share/ceph/mgr/mgr_module.py", line 1106, in pool_exists pools = [p['pool_name'] for p in self.get('osd_map')['pools']] File "/usr/share/ceph/mgr/mgr_module.py", line 1416, in get obj = json.loads(obj) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 2378 (char 2377) Unfortunately, I do now know how to turn devicehealth off as it is forbidden using standard commands line ceph device monitoring off. Any hint is appreciated. Thank in advance, guys!

2 9

Guidance on Ceph Squid v19.20 Production Deployment – Best Practices and Requirements
by Altrel Fero 06 Mar '25

06 Mar '25

Dear Ceph Community, We need guidance on the best practices for deployment. Specifically, we would like to know: What are the recommended methods for Ceph deploying in Kubernetes? What are the best practices for a production deployment? What network configurations are recommended for a stable and efficient deployment? Is Ceph Squid v19.20 recommended for production use, or should we consider an alternative version? What are the recommended hardware and resource requirements for this setup? Looking forward to your valuable suggestions and insights Best regards, P.S. Altrel Fero ZyBiSys Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

Error Removing Zone from Zonegroup in Multisite Setup
by Mahnoosh Shahidi 06 Mar '25

06 Mar '25

Hi everyone, We have a Ceph multisite setup with a zonegroup containing three zones: one master and two secondary zones, one of which is designated as an archive. After recently upgrading to Ceph 19.2.1, I attempted to remove the archive zone from the zonegroup following the steps in the Ceph documentation. I ran radosgw-admin zonegroup remove --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> from the cluster containing the archive zone, followed by radosgw-admin period update --commit --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> However, the period update --commit command returns the error: ERROR: zonegroup <zonegroup_id> does not contain zone id <zone_id> The zone appears correctly in the zonegroup before the removal, and I expected the archive zone to be successfully removed with the period update committing without errors. Is there an additional step required, or could this be a bug related to Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you! Best regards, Mahnoosh

3 3

Re: Free space
by quaglio＠bol.com.br 06 Mar '25

06 Mar '25

1 0

mgr module 'orchestrator' is not enabled/loaded
by Alex from North 06 Mar '25

06 Mar '25

Hi all! Faced sad situation I don't know where to dig to. That is why I am here again, hoping for hints. Situation: all the "orch" commands aint available and give error in mgr log mgr.server reply reply (95) Operation not supported Module 'orchestrator' is not enabled/loaded (required by command 'orch ls'): use `ceph mgr module enable orchestrator` to enable it Mar 04 09:03:22 node2.ec.mts conmon[1238298]: 2025-03-04T09:03:22.842+0000 7f0659aed640 -1 mgr.server reply reply (95) Operation not supported Module 'orchestrator' is not enabled/loaded (required by command 'orch ls'): use `ceph mgr module enable orchestrator` to enable it Of course I've tried to enable it as suggested, but it is already enable (always-on). Tried to fail/systemd restart - all for nothing. As I understood, orchestrator is responsible for listening as well on port 8765 to give Prometheus needed info. So, I also have "devicehealth" error, as nothing is on port 8765 where Prom tries to knock. Thinking that maybe redeploy would help and having no possibility to use orchestrator, I tried to deploy new manager manually - the same story, orch is not available. Maybe orchestrator module in a crash loop? Where to check logs? There is nothing in /var/log/ceph/cephadm.log and nothing in mgr log. I am really stuck. Any hint appreciated. Thanks in advance.

3 3

Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute
by Florian Haas 06 Mar '25

06 Mar '25

Hello everyone, I'm seeing some behaviour in CephFS that strikes me as unexpected, and I wonder if others have thoughts about it. Consider this scenario: * Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, CephFS client is running kernel 5.15.0-133-generic. * CephFS is mounted to /mnt, using a CephX identity that has rwp permissions. * CephFS is using a single data pool named cephfs_data. * In the CephFS root directory, there is a subdirectory named "restricted". That subdirectory currently has a single file, "test2.bin", of 8 MiB. There is no other data in the data pool. I can verify this with the following rados command: rados -n client.cephfs -p cephfs_data ls 10000000003.00000001 10000000003.00000000 Now I run this, from the root of my mounted CephFS: cd /mnt setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted I realise this is naughty, because there is currently a file named /mnt/restricted/test2.bin, and I'm not supposed to set layout attributes on a non-empty directory. However, the command does succeed, and I am able to read back the ceph.dir.layout xattr: getfattr -n ceph.dir.layout restricted # file: restricted ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data pool_namespace=restricted" If at this stage I try to read RADOS objects in the "restricted" namespace, I get an empty list: rados -p cephfs_data -N restricted ls (empty output) Now I move my test file out of the restricted subdirectory and back in: mv /mnt/restricted/test2.bin /mnt/ sync mv /mnt/test2.bin /mnt/restricted/ sync No change: rados -p cephfs_data -N restricted ls (empty output) Next, I try moving the file out, and *copying* it back in: mv /mnt/restricted/test2.bin /mnt/ sync cp /mnt/test2.bin /mnt/restricted/ sync Now, I do see objects in the "restricted" namespace: rados -p cephfs_data -N restricted ls 100000001fd.00000001 100000001fd.00000000 Also, if I create a *new* file in the "restricted" subdirectory, then its RADOS objects do end up in the correct namespace: dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M rados -p cephfs_data -N restricted ls 100000001fe.00000000 100000001fe.00000002 100000001fe.00000001 100000001fd.00000001 100000001fd.00000000 By contrast, here's what happens to a non-empty *file,* when I try to set ceph.file.layout: dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin setfattr: test4.bin: Directory not empty So that fails, somewhat expectedly (although the error message is odd). In summary, there are two things that confuse me here: 1. Why does setting ceph.dir.layout.pool_namespace on a non-empty directory succeed, when setting ceph.file.layout.pool_namespace on a non-empty file fails (and even confusingly with a "Directory not empty" message)? 2. Considering that setting ceph.dir.layout.pool_namespace on a non-empty directory does succeed, why does mv'ing a file to a directory with a different pool_namespace, and then mv'ing it back, not result in its RADOS objects moving to the other namespace? So I'm curious: is this a bug (or two), or have I been misunderstanding what's actually the expected behaviour? Cheers, Florian

4 5

Mixed cluster with AMD64 and ARM64 possible?
by Robert Sander 06 Mar '25

06 Mar '25

Hi, just a quick question: Is a mixed cluster with nodes running AMD64 and ARM64 CPUs possible? Is the cephadm orchestrator able to manage such a cluster? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 1

cephfs healthy but mounting it some data cannot be accessed
by xadhoom76＠gmail.com 06 Mar '25

06 Mar '25

Hi, we have access and we can mount cephfs volume but for some reason data in it can be listes but when we try to copy some files from it , some can be and some hangs Is there a way to fix it or skip the files we cannot access ? Best regards

1 0

upgrading ceph without orchestrator
by xadhoom76＠gmail.com 05 Mar '25

05 Mar '25

Is it possible to upgrade mgr without using ceph orch ?

2 1

Upgrade: 5 pgs have unknown state; cannot draw any conclusions
by xadhoom76＠gmail.com 04 Mar '25

04 Mar '25

Hi, we have a cluster in which we have lost the osds containg the 5 pgs. How to proceed to get cluster back working ? of course we will lose data..... We cannot upgrade or downgrade with unknown state. Best regards

1 0

Severe Latency Issues in Ceph Cluster
by Ramin Najjarbashi 04 Mar '25

04 Mar '25

Hi We are currently facing severe latency issues in our Ceph cluster, particularly affecting read and write operations. At times, write operations completely stall, leading to significant service degradation. Below is a detailed breakdown of the issue, our observations, and the mitigation steps we have taken so far. We would greatly appreciate any insights or suggestions. Incident Timeline & Observations: Users reported extreme slowness in write and read operations. Network Issues: Packet drops and a high number of TCP retransmits were identified. TCP Connection Delays: Many TCP connections were stuck in the TIME-WAIT state. Initial Suspicions: We first suspected high list_bucket requests on RGW, but log analysis disproved this. Resharding Efforts: Some high-latency buckets were resharded, but this had minimal impact. HTTP 503 Errors: Indicating request overload and service instability. Earlier Related Incidents: Network Congestion: Routing issues previously caused congestion in the Provision network. RGW Downtime: Several RGWs went down, requiring removal from HAProxy. NIC Troubleshooting: A faulty NIC was replaced, but errors persisted. Troubleshooting & Mitigation Attempts: Network & Traffic Management: Investigated logs and blocked unnecessary list_bucket requests. Disabled HAProxy to isolate Ceph internal traffic → Ceph internals worked fine, but external requests still caused issues. RGW-Specific Troubleshooting: Restarted RGW instances → Temporary 20-minute improvement, but latency returned. Found an RGW node with incomplete configurations → Fixed, but issue resurfaced. TCP & Network Investigations: Adjusted tcp_mem settings → No substantial improvement. packet analysis showed a high number of duplicate packets (DUP ACKs). Observed high TCP retransmit errors and many TIME-WAIT connections. Assistance Requested: Could high TCP retransmits & TIME-WAIT connections indicate a deeper network issue affecting RGW writes? Are there recommended debugging techniques for tracing TCP-related issues in a Ceph RGW environment? Would tweaking KeepAlived settings help in case of incorrect VIP failover behavior? Could an underlying Ceph metadata issue be disproportionately affecting write operations? Any guidance, recommendations, or debugging steps would be immensely helpful. Thank you for your time and support! Best regards, Ramin

7 8

When 18.2.5 will be released?
by wojiaowugen 04 Mar '25

04 Mar '25

Hi, everyone, It's an honor to ask questions here. can I ask when 18.2.5 will be released?

2 1

Replace OSD while cluster is recovering?
by Gustavo Garcia Rondina 04 Mar '25

04 Mar '25

Hello list, We have a Ceph cluster (17.2.6 quincy) with 2 admin nodes and 6 storage nodes, each storage node connected to a JBOD enclosure. Each enclosure houses 28 HDD disks of 18 TB size, totaling 168 OSDs. The pool that houses the majority of the data is erasure-coded (4+2). We have recently had one disk failure, which brought one OSD down: # ceph osd tree | grep down 2 hdd 16.49579 osd.2 down 0 1.00000 This OSD is out of the cluster, but we haven't replaced it physically yet. The problem that we are facing is that the cluster was not in the best shape when this OSD failed. Currently we have the following: ################################################ cluster: id: <redacted> health: HEALTH_ERR 1026 scrub errors Possible data damage: 18 pgs inconsistent 2137 pgs not deep-scrubbed in time 2137 pgs not scrubbed in time services: mon: 5 daemons, quorum xyz-admin1,xyz-admin2,xyz-osd1,xyz-osd2,xyz-osd3 (age 17M) mgr: xyz-admin2.sipadf(active, since 17M), standbys: xyz-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 168 osds: 167 up (since 44h), 167 in (since 6w); 220 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 2137 pgs objects: 448.54M objects, 1.0 PiB usage: 1.6 PiB used, 1.1 PiB / 2.7 PiB avail pgs: 134404830/2676514497 objects misplaced (5.022%) 1902 active+clean 191 active+remapped+backfilling 26 active+remapped+backfill_wait 15 active+clean+inconsistent 2 active+remapped+inconsistent+backfilling 1 active+remapped+inconsistent+backfill_wait io: recovery: 597 MiB/s, 252 objects/s progress: Global Recovery Event (6w) [=========================...] (remaining: 5d) ################################################ I have noticed the number of active+clean increasing (was ~1750 two days ago), and objects misplaced very slowly decreasing. My question is, should I wait until recovery is complete, then repair the 18 damaged pg, and only then replace the disk? My thinking is that replacing the disk will trigger more backfilling which will slow down the recovering even more. Another question, should I disable scrubbing while the recovery is not finalized? Thank you for any insights you may be able to provide! - Gustavo

4 5

March 3rd Ceph Steering Committee Meeting Notes
by Laura Flores 03 Mar '25

03 Mar '25

Hi all, Here is a summary of what was discussed at today's CSC Meeting: - Deciding on the best way for a community vote on the name for U release - Tentacle Kickoff PR merged - https://tracker.ceph.com/issues/69953 being investigated by fs team - v18.2.5 status: Working on including a few outstanding fixes Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

[Cephfs] Can't get snapshot under a subvolume
by ceph＠gurkan.in 02 Mar '25

02 Mar '25

Hello all, We're getting a "Operation not permitted" error while trying to create a snapshot on the client. It is somehow related to previously-asked Pacific issue mentioned here: https://www.spinics.net/lists/ceph-users/msg67908.html We are on squid (19.2.1) and the given workaround seems to be only temporary. Anyway, here are some details about the issue: Cephx side: ``` client.mount-update-production key: *** caps: [mds] allow rws fsname=production-cephfs path=/volumes/_nogroup/update-production caps: [mon] allow r fsname=production-cephfs caps: [osd] allow rw tag cephfs data=production-cephfs ``` FS flags: ``` ~> sudo ceph fs lsflags production-cephfs joinable allow_snaps allow_multimds_snaps ``` I got the path via following command: ``` ~> sudo ceph fs subvolume getpath production-cephfs update-production /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f ``` And mounted to client with mentioned key: ``` ~> mount | grep ceph mount-update-production@00000000-0000-0000-0000-000000000000.production-cephfs=/volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f on /opt/aptly/.aptly type ceph (rw,noatime,name=mount-update-production,secret=<hidden>,acl,mon_addr=[multiple mon addresses here]) ``` Everything works flawlessly. Yet, when we want to get a snapshot: ``` /opt/aptly/.aptly/pool/9c/0b ~> sudo mkdir .snap/test mkdir: cannot create directory ‘.snap/test’: Operation not permitted ``` If I mount the whole FS with admin rights and run `setfattr -n ceph.dir.subvolume -v 0` for _all_ directories of following path one by one: /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f, then it is solving the issue. But it does not last; not sure what is triggering it again, but if I try snapshotting anything next day, getting the same "Operation not permitted" error again. Is there something I can try, or am I missing something obvious? Thanks, Gürkan

3 5

Schrödinger's Server
by Tim Holloway 01 Mar '25

01 Mar '25

Ack. Another fine mess. I was trying to clean things up and the process of tossing around OSD's kept getting me reports of slow responses and hanging PG operations. This is Ceph Pacific, by the way. I found a deprecated server that claimed to have an OSD even though it didn't show in either "ceph osd tree" or the dashboard OSD list. I suspect that a lot of the grief came from it attempting to use resources that weren't always seen as resources. I shut down the server's OSD (removed the daemon using ceph orch), then foolishly deleted the server from the inventory without doing a drain first. Now cephadmin hates me (key not found), and there are still an MDS and MON listed as ceph orch ls daemons even after I powered the host off. I cannot do a ceph orch daemon delete because there's no longer an IP address available to the daemon delete, and I cannot clear the cephadmin queue: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'ceph07.internal.mousetech.com' Any suggestions?

3 9

Reef: draining host with mclock quicker than expected
by Eugen Block 01 Mar '25

01 Mar '25

Hi *, for a change, I just wanted to share some positive experience with mclock. On a customer cluster (recently upgraded to Reef), I drained one host with 23 HDD OSDs (1,1 TB size). And to my surprise, it took less than an hour until all PGs were backfilled. The OSDs weren't too full, I believe something around 1,6 TB had to be moved away from this host. But the recovery rates were quite nice, especially compared to what other users on Reef have been reporting. I saw values between several hundred MB to 2 or 3 GB per second. Most of the configs are default, especially with regards to mclock there haven't been any changes. Although the general recommendation is still to switch back to wpq, I haven't done that yet in this cluster. I usually wait until I encounter any issue pointing to mclock. But since this was a positive experience, I'll keep mclock enabled until otherwise required. Have a nice weekend! Eugen

1 0