May 2025 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

SuSE stops building Ceph packages for its distributions
by Robert Sander 15 Aug '25

15 Aug '25

Hi, SuSE will not build even the Ceph client packages for its distributions any more: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org/threa… https://www.suse.com/releasenotes/x86_64/SUSE-SLES/15-SP6/index.html#jsc-PE… This means that this distribution cannot consume Ceph storage any more, if no other party will build these packages. If you are using SuSE Linux Enterprise Server or OpenSuSE as a client for Ceph storage you need to plan for this. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 4

Announcing go-ceph v0.32.0
by Sven Anderson 12 Aug '25

12 Aug '25

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.32.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

Squid 19.2.1 dashboard javascript error
by Chris Palmer 09 Aug '25

09 Aug '25

Firstly thank you so much for the 19.2.1 release. Initial testing suggests that the blockers that we had in 19.2.0 have all been resolved, so we are proceeding with further testing. We have noticed one small problem in 19.2.1 that was not present in 19.2.0 though. We use the older-style dashboard (mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false). The problem happens on the Dashboard screen when health changes to WARN. If you click on WARN you get a small empty dropdown instead of the list of warnings. A javascript error is logged, and using browser inspection there is the additional bit of info that it happens in polyfill: 2025-02-07T15:59:00.970+0000 7f1d63877640 0 [dashboard ERROR frontend.error] (https://<redacted>:8443/#/dashboard): NG0901 Error: NG0901 at d.find (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3342365) at le.ngDoCheck (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3173112) at Qe (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225586) at bt (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225341) at cs (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225051) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259043) at jf (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3266563) at S1 (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259790) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259801) at fg (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3267248) Also, after this happens, no dropdowns work again until the page is forcibly refreshed. Environment is RPM install on Centos 9 Stream. I've created issue [0]. Thanks, Chris [0] https://tracker.ceph.com/issues/69867 <https://tracker.ceph.com/issues/69867?next_issue_id=69865&prev_issue_id=90>

4 9

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid)
by Edouard FAZENDA 25 Jul '25

25 Jul '25

Dear all, I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2. Here some cluster logs : 8/5/25 09:00 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:50 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:33 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:33 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:30 AM[INF]overall HEALTH_OK 8/5/25 08:22 AM[INF]Cluster is now healthy 8/5/25 08:22 AM[INF]Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:22 AM[WRN]Health check update: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:20 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:03 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:03 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:00 AM[INF]overall HEALTH_OK #ceph fs status cephfs - 50 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818 0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0 POOL TYPE USED AVAIL cephfs_metadata metadata 2630M 2413G cephfs_data data 12.7T 3620G STANDBY MDS cephfs.node3.vdicdn MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) # ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18 }, "mds": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32 } } #ceph orch ps --daemon-type mds NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764 mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18 root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default # ceph mds stat cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ? I saw the folloing issue : <https://tracker.ceph.com/issues/66948> Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ( <https://github.com/ceph/ceph/pull/60838> squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ) maybe related ? Thanks for the help 😊 Best Regards, Edouard Fazenda. <https://www.csti.ch/> Swiss Cloud Provider Edouard Fazenda Technical Support <https://www.csti.ch/> Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40 www.csti.ch

4 6

Re: "BUG: soft lockup" with MDS
by Janek Bevendorff 10 Jul '25

10 Jul '25

Yes, they are older Dell PowerEdges. I have to check whether there's newer firmware, but we've been running Ceph for years without these problems. I checked the logs on the host on which I had a lockup just an hour ago, but there's nothing besides the expected hardreset messages. There are two older watchdog messages, but they are from March: -------------------------------------------------------------------------------- SeqNumber = 2089 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:16:03 Message = The watchdog timer expired. RawEventData = 0x03,0x00,0x02,0x33,0xFB,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- SeqNumber = 2088 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:06:41 Message = The watchdog timer expired. RawEventData = 0x02,0x00,0x02,0x01,0xF9,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- I grepped the logs of another host where it happened, but couldn't find any watchtdog messages there. I believe it's also unlikely that suddenly all MDS hosts (we have five active, five hot standbys, and one cold standby) start having hardware issues. I also ran a memtest on one of the hosts last week and couldn't find anything there either. On 16/04/2025 15:14, Anthony D'Atri wrote: > Curious, are your systems Dells? If so you might see some improvement from running DSU to update all the firmware. It might also be illuminating to run `racadm lclog view` > >> On Apr 16, 2025, at 8:32 AM, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: >> >> Hi, >> >> Since the latest Reef update I have the problem that some of my hosts suddenly go into a state where all CPUs are stuck in kernel mode causing all daemons on that host to become unresponsive. When I connect to the IPMI console, I see a lot of messages like: >> >> watchdog: BUG: soft lockup - CPU#8 stuck for 47s! [cron:868840] >> >> (it's basically a list of all processes running on the machine). >> >> Usually, this resolves itself after several minutes, but sometimes I have to hardreset the host. When this happens, all daemons are marked as down and I cannot interact with the host at all. I don't know what causes this but, I think it happens primarily on the hosts where my MDS run and it seems to be triggered by events such as cluster rebalances, MDS restarts, or just randomly. >> >> I found a few reports about similar issues on the bug tracker and mailing list, but they are all very unspecific, unanswered, or more than 6 years old. >> >> Is there any way I can debug this? I upgraded to Squid already, but that didn't solve the problem. I also had massive issues with this during the upgrade. Particularly at the end when the MDS were upgraded, I had constant struggles with it. I had to set the noout flag and then literally sit next to it to resume the upgrade every few minutes until it finally went through, because random MDS hosts went intermittently dark all the time. >> >> All hosts run Ubuntu 22.04 with kernel 6.8.0. >> >> Any ideas? Thanks! >> Janek >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 13

Squid and Ganesha and ingress options?
by Nigel Williams 08 Jul '25

08 Jul '25

We deployed a new Cephadm Squid cluster (400 OSDs) and are happy with everything except this time we tried Ganesha again with the ingress option (HAproxy), but soon we ran into a failed daemon event. And there is a github issue for it: https://github.com/nfs-ganesha/nfs-ganesha/issues/1158 Our questions are: Is a hotfix pending for this github issue? Is anyone using Ganesha with or without ingress, is it reliable for you? How is the relative performance between Ganesha and kernel NFS? We seem to try Ganesha every release but return to the usual kernel NFS server setup because Ganesha is unreliable, we would be interested to know what others in the community think about the Ganesha option. thanks.

2 1

Adding OSD nodes
by Sinan Polat 21 Jun '25

21 Jun '25

Hello, I am currently managing a Ceph cluster that consists of 3 racks, each with 4 OSD nodes. Each node contains 24 OSDs. I plan to add three new nodes, one to each rack, to help alleviate the high OSD utilization. The current highest OSD utilization is 85%. I am concerned about the possibility of any OSD reaching the osd_full_ratio threshold during the rebalancing process. This would cause the cluster to enter a read-only state, which I want to avoid at all costs. I am planning to execute the following commands: ceph orch host add new-node-1 ceph orch host add new-node-2 ceph orch host add new-node-3 ceph osd crush move new-node-1 rack=rack-1 ceph osd crush move new-node-2 rack=rack-2 ceph osd crush move new-node-3 rack=rack-3 ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 1 ceph config set osd osd_recovery_sleep 0.1 ceph orch apply osd --all-available-devices Before proceeding, I would like to ask if the above steps are safe to execute in a cluster with such high utilization. My main concern is whether the rebalancing could cause any OSD to exceed the osd_full_ratio and result in unexpected failures. Any insights or advice on how to safely add these nodes without impacting cluster stability would be greatly appreciated. Thanks!

7 7

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

How to clear the "slow operations" warning?
by Jan Kasprzak 18 Jun '25

18 Jun '25

Hello, Ceph users, TL;DR: how to clear/acknowledge the following warning from ceph -s? health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD. # ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore When I restart osd.33 using "systemctl restart ceph-osd(a)33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

11 15

Re: v18.2.7 Reef released
by Erwin Bogaard 16 Jun '25

16 Jun '25

It looks like there is an issue with the package-based 18.2.7 release, when upgrading from 18.2.6 on el9. There seems to be a new (unfulfilled) dependency that prevents the packages from installing: Problem 1: cannot install the best update candidate for package librgw2-2:18.2.6-0.el9.x86_64 - problem with installed package librgw2-2:18.2.6-0.el9.x86_64 - nothing provides libcrypto.so.3(OPENSSL_3.4.0)(64bit) needed by librgw2-2:18.2.7-0.el9.x86_64 from ceph-reef This issue prevents almost all ceph packages from installing. 18.2.6 is installed and fully functional on this machine, so it seems something happened between 18.2.6 and 18.2.6. Hope someone can confirm this is happening at their severs as well and/or confirm or deny this is a packaging problem. Regards, Erwin

11 17

MDS Repeatedly Crashing/Restarting - Unable to get CephFS Active
by Kasper Rasmussen 05 Jun '25

05 Jun '25

Ceph Version: 18.2.7 I've just migrated to cephadm, and upgrade from pacific to reef 18.2.7 last week. All successful except some minor issues with BlueFS Spillover Today the MDS of a specific fs refuse to start, and the ceph orch ps shows the daemons with status "error". I have three other cephfs that still works(though I haven't tested if they can fail over.) I've restartet the MDSs - No luck (the selected MDS just start/crash in a loop until it gives up) I've deployed 2 new MDSs - No luck same issue In all scenarios I see in ceph fs status, that a MDS is chosen. FS status goes to "replay" or "replay(laggy)" On the host with the MDS I see the MDS container just crashes after way less than 5 mins.. And status reported by ceph orch ps is error. (btw - mds_beacon_grace has been set to 360) I've managed to get a good 500 lines of log out with info like this: << ----------------- LOG EXAMPLE START ----------------- >> -7> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient: _check_auth_tickets -6> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2025-05-19T16:04:32.845551+0000) -5> 2025-05-19T16:05:02.860+0000 7f673e3c1640 10 monclient: get_auth_request con 0x5616e9616c00 auth_method 0 -4> 2025-05-19T16:05:02.916+0000 7f673dbc0640 10 monclient: get_auth_request con 0x5616e7422800 auth_method 0 -3> 2025-05-19T16:05:02.968+0000 7f673d3bf640 10 monclient: get_auth_request con 0x5616f5eac800 auth_method 0 -2> 2025-05-19T16:05:02.972+0000 7f6736bb2640 2 mds.0.cache Memory usage: total 574800, rss 343772, heap 207124, baseline 182548, 0 / 7535 inodes have caps, 0 caps, 0 caps per inode -1> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f67333ab640 time 2025-05-19T16:05:03.680495+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7f67406e6d2c] 2: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb] 3: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe] 4: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745] 5: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c] 6: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd] 7: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e] 8: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1] 9: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a] 10: clone() 0> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 *** Caught signal (Aborted) ** in thread 7f67333ab640 thread_name:mds-log-replay ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f674004bbf0] 2: /lib64/libc.so.6(+0x8bf5c) [0x7f6740098f5c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x178) [0x7f67406e6d86] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb] 7: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe] 8: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745] 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c] 10: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd] 11: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e] 12: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1] 13: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a] 14: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. << ----------------- LOG EXAMPLE END ----------------- >> But to be honest, out of all those lines, I don't know what to provide (all +500 might be a bit to much) I really need this FS back online, so help will be very much appreciated

4 14

Help in upgrading CEPH
by Sergio Rabellino 05 Jun '25

05 Jun '25

Dear list, we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow: > # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too). It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous: > "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag. All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us. Any hints ? thanks in advance. -- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>

5 16

Write throughput increasing on CephFS metadata pool
by Adrien Georget 05 Jun '25

05 Jun '25

Hi, We have a CephFS Quincy cluster (17.2.7) used by Openshift for PVC provisioning (and lot of snapshots) with ceph-csi driver. On the metadata pool, we observed a continuous increase of the write throughput activity from 10MB/s just after a restart of the MDS to +200MB/s after 2 weeks. We first thought it was related to the snapshots deletion activity (lot of creations and deletions every day) because we saw the same kind of increase for the number of Strays. We tried to evaluate strays using the recursive scrub [1], Strays decrease from 450K to 350K but no impact on the metadata write throughput. Only a MDS restart makes the write throughput back to normal. I saw this very familiar issue https://tracker.ceph.com/issues/53542 and I'm wondering if this has been fixed in Quincy? I tried to increase "mds_log_events_per_segment" and "mds_log_max_segments" but nothing helped. As I answered in the tracker, I also observed large 4M objects in objecter_requests. Anything we can do to prevent this and avoid restarting the MDS every week? If we don't do that, PVC operations stay blocked on Openshift side. Cheers, Adrien [1] https://docs.ceph.com/en/quincy/cephfs/scrub/#evaluate-strays-using-recursi… The MDS perf dump if it can helps : { "AsyncMessenger::Worker-0": { "msgr_recv_messages": 452889868, "msgr_send_messages": 210932005, "msgr_recv_bytes": 381554341884, "msgr_send_bytes": 31897539519042, "msgr_created_connections": 5392, "msgr_active_connections": 213, "msgr_running_total_time": 121964.941943159, "msgr_running_send_time": 85193.342708858, "msgr_running_recv_time": 40274.249544605, "msgr_running_fast_dispatch_time": 15899.702855460, "msgr_send_messages_queue_lat": { "avgcount": 210932003, "sum": 140532.121408536, "avgtime": 0.000666243 }, "msgr_handle_ack_lat": { "avgcount": 110313532, "sum": 120.513184629, "avgtime": 0.000001092 } }, "AsyncMessenger::Worker-1": { "msgr_recv_messages": 393334601, "msgr_send_messages": 170841678, "msgr_recv_bytes": 307649876710, "msgr_send_bytes": 23017109865662, "msgr_created_connections": 12712, "msgr_active_connections": 218, "msgr_running_total_time": 93136.265793790, "msgr_running_send_time": 63604.143090523, "msgr_running_recv_time": 30881.136884349, "msgr_running_fast_dispatch_time": 12763.126679902, "msgr_send_messages_queue_lat": { "avgcount": 170841678, "sum": 153574.265735747, "avgtime": 0.000898927 }, "msgr_handle_ack_lat": { "avgcount": 81594240, "sum": 65.480936528, "avgtime": 0.000000802 } }, "AsyncMessenger::Worker-2": { "msgr_recv_messages": 481956104, "msgr_send_messages": 301348856, "msgr_recv_bytes": 349838769013, "msgr_send_bytes": 26783354654792, "msgr_created_connections": 34394, "msgr_active_connections": 215, "msgr_running_total_time": 108807.807447203, "msgr_running_send_time": 74577.666991790, "msgr_running_recv_time": 41392.667149426, "msgr_running_fast_dispatch_time": 13971.142479134, "msgr_send_messages_queue_lat": { "avgcount": 301348847, "sum": 138655.862052108, "avgtime": 0.000460117 }, "msgr_handle_ack_lat": { "avgcount": 156375610, "sum": 144.168097813, "avgtime": 0.000000921 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "queue_len": 0, "complete_latency": { "avgcount": 121564865, "sum": 160601.142778677, "avgtime": 0.001321114 } }, "finisher-PurgeQueue": { "queue_len": 0, "complete_latency": { "avgcount": 27338289, "sum": 8785.146872773, "avgtime": 0.000321349 } }, "mds": { "request": 210298249, "reply": 210297956, "reply_latency": { "avgcount": 210297956, "sum": 1148337.019419176, "avgtime": 0.005460523 }, "slow_reply": 15, "forward": 0, "dir_fetch": 49843551, "dir_commit": 65281430, "dir_split": 6, "dir_merge": 17, "inodes": 5113817, "inodes_top": 2080514, "inodes_bottom": 2263929, "inodes_pin_tail": 769374, "inodes_pinned": 2141653, "inodes_expired": 538454667, "inodes_with_caps": 413837, "caps": 457168, "subtrees": 2, "traverse": 262113637, "traverse_hit": 245551660, "traverse_forward": 0, "traverse_discover": 0, "traverse_dir_fetch": 1642726, "traverse_remote_ino": 0, "traverse_lock": 140893, "load_cent": 392174, "q": 0, "exported": 0, "exported_inodes": 0, "imported": 0, "imported_inodes": 0, "openino_dir_fetch": 407633, "openino_backtrace_fetch": 7011755, "openino_peer_discover": 0, "root_rfiles": 11618539, "root_rbytes": 11106041675168, "root_rsnaps": 7576, "scrub_backtrace_fetch": 7009788, "scrub_set_tag": 0, "scrub_backtrace_repaired": 0, "scrub_inotable_repaired": 0, "scrub_dir_inodes": 1526050, "scrub_dir_base_inodes": 3, "scrub_dirfrag_rstats": 1526047, "scrub_file_inodes": 5483733, "handle_inode_file_caps": 0, "ceph_cap_op_revoke": 5076186, "ceph_cap_op_grant": 60192925, "ceph_cap_op_trunc": 658164, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_flush_ack": 6, "handle_client_caps": 70349707, "handle_client_caps_dirty": 21547109, "handle_client_cap_release": 11468771, "process_request_cap_release": 83631146 }, "mds_cache": { "num_strays": 347989, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 19259955, "strays_enqueued": 20275628, "strays_reintegrated": 469, "strays_migrated": 0, "num_recovering_processing": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "recovery_started": 484, "recovery_completed": 484, "ireq_enqueue_scrub": 2, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 23, "ireq_fragstats": 0, "ireq_inodestats": 0 }, "mds_log": { "evadd": 148604638, "evex": 148687537, "evtrm": 148686957, "ev": 4640, "evexg": 0, "evexd": 580, "segadd": 5143165, "segex": 5143142, "segtrm": 5143128, "seg": 166, "segexg": 0, "segexd": 14, "expos": 2655552712972056, "wrpos": 2655556750517207, "rdpos": 2574717722176006, "jlat": { "avgcount": 23948614, "sum": 1026045.477642289, "avgtime": 0.042843626 }, "replayed": 86959 }, "mds_mem": { "ino": 5091233, "ino+": 537662754, "ino-": 532571521, "dir": 385511, "dir+": 48473070, "dir-": 48087559, "dn": 5114534, "dn+": 564239550, "dn-": 559125016, "cap": 457198, "cap+": 296065469, "cap-": 295608271, "rss": 21947020, "heap": 223516 }, "mds_server": { "dispatch_client_request": 265192774, "dispatch_server_request": 0, "handle_client_request": 210298249, "handle_client_session": 44456292, "handle_peer_request": 0, "req_create_latency": { "avgcount": 19045215, "sum": 243874.341572325, "avgtime": 0.012805019 }, "req_getattr_latency": { "avgcount": 13244393, "sum": 122372.637049784, "avgtime": 0.009239580 }, "req_getfilelock_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_link_latency": { "avgcount": 108, "sum": 0.089209875, "avgtime": 0.000826017 }, "req_lookup_latency": { "avgcount": 42591076, "sum": 61153.611919024, "avgtime": 0.001435831 }, "req_lookuphash_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupino_latency": { "avgcount": 6119, "sum": 12.051986148, "avgtime": 0.001969600 }, "req_lookupname_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupparent_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupsnap_latency": { "avgcount": 11789, "sum": 1.916955778, "avgtime": 0.000162605 }, "req_lssnap_latency": { "avgcount": 174999, "sum": 59.628852495, "avgtime": 0.000340738 }, "req_mkdir_latency": { "avgcount": 386018, "sum": 5323.027974258, "avgtime": 0.013789584 }, "req_mknod_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_mksnap_latency": { "avgcount": 11784, "sum": 4342.774640033, "avgtime": 0.368531452 }, "req_open_latency": { "avgcount": 1719118, "sum": 2957.961208770, "avgtime": 0.001720627 }, "req_readdir_latency": { "avgcount": 49303877, "sum": 153658.488737674, "avgtime": 0.003116559 }, "req_rename_latency": { "avgcount": 1021332, "sum": 4173.200914205, "avgtime": 0.004086037 }, "req_renamesnap_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_rmdir_latency": { "avgcount": 355348, "sum": 6609.629921457, "avgtime": 0.018600442 }, "req_rmsnap_latency": { "avgcount": 11476, "sum": 4647.171801363, "avgtime": 0.404947002 }, "req_rmxattr_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setattr_latency": { "avgcount": 951092, "sum": 60918.695526648, "avgtime": 0.064051317 }, "req_setdirlayout_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setfilelock_latency": { "avgcount": 42236246, "sum": 87407.285106607, "avgtime": 0.002069485 }, "req_setlayout_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setxattr_latency": { "avgcount": 20450454, "sum": 351779.625311317, "avgtime": 0.017201555 }, "req_symlink_latency": { "avgcount": 2, "sum": 0.002286490, "avgtime": 0.001143245 }, "req_unlink_latency": { "avgcount": 18777510, "sum": 39044.878444925, "avgtime": 0.002079342 }, "cap_revoke_eviction": 0, "cap_acquisition_throttle": 0, "req_getvxattr_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "mds_sessions": { "session_count": 545, "session_add": 26598, "session_remove": 26053, "sessions_open": 545, "sessions_stale": 0, "total_load": 11076, "average_load": 20, "avg_session_uptime": 37409721, "mdthresh_evicted": 0 }, "mempool": { "bloom_filter_bytes": 10309112, "bloom_filter_items": 10309112, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_items": 0, "bluefs_bytes": 0, "bluefs_items": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "buffer_anon_bytes": 284731060, "buffer_anon_items": 5959770, "buffer_meta_bytes": 616, "buffer_meta_items": 7, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 81480, "osdmap_items": 2396, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "mds_co_bytes": 16277298996, "mds_co_items": 260156681, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "op_active": 11, "op_laggy": 0, "op_send": 253406063, "op_send_bytes": 81014361089067, "op_resend": 0, "op_reply": 253406052, "oplen_avg": { "avgcount": 253406063, "sum": 451959672 }, "op": 253406063, "op_r": 58269533, "op_w": 195136530, "op_rmw": 0, "op_pg": 0, "osdop_stat": 64974069, "osdop_create": 11505623, "osdop_read": 1275145, "osdop_write": 34882891, "osdop_writefull": 1169760, "osdop_writesame": 0, "osdop_append": 0, "osdop_zero": 2, "osdop_truncate": 4, "osdop_delete": 79836916, "osdop_mapext": 0, "osdop_sparse_read": 0, "osdop_clonerange": 0, "osdop_getxattr": 56700621, "osdop_setxattr": 17474636, "osdop_cmpxattr": 0, "osdop_rmxattr": 0, "osdop_resetxattrs": 0, "osdop_call": 0, "osdop_watch": 0, "osdop_notify": 0, "osdop_src_cmpxattr": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_other": 683839, "linger_active": 0, "linger_send": 0, "linger_resend": 0, "linger_ping": 0, "poolop_active": 0, "poolop_send": 0, "poolop_resend": 0, "poolstat_active": 0, "poolstat_send": 0, "poolstat_resend": 0, "statfs_active": 0, "statfs_send": 0, "statfs_resend": 0, "command_active": 0, "command_send": 0, "command_resend": 0, "map_epoch": 685274, "map_full": 0, "map_inc": 19880, "osd_sessions": 96, "osd_session_open": 96, "osd_session_close": 0, "osd_laggy": 0, "omap_wr": 78737471, "omap_rd": 99825478, "omap_del": 4893217 }, "oft": { "omap_total_objs": 4, "omap_total_kv_pairs": 105334, "omap_total_updates": 80286291, "omap_total_removes": 72648222 }, "purge_queue": { "pq_executing_ops": 0, "pq_executing_ops_high_water": 1559, "pq_executing": 0, "pq_executing_high_water": 64, "pq_executed": 20275628, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "val": 0, "max": 104857600, "get_started": 0, "get": 1328180571, "get_sum": 935681276002, "get_or_fail_fail": 0, "get_or_fail_success": 1328180571, "take": 0, "take_sum": 0, "put": 1328180571, "put_sum": 935681276002, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-objecter_bytes": { "val": 21734389, "max": 104857600, "get_started": 0, "get": 0, "get_sum": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "take": 253406063, "take_sum": 81151894362586, "put": 253406052, "put_sum": 81151872628197, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-objecter_ops": { "val": 11, "max": 1024, "get_started": 0, "get": 0, "get_sum": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "take": 253406063, "take_sum": 253406063, "put": 253406052, "put_sum": 253406052, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-write_buf_throttle": { "val": 0, "max": 3758096384, "get_started": 0, "get": 20275628, "get_sum": 2230259884, "get_or_fail_fail": 0, "get_or_fail_success": 20275628, "take": 0, "take_sum": 0, "put": 1242717, "put_sum": 2230259884, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-write_buf_throttle-0x56401344c0a0": { "val": 0, "max": 3758096384, "get_started": 0, "get": 148604638, "get_sum": 80839025768070, "get_or_fail_fail": 0, "get_or_fail_success": 148604638, "take": 0, "take_sum": 0, "put": 23948617, "put_sum": 80839025768070, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } } }

2 2

radosgw-admin bucket radoslist returns 1.8 billion entries for a bucket with 17m objects
by Boris 04 Jun '25

04 Jun '25

Hi, I am in the process of checking orphan objects and the radoslist list grep over 300GB while the rados ls list was only 50GB. After some investigation I identified a saw that one bucket fell out with a lot of objects in the radoslist part. I did some sorting on the output file and just used the marker part to check if something is wrong: These are the last two lines after sorting: 16145185 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297644274.205 1882589930 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.1805769661.3772 The top bucket has roughtly the same amount ob objects in the index, but the other bucket only hast 17million object in the index. I stopped the search for orphan objects and pulled a fresh `radosgw-admin bucket radoslist --bucket BUCKET > file`. Then I sorted that list and ran a uniq -c against it and sorted that list again. The unique list ist now 34million objects long and there are 32026 entries that appear more than once and there are entries that 68849 times. 68849 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.1805769661.3772__shadow_server/download/XXX/YYYY.2~k9UuEdtUkAHkikXz1ONc8UzgwKryYco.6_1 How can I clean that up? Or is this just a bug in the radosgw-admin bucket radoslist? The bucket has 401 shards and got "versioned: true" and "versioning_enabled: true" -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 2

Make backup to another server.
by Nmz 02 Jun '25

02 Jun '25

Hello everyone, I came to Ceph from ZFS world. In ZFS I can use snapshots as backup way. Keep it localy or send to another server/pool. Is it possible to do it the same using Ceph? Right now in my cluster I use RBD and CephFS. RBD data structure: Ceph pools pool proxmox_metadata Rep SSD pool proxmox_data EC SSD storage.cfg rbd: ceph_rbd content images data-pool proxmox_data pool proxmox_metadata Ceph pools pool proxmox_data_1_ssd_other_hdd R 1 SSD 2 HDD storage.cfg rbd: ceph_rbd_1ssd_hdd content images pool proxmox_data_1_ssd_other_hdd CephFS data structure: Ceph pools pool cephfs_metadata Rep SSD pool cephfs_data Rep SSD pool web_data EC SSD pool db_data EC SSD pool data EC HDD # ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data web_data db_data data ] subvolumegroup | data_pool | subvolume home | web_data | nmz .... db | db_data | sql persoanl | data | Video .... As you can see subvolumes use specific ceph pools. For every subvolume I use snapshots. Ceph backup server will work in cold mode, single node, not 24/7. What is the best way to send data to backup server ? I want to use incremental backup sync. Can I achieve it ? Thanks

2 1

Mon Election Leader goes unresponsive with 100% cpu usage on fn_monstore
by Wesley Dillingham 01 Jun '25

01 Jun '25

Tracker issue made here with some additional details: https://tracker.ceph.com/issues/71501 Cluster version 18.2.4 I came to assist with a non-functional cluster which had OSDs erroneously --force purged and led to multiple (6) degraded + inactive PGs (4 remaining shards in a 4+2) and 1 remapped+incomplete PG (3 shards in a 4+2) In an effort to restore order to the cluster and get backfill working: the degraded PGs shared a common primary OSD and that OSD was restarted. Additionally min_size was dropped from 5 to 4 for this pool (a temporary measure while the cluster recovered). This caused the inactive degraded PGs to go active and start their backfill. The PGs steadily worked on backfill for a few hours. Near immediately after the final of the 6 degraded PGs finished its backfill the monitor quorum broke and the cluster became unresponsive. In this state 2 of the 5 mons showed 100% cpu usage. In an attempt to fix the mon quorum some combination of monitor service restart attempts occurred and ultimately all ceph services were brought down in order to isolate the MON issue. As the situation stands currently any combination of starting 3 MONs (quorum eligible at 3) causes the lowest MON (the to-be-leader) to hit a 100% cpu with fn_monstore thread and render unresponsive the admin socket of that mon only (the other 2 respond via admin socket and show either probing or electing). I have captured logs of the to-be-leader mon (its daemon started first) with debug_mon = 20 and debug_ms = 20 (I should probably recapture with debug_paxos = 20). The pegged cpu only occurs once the third MON is started (seems to be an election issue). The MON has been allowed to run for hours with no progress in that state. Eventually it seems the leader goes into the (leader) state (Claims "I win" in the logs) but the other 2 mons continue their election cycle. In states of probing or electing still. I have verified NTP clock is fine and sync'd to same source on all the mons and connectivity between the mons at both ports is functional. The situation the PGs were in during the subsequent MON fault leads me to believe the problem is more complex than typical monitor election issues. Also of note the backing disk of the to-be-leader MON is mostly idle. At this point I am interested in taking backups of all the mon stores and injecting a modified single-mon monmap in a mon and seeing if we can get back up but am also concerned that that single mon will be the de-facto leader and also be unresponsive. Interested in any suggestions from the wider community. Thanks! Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes(a)wesdillingham.com

2 5

19.2.2: Warning,Smartctl has received an unknown argument (error code -22)
by Harry G Coin 31 May '25

31 May '25

Using 19.2.2, we notice under cluster/osds/'device health' on the dashboard, for all osds no matter the server: Warning Smartctl has received an unknown argument (error code -22). You may be using an incompatible version of smartmontools. Version >= 7.0 of smartmontools is required to successfully retrieve data. That error code resolves to 'unknown attribute' in the smartctl docs. However, the same result occurs whether the drive is HGST, Seagate, or Western Digital. "State of Health" is always "Stale" "Life Expectancy" is always "n/a> 6 weeks" Of course the 'diskprediction_local" module has been broken for over a year, as it requires a no-longer-distributed rev of a sub-package, but that shouldn't stop the smartctl command from normal operations. Any ideas? Thanks! Harry Coin

2 2

How does Ceph delete objects / data on disk?
by Stefan Kooman 29 May '25

29 May '25

Hi, I would like to get a better understanding at what happens under the hood when an object is deleted in a Ceph cluster. Is there a (detailed) write up of the deletion process? I would like to be able to answer questions like: - How can we be sure that after the successful completion of for example a "rbd rm $image" command it is impossible for the data to be restored. - How do we know we don't get deleted data returned that was marked as deleted (but never overwritten with new data) Can we be sure? Or are there ways to perform a recovery procedure (as a storage admin with access to all components of the cluster), i.e. gather information about higher level Ceph abstractions (rbd, file in CephFS, RGW bucket) that have existed in the past, gather information about what objects where in use, do CRUSH calculation (and consult upmaps) to obtain OSDs involved, Query RocksDB / Bluestore to obtain information about LBA on disk, read the data and in this way reconstruct the object(s). Pathological example: rbd rm $image (successful deletion) ceph pause immediately after that Do the recovery procedure noted above How likely is it that we would be able to recovery the data? I understand it's hard to put a number on it, so pointers to where (parts) of this process is described will help me getting a better understanding. I'm also aware that it highly depends on how busy a cluster would be, what type of media involved, if the discard option of Ceph and disks is in use, and many other factors. Thanks, Stefan

4 3

Adding OSD with separate DB via "ceph orch daemon add osd"
by Ryan Rempel 28 May '25

28 May '25

I'm expanding a small Ceph cluster from 4 nodes to 5 nodes. The new node is a bit more sophisticated than the others, since it has some SSD storage that I'd like to use for DB+WAL (which I haven't done before, it has just been rotational disks). I'm using cephadm for orchestration, and normally add osds via "ceph orch daemon add osd". I prefer to add the osds in this "manual" way (rather than "ceph orch apply" with a spec) mainly because my infrastructure is not uniform (for better or worse, I'm working with hardware that becomes available in different ways over time, as I gradually upgrade things and add things). Looking at this page: https://docs.ceph.com/en/squid/cephadm/services/osd/ ... it isn't entirely clear to me whether it's possible to specify a separate DB device when using the "ceph orch daemon add osd" procedure. There is a description of how to do it with a service spec, but how you would specify the DB device for "ceph orch daemon add osd" does not appear to be described. So, my first question is whether it's possible to specify a separate DB via "ceph orch daemon add osd"? If not, I'll need to explore the service spec approach. I suppose I can use the "unmanaged: true" option in the spec (to keep it as "manual" as possible). The remaining puzzle is how to use the SSD as DB+WAL for more than one OSD. At this point, the SSD is the raw device — I haven't done anything with it manually in LVM or whatever. In the service spec description above, I see that there is a "db_slots" key. So, I suppose that I could specify the "whole" SSD and provide for the number of slots? However, I don't necessarily want every slot to be the same size (because of my unfortunately heterogeneous hardware). So, I also see that there is a "block_db_size" and "block_wal_size". But it's unclear how this relates to "db_slots" — which one would determine how the SSD is sliced up? I'd actually be happy to pre-slice the SSD (e.g. with LVM) and then directly specify which SSD slice is the DB+WAL for which OSD, if that's a feasible approach. Though, I'd still be interested in knowing whether I need to set something for "block_db_size" and "block_wal_size", or whether it's enough to just actually make a certain size of LVM volume available for DB+WAL. Normally I'd just experiment, but that might be disruptive to the working cluster. I guess I could at least turn off rebalancing while I try things out? The other documentation I'm now reading is the documentation for ceph-volume, which appears to be related: https://docs.ceph.com/en/squid/ceph-volume/lvm/batch/ It mentions, for instance, things like db_slots and block_db_size. The implication is that db_slots is an alternative to block_db_size — that you wouldn't specify both, for instance. I'm also reading the ceph-volume docs for "prepare". I suppose if I find that more suitable, it might be possible to "prepare" and OSD with ceph-volume and then "adopt" it with cephadm? Well, just writing the email has given me a bit more clarity about things to try, but I'd certainly be happy for any guidance. Ryan Rempel Director of Information Technology Canadian Mennonite University

3 4

radosgw-admin: the RGW management nightmare
by Michel Jouvin 28 May '25

28 May '25

Hi, We have a Ceph cluster hosting ~10 RGW realms and in our experience, radosgw-admin is making the Ceph admins life very hard... The opposite experience to other Ceph management tools. In addition to the lack of verifications before applying modifications that leads to the issues describe in another thread were you can have a configuration that cannot be commited and at the same time cannot be fixed, I have the feeling that one of the major issue is that radosgw-admin predates the rgw/zonegroup/zone paradigm and manages these 3 different kinds of objects as completely independent from the others. It is made worst by the fact that radosgw-admin defines a default object in each category, used implicitely (and slilently) if you don't select one specific object. The consequence of these 2 problems is that if you modify a zone, you can endup modifying the zonegroup of another realm than the one the zone belongs to (as there is no concept of a zone belonging to a zonegroup in radosgw-admin). It happens for example if you modify a zone (with zone modify) to make it the master zone (of a zonegroup in fact) and you don't specify the zonegroup the zone belongs to/is associated with: it will in fact update the default zonegroup that can be in another realm or at least not be the zonegroup the zone is associated with. A less impactful but nevertheless unexpected result is if you do something like: 'zonegroup list --rgw-realm my_realm'. You would expect to see only the zonegroups associated with the realm you specified where you'll get the list of all the zonegroups defined. I can imagine that chaning this is difficult without breaking backward compatibility. IMO, an improvement would be the ability not to have a default realm, zonegroup, zone. At least you would have an error if you don't specify explicitely the object to modify. Am I the only one struggling with radosgw-admin? Is there any plan to improve this? Best regards, Michel

3 3

[ceph-user] ceph-ansible OS Compatibility Matrix.
by Danish Khan 28 May '25

28 May '25

Dear Team, We have few ceph-ansible clusters which are still running on Ubuntu 18 and 20. We want to upgrade OS to ubuntu 22. But my ceph clusters are still running on octopus version. Is there any document which can suggest if I can upgrade OS first or do I need to upgrade ceph major version first. Regards, Danish Khan

2 1

Not using image 'xxx' as it's not in list of non-dangling images with ceph=True label
by Patrick Begou 27 May '25

27 May '25

Hi Ceph community, I'm deploying a fresh Ceph cluster using Almalinux9 and Squid version. Deployment with: "dnf -y install https://download.ceph.com/rpm-19.2.2/el9/noarch/cephadm-19.2.2-0.el9.noarch…" Then bootstrap (all runs fine). But launching cephadm shell I get this message that I cannot fully understand from podman: "Not using image 'xxx' as it's not in list of non-dangling images with ceph=True label" I saw that I can clean old images with "podman system prune -a" but this is a fresh install. Can I continue safely to deploy or is this a problem ? Thanks Patrick

2 1

CEPH Reef - HDD with WAL and DB on NVME
by Steven Vacaroaia 27 May '25

27 May '25

Hi, I am a bit confused by the documentation as it seems to say that, when using bluestore, there is no need to put the journaling on a faster storage when creating OSD https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/ <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.ceph…> But then I found this : “… BlueStore is deployed across one or two additional devices: - A *write-ahead log (WAL) device* (identified as block.wal in the data directory) can be used to separate out BlueStore’s internal journal or write-ahead log. Using a WAL device is advantageous only if the WAL device is faster than the primary device (for example, if the WAL device is an SSD and the primary device is an HDD). - A *DB device* (identified as block.db in the data directory) can be used to store BlueStore’s internal metadata. BlueStore (or more precisely, the embedded RocksDB) will put as much metadata as it can on the DB device in order to improve performance. If the DB device becomes full, metadata will spill back onto the primary device (where it would have been located in the absence of the DB device). Again, it is advantageous to provision a DB device only if it is faster than the primary device. …” Is it still recommended to put WAL and DB on a faster disk , right ? How can this be done with cephadm .? Is there a documented procedure for fixing this if it was not done from the beginning ? Many thanks Steven

1 0

v17.2.9 Quincy released
by Yuri Weinstein 27 May '25

27 May '25

This is the ninth backport (hotfix) release in the Quincy series. We recommend that all users update to this release. https://ceph.io/en/news/blog/2025/v17-2-9-quincy-released/ Please be aware that this hotfix was finalized after its end-of-life (EOL) declaration and subsequent removal from the build system. As a result, container builds were not created, and the hotfix has only been subject to limited testing. Users should expect to see the el8 rpm subdirectory empty and the "dnf" commands are expected to fail with 17.2.9. They can choose to use 17.2.9 RPM packages for centos 8/el8 provided by CERN as a community member or stay at 17.2.7 following instructions from https://docs.ceph.com/en/latest/install/get-packages/#rhel, the ceph.repo file should point to https://download.ceph.com/rpm-17.2.7/el8 instead of https://download.ceph.com/rpm-quincy/el8 These CERN packages come with no warranty and have not been tested. The software in them has been tested by Ceph according to `platforms <https://docs.ceph.com/en/latest/start/os-recommendations/#platforms>`_ . The repository for el8 builds is hosted by CERN on `Linux@CERN <https://linuxsoft.cern.ch/repos/ceph-ext-quincy8el-stable/>`_ . The public part of the GPG key used to sign the packages is available at `RPM-GPG-KEY-Ceph-Community <https://linuxsoft.cern.ch/repos/RPM-GPG-KEY-Ceph-Community>`_ . Also note that Ceph now builds against OpenSSL 3.5.0, which may affect EL package users who are on distros that still reference an older version. Notable Changes --------------- This release fixes a critical BlueStore regression in versions 17.2.8 (#63122). Users running this release are recommended to upgrade at their earliest convenience. Changelog --------- - [quincy] os/bluestore: fix \_extend\_log seq advance ([pr#63122](https://github.com/ceph/ceph/pull/63122), aclamk) Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.9.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: f817ceb7f187defb1d021d6328fa833eb8e943b3

1 0

Newly added node osds getting full before rebalance finished
by Szabo, Istvan (Agoda) 27 May '25

27 May '25

Hi, For some reason this is the 2nd time I got into the same issue, added a new node at the time when in the cluster the most full osd was 73% and all the newly added osds now started to be between 70-80% full half way on the way to finish the rebalance. I don't understand why, could this be because mgr upmap mode with deviation 1? Another question, I have 7 disks in each servers, however in the newly added one I still have 4 as spare which is not added. Just to get out of this space issue, would it cause any issue if I add the remaining 4 spare disk (which will cause unbalanced server size in the cluster, the weight of each server is 100TB, this would cause only this node to be 160TB). Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 11

Multisite bucket removal stuck
by Matthew Vernon 27 May '25

27 May '25

Hi, I'm running reef / cephadm. A user wanted two buckets removed (one of which was empty, and one of which had a large number (~half a million) of small objects in). In the primary zone I ran: radosgw-admin bucket rm --bucket=name --bypass-gc --purge-objects [which I've previously used for removing buckets with very-many-objects in single-site just fine] That returned OK for both buckets (quickly for the empty one, after about 20m for the many-objects one). 24 hours later, the empty bucket is gone in both zones, but the many-object one is still present in the secondary zone: source bucket :gitlab-artifacts[64f0dd71-48bf-45aa-9741-69a51c083556.75705.1]) incremental sync on 11 shards bucket is behind on 11 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10] ...which I think is showing that no shards have caught up, even a day later? Similar, sync status shows being behind on 1 metadata shard, and 11 data shards. There are no reported errors in sync. Is there a way to see if this is actually making headway? And/or to get rid of the bucket in the secondary zone? For future reference, would I have been better off trying to stop sync of the bucket and then run a bucket rm in both zones separately? FWIW, this is a small (3-4 OSD hosts per site) and slow (hdds, with nvme for block.db) cluster. Thanks, Matthew

1 2

Re: About pg incomplete
by Phong Tran Thanh 27 May '25

27 May '25

Yes i was remove osd down and purge it now pg can't get active state Vào Th 2, 26 thg 5, 2025 vào lúc 21:31 Anthony D'Atri < anthony.datri(a)gmail.com> đã viết: > Did an OSD go down that you can bring back? > > > On May 26, 2025, at 5:23 AM, Phong Tran Thanh <tranphong079(a)gmail.com> > wrote: > > > > Hi community > > > > How can I force create a new pg when pg status is incomplete. I accept > data > > lost for creating a new pg. > > > > health: HEALTH_WARN > > Reduced data availability: 1 pg inactive, 1 pg incomplete > > 1 pgs not deep-scrubbed in time > > 1 pgs not scrubbed in time > > 154 slow ops, oldest one blocked for 1355676 sec, osd.101 has > > slow ops > > > > pg 8.87b is incomplete, acting [101,70,4,36,16,18] (reducing pool > > fss.motion.data min_size from 4 may help; search ceph.com/docs for > > 'incomplete') > > > > My pool ec4+2 and i was set min_size of pool 4 and did not resolve the > > problem. > > > > PG incomplete effect to cluster slow ops. > > > > > ---------------------------------------------------------------------------- > > *Tran Thanh Phong* > > > > Email: tranphong079(a)gmail.com > > Skype: tranphong079 > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

1 0

ceph reef on AlmaLinux9 (and the dependency problem on openssl-libs)
by Massimo Sgaravatto 26 May '25

26 May '25

Dear all Some weeks ago we updated our ceph cluster (installed on AlmaLinux9 servers) to Ceph reef v. 18.2.6. The day after the email talking about the serious regression bug in that release was sent to this mailing list Frankly it is not fully clear to me if/how that bug is relevant for our deployment: I can say that the OSD start without problems and I can't see obvious problems 18.2.7 can't be installed because it needed openssl-libs v. 3.4 (as discussed in this mailing list) which is in CentOS9 stream but not yet in AlmaLinux9 I have no idea if/when that openssl-libs 3.4 stuff will be released in AlmaLinux9 (AlmaLinux 9.6, released a couple of days ago, doesn't provide an update for openssl-libs). I have now to install some new servers and I have no clear ideas how to proceed Should I keep version 18.2.6 on the existing servers and install 18.2.5 on the new ones ? Should I try installing version 18.2.6 also on the new servers ? Any suggestions? Thanks, Massimo

6 6

CSC Meeting Notes - 05/19
by Venky Shankar 26 May '25

26 May '25

(sending this rather late) * Are we ever going to make a statement about Reef and later shirts, or do we just quietly hope nobody remembers that they used to be a thing we promised? * RH was paying that and costed 80$ * we are 3 release behind and we need to take a call * [Patrick] open source it and allow folks to print it * [Greg] ask our new community manager * [Anthony Middleton] Board has bought this topics up - need to evaluate and process * [Matan] Undefined cmake build type https://tracker.ceph.com/issues/71360 * RelWithDebInfo resulting in a regression * Need to take a decision for BUILD_TYPE - [Sam] set it to DEBUG by default (for shaman builds) - For crimson: s/asert/ceph_assert and stick with RelWithDebInfo and measure performance - [Matan] summary: fine with falling back to RelWithDebInfo and giving up on builtin asserts (crimson specific things can be dealt separately) - (option #1) [Patrick] We might need a build type that has the asserts but has all the optimization benefits - (option #2) Change RelWithDebInfo to include builtin asserts - Not merge the PR and discuss in the tracker (for Crimson, continue offline) * [Mike] Next release: Umbrella * https://ceph-storage.slack.com/archives/C05R975114L/p1747358988415529 * https://en.m.wikipedia.org/wiki/Umbrella_octopus * [Yuri] 17.2.9 final Q release * currently stuck - get quincy (temp) back into build process * [David] Did the needful to get build done * [Yuri] centos8 build by CERN * Yuri and David to catch up to discuss centos8 build -- Cheers, Venky

1 0

18.2.7: modification of osd_deep_scrub_interval in config ignored by mon
by Michel Jouvin 26 May '25

26 May '25

Hi, Last week we increased osd_deep_scrub_interval from 10 days to 14 days as we tended to have permanently 1 PG with a late deep scrub (the PG changing all the time). We did it with `ceph config set ...`. From what we have seen, the deep scrubs are now spread over 14 days (the oldest are 14 days) meaning that OSDs took this change into account (without being restarted). But the number of late deep scrubs reported by `ceph -s) is ~700 which is unexpected. Does it mean that the mon (who is in charge of the report if I am right) have not seen the changes (they have not been restarted)? Cheers, Michel

4 7

About pg incomplete
by Phong Tran Thanh 26 May '25

26 May '25

Hi community How can I force create a new pg when pg status is incomplete. I accept data lost for creating a new pg. health: HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg incomplete 1 pgs not deep-scrubbed in time 1 pgs not scrubbed in time 154 slow ops, oldest one blocked for 1355676 sec, osd.101 has slow ops pg 8.87b is incomplete, acting [101,70,4,36,16,18] (reducing pool fss.motion.data min_size from 4 may help; search ceph.com/docs for 'incomplete') My pool ec4+2 and i was set min_size of pool 4 and did not resolve the problem. PG incomplete effect to cluster slow ops. ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

1 1

Unexpected data sync on deleted bucket instances with sync disabled
by Ramin Najjarbashi 25 May '25

25 May '25

Hi I am running a multi-region Ceph cluster with version 18.2.6. Our system has been operating reliably for a long time and currently holds over 100,000 buckets and approximately 1.5 billion objects. We recently attempted to add a new zone to the primary region. To control the synchronization scope, we globally disabled bucket sync using sync disable on all buckets, intending to enable sync selectively only for specific buckets in a bidirectional manner. A Lua script is in place and functioning correctly, issuing a sync disable request via POST whenever a new bucket is created. Despite this setup, we’ve observed that once the new zone was added, data synchronization began across all buckets, including those where sync was explicitly disabled. Upon investigation, we found that a large number of deleted buckets still have existing bucket instances on the master zone with sync flag set to 0. For example, while we have ~100k active buckets, we have over 150k bucket instances. To mitigate this, we manually updated all stale bucket instances using radosgw-admin metadata put to set the sync flag to 8. The change is reflected correctly on the secondary zone, and the metadata synchronization is working as expected. However, the data is still being fully synced for those deleted buckets. An example scenario: • A bucket x was previously created with bucket ID 123456 and later deleted. • A new bucket x is created by a user with a different ID 654321. • Both bucket instances exist: x:123456 and x:654321. • During data sync, it appears the system tries to replicate bucket x regardless of which bucket ID is associated with it. We also attempted to clean up logs by deleting all relevant data sync and full sync logs from the logging pool, and performed datalog trim, but the issue persists. I have two main questions: 1. Why is full data sync still occurring for bucket instances with sync flag set to 8? 2. What is the recommended and safe approach to remove old/stale bucket instances without risking data loss for users? Currently, the sync configuration is set to replicate data between all zones (sync from all), with "log_meta": false and "log_data": true. Looking forward to your insights. Please let me know if further details or logs are needed. Best regards, Ramin

1 0

SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
by Dmitrijs Demidovs 23 May '25

23 May '25

Hi List! We are in the middle of migration from Ceph v16 to v18. As a first step of migration we decided to change OS version on all OSD hosts from Rocky Linux 8.10 to Rocky Linux 9.5 (by using clean reinstall). We also upgraded OSD Block disks from 4TB to 10TB. Now we have 6 OSD hosts running Rocky 9 with new 10TB disks (8 OSD Block disks per Host). OSD hosts have 64GB of RAM. At this moment Ceph cluster is still on version 16.2.15 "Pacific". Migration to Ceph 18.2.x will be performed later after upgrading to Rocky 9 all remaining parts (MGR+MON, MDS, RGW). Problem: After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%. Docker containers running OSD instances are restarted from time to time (and there is a lot of ugly "Call Trace" messages in dmesg). I was wondering if anyone else has similar issues? Will it be gone after migration to Ceph v18? P.S. Configuration Details: ===== [ceph: root@ceph-mon12 /]# ceph versions { "mon": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "osd": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 48 }, "mds": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "rgw": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "overall": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 58 } } [ceph: root@ceph-mon12 /]# === Old Rocky 8 OSD Host: [root@ceph-osd11 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1a4c959eb813 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-37 e37a366347db quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-1 0fdfb95fd631 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-14 276ad8a11148 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-31 9dbd30d11124 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-19 0ad59f583e6a quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-7 e551a652c3ae quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-42 07ff632ae3e1 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-25 605b7a1c00f7 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd11 fe2a6b8a35e8 quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd11 [root@ceph-osd11 ~]# uptime 10:52:59 up 216 days, 20:12, 1 user, load average: 0.20, 0.41, 0.53 [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi [root@ceph-osd11 ~]# rpm -qa | grep docker docker-buildx-plugin-0.14.0-1.el8.x86_64 docker-ce-cli-26.1.3-1.el8.x86_64 docker-ce-26.1.3-1.el8.x86_64 docker-scan-plugin-0.23.0-3.el8.x86_64 docker-ce-rootless-extras-26.1.3-1.el8.x86_64 docker-compose-plugin-2.27.0-1.el8.x86_64 [root@ceph-osd11 ~]# cat /etc/rocky-release Rocky Linux release 8.10 (Green Obsidian) [root@ceph-osd11 ~]# uname -a Linux ceph-osd11 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 8 17:47:08 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux === New Rocky 9 OSD Host: [root@ceph-osd17 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5cc58e4a77b2 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 days ago Up 4 days ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 3a60fecf648d quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 9c20407e79eb quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 9deadafef9dd quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 fcfe62a25fd9 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 38d2d96cc491 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 e29c6bbc1ae7 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 40346a7a45ea quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 43c3e3a65531 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 d9e436f9788c quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 [root@ceph-osd17 ~]# uptime 10:25:10 up 122 days, 22:37, 1 user, load average: 0.37, 0.26, 0.27 [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi [root@ceph-osd17 ~]# rpm -qa | grep docker docker-compose-plugin-2.31.0-1.el9.x86_64 docker-buildx-plugin-0.19.2-1.el9.x86_64 docker-ce-cli-27.4.0-1.el9.x86_64 docker-ce-rootless-extras-27.4.0-1.el9.x86_64 docker-ce-27.4.0-1.el9.x86_64 [root@ceph-osd17 ~]# cat /etc/rocky-release Rocky Linux release 9.5 (Blue Onyx) [root@ceph-osd17 ~]# uname -a Linux ceph-osd17 5.14.0-503.15.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 26 17:24:29 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux =====

6 6

rgw crash in RGWDeleteMultiObj::handle_individual_object for bucket notification reserve
by xu chenhui 23 May '25

23 May '25

Hi, RGW enable bucket notification for http push_endpoint, it will crash when exec delete multi objects command. ceph version: v16.2.13 Thread 483 "radosgw" received signal SIGABRT, Aborted. [Switching to Thread 0x7f94c0ac1700 (LWP 500)] 0x00007f95b86f2acf in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f95b86f2acf in raise () from /lib64/libc.so.6 #1 0x00007f95b86c5ea5 in abort () from /lib64/libc.so.6 #2 0x00007f95b86c5d79 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x00007f95b86eb426 in __assert_fail () from /lib64/libc.so.6 #4 0x00007f95c7480332 in boost::context::continuation::resume() && (this=0x55a1b3705158) at /opt/ceph-16.2.13/build/boost/include/boost/context/continuation_fcontext.hpp:263 #5 0x00007f95c74802cd in boost::context::continuation::resume() & (this=0x55a1b3705158) at /opt/ceph-16.2.13/build/boost/include/boost/context/continuation_fcontext.hpp:259 #6 0x00007f95c74803db in spawn::detail::continuation_context::resume (this=0x55a1b3705158) at /opt/ceph-16.2.13/src/spawn/include/spawn/impl/spawn.hpp:37 #7 0x00007f95c7499de0 in spawn::detail::coro_async_result<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, void>::get (this=0x55a1b4b0b248) at /opt/ceph-16.2.13/src/spawn/include/spawn/impl/spawn.hpp:256 #8 0x00007f95c7d28659 in librados::async_operate<boost::asio::io_context, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > > > (ctx=..., io=..., oid="ifind-project-other-tv", write_op=0x55a1b4b0b460, flags=1024, token=...) at /opt/ceph-16.2.13/src/librados/librados_asio.h:185 #9 0x00007f95c7d24d9d in rgw_rados_operate (dpp=0x55a1b463c000, ioctx=..., oid="ifind-project-other-tv", op=0x55a1b4b0b460, y=..., flags=1024) at /opt/ceph-16.2.13/src/rgw/rgw_tools.cc:269 #10 0x00007f95c7f4caff in rgw::notify::publish_reserve (dpp=0x55a1b463c000, event_type=rgw::notify::ObjectRemovedDelete, res=..., req_tags=0x0) at /opt/ceph-16.2.13/src/rgw/rgw_notify.cc:796 #11 0x00007f95c7b182e0 in RGWDeleteMultiObj::handle_individual_object (this=0x55a1b463c000, o=..., y=..., formatter_flush_cond=0x7f9485248130) at /opt/ceph-16.2.13/src/rgw/rgw_op.cc:7164 #12 0x00007f95c7b18c45 in RGWDeleteMultiObj::<lambda(yield_context)>::operator()(yield_context) const (__closure=0x55a1b0160fa8, yield=...) at /opt/ceph-16.2.13/src/rgw/rgw_op.cc:7267 #13 0x00007f95c7b2f43f in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>, boost::context::basic_fixedsize_stack<boost::context::stack_traits> >::<lambda(boost::context::continuation&&)>::operator()(boost::context::continuation &&) const ( this=0x7f9485247f60, c=...) at /opt/ceph-16.2.13/src/spawn/include/spawn/impl/spawn.hpp:390 #14 0x00007f95c7b30610 in std::__invoke_impl<boost::context::continuation, spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(std::__invoke_other, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>, boost::context::basic_fixedsize_stack<boost::context::stack_traits> >::<lambda(boost::context::continuation&&)> &, boost::context::continuation &&) (__f=..., __args#0=...) at /usr/include/c++/8/bits/invoke.h:60 #15 0x00007f95c7b30531 in std::__invoke<spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>, boost::context::basic_fixedsize_stack<boost::context::stack_traits> >::<lambda(boost::context::continuation&&)> &, boost::context::continuation &&) (__fn=..., __args#0=...) at /usr/include/c++/8/bits/invoke.h:96 #16 0x00007f95c7b304a2 in std::invoke<spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>, boost::context::basic_fixedsize_stack<boost::context::stack_traits> >::<lambda(boost::context::continuation&&)> &, boost::context::continuation &&) (__fn=..., __args#0=...) at /usr/include/c++/8/functional:82 #17 0x00007f95c7b3036f in boost::context::detail::record<boost::context::continuation, boost::context::basic_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)> >::run(boost::context::detail::fcontext_t) ( this=0x55a1b4b0bf00, fctx=0x7f9485247d10) at /opt/ceph-16.2.13/build/boost/include/boost/context/continuation_fcontext.hpp:146 #18 0x00007f95c7b3015a in boost::context::detail::context_entry<boost::context::detail::record<boost::context::continuation, boost::context::basic_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)> > >(boost::context::detail::transfer_t) (t=...) at /opt/ceph-16.2.13/build/boost/include/boost/context/continuation_fcontext.hpp:80 #19 0x00007f95c809175f in make_fcontext () from /lib64/libradosgw.so.2 #20 0x9435c9c14cf40130 in ?? () #21 0xacf2d40e6f64164e in ?? () #22 0x00000000d9ff03a8 in ?? () #23 0x0000000000000025 in ?? () #24 0x5f7265767265535f in ?? () #25 0x000055a1b4b0bef8 in ?? () #26 0x0000000000000007 in ?? () #27 0x00322d7466697773 in ?? () #28 0x0000000000020000 in ?? () #29 0x000055a1b4b0c000 in ?? () #30 0x0000000000020000 in ?? () #31 0x00007f9485247f60 in ?? () #32 0x3234346330623365 in ?? () Reproduce： 1. create topic rgw_topic='{"push-endpoint": "'$httpserver'","verify-ssl": "false","persistent":"true"}' aws --region=default --endpoint-url $srcEndpoint sns create-topic --name=$userId --attributes="$rgw_topic" 2. create bucket notification bucket_notify='{"TopicConfigurations": [{"Id": "'$bk'", "TopicArn": "'$topic'", "Events": ["s3:ObjectCreated:*", "s3:ObjectRemoved:*"]}]}' aws --region=default --endpoint-url $srcEndpoint s3api put-bucket-notification-configuration --bucket=$bucketName --notification-configuration="$bucket_notify" 3. delete multi objects aws s3api --endpoint-url $srcEndpoint delete-objects --bucket $bucketName --delete '{"Objects":[{"Key":"obj1"}]}' I had report this problem in https://tracker.ceph.com/issues/71427 rgw_multi_object_del_aio set to 1 didn't help, anyone can have a workaround to resolve this problem ? thanks.

1 0

CephFS MDS - Crash restart loop
by Michael Götting 23 May '25

23 May '25

Hi all, we have the following problem with our CephFS Setup (Ceph version 19.2.2). Today our two active MDS nodes failed and then the nodes that were in „stand-by replay“ took over and failed as well. The CephFS system is equipped as follows: - 3 monitor nodes - 4 MDS nodes (2 active/ 2 stand-by) - 2 active - 2 stand-by replay - CephFS Pool - max_mds = 2 - 1x Meta data pool - 2x data pools (hdd_pool, ssd_pool) << ----------------- Ceph fs status output START----------------- >> cephfs - 0 clients RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 failed << ----------------- Ceph fs status output END----------------- >> To restore the service, we used the documentation https://docs.ceph.com/en/quincy/cephfs/disaster-recovery-experts/?highlight… we carried out the steps up to and including "MDS table wipes". We did not carry out the MDS MAP RESET step as we were not sure that we would then lose all the data from RANK 1. We also carried out the steps under "Avoiding recovery roadblocks" https://docs.ceph.com/en/quincy/cephfs/troubleshooting/#avoiding-recovery-r…. Parameters set of the MDS nodes: mds advanced mds_abort_on_newly_corrupt_dentry false mds advanced mds_bal_interval 0 mds basic mds_cache_memory_limit 274877906944 mds advanced mds_cache_trim_threshold 524288 mds advanced mds_go_bad_corrupt_dentry false mds advanced mds_heartbeat_grace 3600. 000000 mds advanced mds_min_caps_working_set 60000 mds advanced mds_oft_prefetch_dirfrags false * After trying the recovery steps (truncating the journal) the MDS daemons are in a crash -> restart loop behavior. << ----------------- Example log file mds-1: START ----------------- >> -14> 2025-05-21T17:49:27.003+0200 7f36cfc7e640 1 mds.0.42052 active_start -13> 2025-05-21T17:49:27.003+0200 7f36d2c84640 10 monclient: get_auth_request con 0x559a76e8a400 auth_method 0 -12> 2025-05-21T17:49:27.003+0200 7f36d2483640 10 monclient: get_auth_request con 0x559a7e041400 auth_method 0 -11> 2025-05-21T17:49:27.003+0200 7f36d3485640 10 monclient: get_auth_request con 0x559a76e8b000 auth_method 0 -10> 2025-05-21T17:49:27.003+0200 7f36d3485640 10 monclient: get_auth_request con 0x559ab89cb800 auth_method 0 -9> 2025-05-21T17:49:27.003+0200 7f36d2c84640 10 monclient: get_auth_request con 0x559a7bcc6400 auth_method 0 -8> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 1 mds.0.42052 cluster recovered. -7> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 4 mds.0.42052 set_osd_epoch_barrier: epoch=492573 -6> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 5 quiesce.mds.0 <quiesce_cluster_update> epoch:42055 me:7764062 leader:7764062 members:7764062 -5> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 5 quiesce.mgr.0 <update_membership> starting the db mgr thread at epoch: 42055 -4> 2025-05-21T17:49:27.015+0200 7f36c5c6a640 5 quiesce.mgr.0 <quiesce_db_thread_main> Entering the main thread -3> 2025-05-21T17:49:27.015+0200 7f36c5c6a640 5 quiesce.mgr.0 <membership_upkeep> a reset of the db has been requested -2> 2025-05-21T17:49:27.015+0200 7f36c9471640 -1 mds.0.cache.den(0x1 techfak) newly corrupt dentry to be committed: [dentry #0x1/techfak [c,head] auth (dversion lock) pv=0 v=52947746 ino=0x1000a58d072 state=1073741824 | inodepin=1 0x559a755b2c80] -1> 2025-05-21T17:49:27.015+0200 7f36c9471640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/mds/MDCache.cc: In function 'void MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)' thread 7f36c9471640 time 2025-05-21T17:49:27.020101+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/mds/MDCache.cc: 1687: FAILED ceph_assert(follows >= realm->get_newest_seq()) ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f36d5709cf9] 2: /usr/lib64/ceph/libceph-common.so.2(+0x182eb8) [0x7f36d5709eb8] 3: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xac3) [0x559a51ca4583] 4: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xbd) [0x559a51ca50cd] 5: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0xe71) [0x559a51cab8f1] 6: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x473) [0x559a51d55a33] 7: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x390) [0x559a51d2e750] 8: (MDSContext::complete(int)+0x5c) [0x559a51e4617c] 9: (MDSIOContextBase::complete(int)+0x34c) [0x559a51e4884c] 10: /usr/bin/ceph-mds(+0x4f5970) [0x559a51eed970] 11: /usr/bin/ceph-mds(+0x160f0d) [0x559a51b58f0d] 12: (Finisher::finisher_thread_entry()+0x17d) [0x7f36d57c885d] 13: /lib64/libc.so.6(+0x8a0ca) [0x7f36d50a30ca] 14: /lib64/libc.so.6(+0x10f150) [0x7f36d5128150] 0> 2025-05-21T17:49:27.019+0200 7f36c9471640 -1 *** Caught signal (Aborted) ** in thread 7f36c9471640 thread_name: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f36d5057bf0] 2: /lib64/libc.so.6(+0x8be0c) [0x7f36d50a4e0c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f36d5709d53] 6: /usr/lib64/ceph/libceph-common.so.2(+0x182eb8) [0x7f36d5709eb8] 7: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xac3) [0x559a51ca4583] 8: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xbd) [0x559a51ca50cd] 9: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0xe71) [0x559a51cab8f1] 10: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x473) [0x559a51d55a33] 11: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x390) [0x559a51d2e750] 12: (MDSContext::complete(int)+0x5c) [0x559a51e4617c] 13: (MDSIOContextBase::complete(int)+0x34c) [0x559a51e4884c] 14: /usr/bin/ceph-mds(+0x4f5970) [0x559a51eed970] 15: /usr/bin/ceph-mds(+0x160f0d) [0x559a51b58f0d] 16: (Finisher::finisher_thread_entry()+0x17d) [0x7f36d57c885d] 17: /lib64/libc.so.6(+0x8a0ca) [0x7f36d50a30ca] 18: /lib64/libc.so.6(+0x10f150) [0x7f36d5128150] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. << ----------------- Example log file mds-1: END ----------------- >> << ----------------- Ceph fs fs dump output START----------------- >> e41371 btime 2025-05-21T16:19:27:085643+0200 enable_multiple, ever_enabled_multiple: 1,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 3 Filesystem 'cephfs' (3) fs_name cephfs epoch 41370 flags 73 allow_snaps allow_multimds_snaps allow_standby_replay refuse_client_session created 2024-03-31T23:36:25.302389+0200 modified 2025-05-21T16:19:09.237977+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 max_xattr_size 65536 required_client_features {} last_failure 0 last_failure_osd_epoch 492429 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} max_mds 2 in 0,1 up {1=7761302} failed 0 damaged stopped data_pools [5,3] metadata_pool 4 inline_data disabled balancer bal_rank_mask -1 standby_count_wanted 2 qdb_cluster leader: 7761302 members: 7761302 [mds.mds-1{1:7761302} state up:active seq 5 addr [v2:[2001:638:504:2011:9:3:1:1]:6800/2463826788,v1:[2001:638:504:2011:9:3:1:1]:6801/2463826788] compat {c=[1],r=[1],i=[1fff]}] Standby daemons: [mds.mds-2{-1:7771506} state up:standby seq 1 addr [v2:[2001:638:504:2011:6:3:2:2]:6800/2809657192,v1:[2001:638:504:2011:6:3:2:2]:6801/2809657192] compat {c=[1],r=[1],i=[1fff]}] dumped fsmap epoch 41371 << ----------------- Ceph fs fs dump output END———————— >> But to be honest, out of all those things we tried, I don't know what to provide exactly. We can provide much more but ... We really need the service back online, so help will be very much appreciated. Regards, Michael

2 2

ceph node upgrades
by Bandelow, Gunnar 23 May '25

23 May '25

Hi everyone out there, i have a hopefully simple to answer question: For example a reef cluster with cephadm and all containers and packages on version 18.2.4 on debian12 nodes. The nodes itself have now ceph debian package upgrades available for the ceph packages with version 18.2.7. Do I have to maintain the same version between the version within the containers and the OS? Can I just upgrade the nodes with a newer version of packages and keep the containers at 18.2.4? I marked ceph-common and cephadm as hold, because I wasn't sure, but now I want to know. Any input is appreciated. Best regards, Gunnar ======================================================= Gunnar Bandelow (dipl. phys.) Universitätsrechenzentrum (URZ) Universität Greifswald Felix-Hausdorff-Straße 18 17489 GreifswaldGermany

3 2

Re: Getting alarm emails every 600s after Ceph Pacific install
by Malte Stroem 22 May '25

22 May '25

Hi Stefan, why did you install Ceph with the almost obsolete ceph-ansible and why did you install the very old Pacific? Since this is a new cluster try the easy way and install Ceph Squid with cephadm. https://docs.ceph.com/en/latest/cephadm/install/ Best, Malte On 11.08.21 21:21, Stefan Schneebeli wrote: > Hi all, > > I just installed a new Ceph Pacific Cluster with ceph-ansible which > worked very well. > > But now one of the mon servers sends every 600s an email with this subject: > > ALERT localhost/trap: trap timeout > > and this body: > > Summary output : trap timeout > > Group : localhost > Service : trap > Time noticed : Wed Aug 11 21:06:16 2021 > Secs until next alert : > Members : localhost > > Detailed text (if any) follows: > ------------------------------- > trap timeout after 600s at Wed Aug 11 21:06:16 2021 > > > I have never seen this before. I also don't see any messages like this > in the mon logfiles. > The two other mon servers don't sent such emails. > > Does anybody have an idea what the problem could be and how to solve it? > > Thank you! > > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
by Dmitrijs Demidovs 22 May '25

22 May '25

Hi List! We are in the middle of migration from Ceph v16 to v18. As a first step of migration we decided to change OS version on all OSD hosts from Rocky Linux 8.10 to Rocky Linux 9.5 (by using clean reinstall). We also upgraded OSD Block disks from 4TB to 10TB. Now we have 6 OSD hosts running Rocky 9 with new 10TB disks (8 OSD Block disks per Host). OSD hosts have 64GB of RAM. At this moment Ceph cluster is still on version 16.2.15 "Pacific". Migration to Ceph 18.2.x will be performed later after upgrading to Rocky 9 all remaining parts (MGR+MON, MDS, RGW). Problem: After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%. Docker containers running OSD instances are restarted from time to time (and there is a lot of ugly "Call Trace" messages in dmesg). I was wondering if anyone else has similar issues? Will it be gone after migration to Ceph v18? P.S. Configuration Details: ===== [ceph: root@ceph-mon12 /]# ceph versions { "mon": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "osd": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 48 }, "mds": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "rgw": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "overall": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 58 } } [ceph: root@ceph-mon12 /]# === Old Rocky 8 OSD Host: [root@ceph-osd11 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1a4c959eb813 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-37 e37a366347db quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-1 0fdfb95fd631 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-14 276ad8a11148 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-31 9dbd30d11124 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-19 0ad59f583e6a quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-7 e551a652c3ae quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-42 07ff632ae3e1 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-25 605b7a1c00f7 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd11 fe2a6b8a35e8 quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd11 [root@ceph-osd11 ~]# uptime 10:52:59 up 216 days, 20:12, 1 user, load average: 0.20, 0.41, 0.53 [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi [root@ceph-osd11 ~]# rpm -qa | grep docker docker-buildx-plugin-0.14.0-1.el8.x86_64 docker-ce-cli-26.1.3-1.el8.x86_64 docker-ce-26.1.3-1.el8.x86_64 docker-scan-plugin-0.23.0-3.el8.x86_64 docker-ce-rootless-extras-26.1.3-1.el8.x86_64 docker-compose-plugin-2.27.0-1.el8.x86_64 [root@ceph-osd11 ~]# cat /etc/rocky-release Rocky Linux release 8.10 (Green Obsidian) [root@ceph-osd11 ~]# uname -a Linux ceph-osd11 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 8 17:47:08 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux === New Rocky 9 OSD Host: [root@ceph-osd17 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5cc58e4a77b2 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 days ago Up 4 days ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 3a60fecf648d quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 9c20407e79eb quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 9deadafef9dd quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 fcfe62a25fd9 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 38d2d96cc491 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 e29c6bbc1ae7 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 40346a7a45ea quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 43c3e3a65531 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 d9e436f9788c quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 [root@ceph-osd17 ~]# uptime 10:25:10 up 122 days, 22:37, 1 user, load average: 0.37, 0.26, 0.27 [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi [root@ceph-osd17 ~]# rpm -qa | grep docker docker-compose-plugin-2.31.0-1.el9.x86_64 docker-buildx-plugin-0.19.2-1.el9.x86_64 docker-ce-cli-27.4.0-1.el9.x86_64 docker-ce-rootless-extras-27.4.0-1.el9.x86_64 docker-ce-27.4.0-1.el9.x86_64 [root@ceph-osd17 ~]# cat /etc/rocky-release Rocky Linux release 9.5 (Blue Onyx) [root@ceph-osd17 ~]# uname -a Linux ceph-osd17 5.14.0-503.15.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 26 17:24:29 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux =====

1 0

HBA Card vs. HBA Mode (on a RAID card)
by Anthony Fecarotta 21 May '25

21 May '25

Does running a RAID controller in HBA mode (not to be confused with IT mode) impact Ceph performance compared to using a dedicated HBA card? Is there any documentation or benchmarking data showing improved performance with true HBA hardware? For what it's worth my cluster is on Dell PowerEdge R740 machines. Thank you for your insights. Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

3 3

radosgw-admin: period update error after deleting and recreating zones
by Michel Jouvin 21 May '25

21 May '25

Hi, It is not the first time that after doing configuration changes in RADOS for a realm/zonegroup/zone with radosgw-admin, we get errors when trying to do a "period update --commit". We never found a good documentation on how to fix these problems, up to now we always managed at some point to restore a good configuration that can be commited but it is probably time for us to have a more informed approach! Last occurence of the problem happened today with a realm/zonegroup/zone created recently. Trying to fix a problem with the non working haproxy associated with it, one of my colleagues decided to delete and recreate the zone and zonegroup (with the same names). The related commands worked but since it has been done any attempt to do "period update --commit" results in the following error: ------- 2025-05-06T11:56:20.939+0200 7fdc7d41da80 0 failed reading obj info from .rgw.root:zone_info.93af6e0c-4552-4c2e-b167-36114a5a81e4: (2) No such file or directory 2025-05-06T11:56:20.945+0200 7fdc7d41da80 0 failed reading obj info from .rgw.root:zonegroup_info.d7221099-4e7d-43cb-a1e8-28a750de1cd5: (2) No such file or directory 2025-05-06T11:56:21.160+0200 7fdc7d41da80 0 failed reading obj info from .rgw.root:zone_info.93af6e0c-4552-4c2e-b167-36114a5a81e4: (2) No such file or directory 2025-05-06T11:56:21.160+0200 7fdc7d41da80 -1 Cannot find zone id=93af6e0c-4552-4c2e-b167-36114a5a81e4 (name=default) 2025-05-06T11:56:21.160+0200 7fdc7d41da80 0 ERROR: failed to start notify service ((22) Invalid argument 2025-05-06T11:56:21.160+0200 7fdc7d41da80 0 ERROR: failed to init services (ret=(22) Invalid argument) couldn't init storage provider ------- I have the feeling that it is related to the delete objects that are no longer found but it is not completely clear what is the way out of it? Is the problem related to recreating the zone/zonegroup with the same names? There are several realms already in production so we cannot do a .rgw.root reset but this particular realm has never been put in production so we can delete everything related to it. Thanks in advance for any hint or pointer. Best regards, Michel

3 5

rgw crash on thread notif-worker0
by xu chenhui 21 May '25

21 May '25

Hi, all rgw crash on thread notif-worker0 when using rgw persistent notification function in ceph version: v16.2.13. I haven't find root cause. Backtrace: 0> 2025-05-13T13:36:45.435+0000 7f0813c64700 -1 *** Caught signal (Aborted) ** in thread 7f0813c64700 thread_name:notif-worker0 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f084607acf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f084506809b] 5: /lib64/libstdc++.so.6(+0x9654c) [0x7f084506e54c] 6: /lib64/libstdc++.so.6(+0x965a7) [0x7f084506e5a7] 7: /lib64/libstdc++.so.6(+0x96808) [0x7f084506e808] 8: (rgw::notify::Manager::Manager(ceph::common::CephContext*, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, rgw::sal::RGWRadosStore*)::{lambda()#2}::operator()() const+0x240) [0x7f0852c176c0] 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f084509ab23] 10: /lib64/libpthread.so.0(+0x81ca) [0x7f08460701ca] 11: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. =================== Above coredump info had no symbol info , so I compile same version with debuginfo. However, there are differen coredump info: gdb) bt #0 0x00007fd4edc6dacf in raise () from /lib64/libc.so.6 #1 0x00007fd4edc40ea5 in abort () from /lib64/libc.so.6 #2 0x00007fd4edc40d79 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x00007fd4edc66426 in __assert_fail () from /lib64/libc.so.6 #4 0x00007fd4fb81d7cd in boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>::allocate() () from /lib64/libradosgw.so.2 #5 0x00007fd4fbe8491b in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> > boost::context::callcc<boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >::operator()()::{lambda(boost::context::continuation&&)#1}>(std::allocator_arg_t, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>&&, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >::operator()()::{lambda(boost::context::continuation&&)#1}&&) () from /lib64/libradosgw.so.2 #6 0x00007fd4fbe8554f in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >::operator()() () from /lib64/libradosgw.so.2 #7 0x00007fd4fbe85999 in void boost::asio::detail::strand_executor_service::dispatch<boost::asio::io_context::executor_type const, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >&, std::allocator<void> >(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl> const&, boost::asio::io_context::executor_type const&, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >&, std::allocator<void> const&) () from /lib64/libradosgw.so.2 #8 0x00007fd4fbe85c55 in std::enable_if<spawn::detail::is_stack_allocator<std::decay<boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >::type, void>::value, void>::type spawn::spawn<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}&&, std::decay&&) () from /lib64/libradosgw.so.2 #9 0x00007fd4fbe888fe in rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >) () from /lib64/libradosgw.so.2 #10 0x00007fd4fbe8c711 in rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}::operator()(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >) const () from /lib64/libradosgw.so.2 #11 0x00007fd4fbe8cab4 in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}::stack_traits> >::operator()()::{lambda(rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}::continuation&&)#1}::operator()(rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}::continuation) const () from /lib64/libradosgw.so.2 #12 0x00007fd4fbe8cc90 in boost::context::detail::record<boost::context::continuation, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >::operator()()::{lambda(boost::context::continuation&&)#1}>::run(void*) () from /lib64/libradosgw.so.2 #13 0x00007fd4fbe8cd58 in void boost::context::detail::context_entry<boost::context::detail::record<boost::context::continuation, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_c--Type <RET> for more, q to quit, c to continue without paging-- ontext<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >::operator()()::{lambda(boost::context::continuation&&)#1}> >(boost::context::detail::transfer_t) () from /lib64/libradosgw.so.2 #14 0x00007fd4fbf5e94f in make_fcontext () from /lib64/libradosgw.so.2 #15 0x0000000000000000 in ?? () I have found a similar issue in https://github.com/ceph/ceph/pull/39521, but it haven't reslove it.

2 2

RGW Bucket Replication in Admin Ops API
by Gregory O'Neill 20 May '25

20 May '25

Hey folks, I have three rgw clusters setup with multi-site. replication is bi-direcitonal for all zones between each cluster. They were running Luminous 12.2.12 before I upgraded them to Reef 18.2.2 We have an internal app that manages S3 buckets in ceph for users which uses the admin ops api. There was an option in the api to enable or disable replication but it looks like it was removed in Nautilus (`OPT_BUCKET_SYNC_ENABLE` in src/rgw/rgw_admin.cc in luminous or mimic branches). I know that `radosgw-admin` has a command to toggle bucket sync, but that's not scalable for our user count. What is the intended way to S3 manage bucket replication in Ceph? Thanks, Greg -- Gregory O’Neill

2 1

rpm-quincy
by farhad kh 20 May '25

20 May '25

Hi, I need to install ceph-common from the Quincy repository, but I'm getting this error: --- Ceph x86_64 0.0 B/s | 0 B 00:01 Errors during downloading metadata for repository 'Ceph': - Status code: 404 for http://download.ceph.com/rpm-quincy/el8/x86_64/repodata/repomd.xml (IP: 172.16.110.219) Error: Failed to download metadata for repo 'Ceph': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried --- I think the Ceph Quincy repository might have been removed or is no longer functional. Is that the case? And how can I still use the Ceph Quincy repository for EL8 if possible?

3 2

BLUEFS_SPILLOVER after Reef upgrade
by Kasper Rasmussen 20 May '25

20 May '25

I've just upgraded our ceph cluster from pacific 16.2.15 -> Reef 18.2.7 After that I see the warnings: [WRN] BLUEFS_SPILLOVER: 5 OSD(s) experiencing BlueFS spillover osd.110 spilled over 4.5 GiB metadata from 'db' device (8.0 GiB used of 83 GiB) to slow device osd.455 spilled over 1.1 GiB metadata from 'db' device (11 GiB used of 83 GiB) to slow device osd.533 spilled over 426 MiB metadata from 'db' device (10 GiB used of 83 GiB) to slow device osd.560 spilled over 389 MiB metadata from 'db' device (9.8 GiB used of 83 GiB) to slow device osd.597 spilled over 8.6 GiB metadata from 'db' device (7.7 GiB used of 83 GiB) to slow device [WRN] BLUESTORE_SLOW_OP_ALERT: 4 OSD(s) experiencing slow operations in BlueStore osd.410 observed slow operation indications in BlueStore osd.443 observed slow operation indications in BlueStore osd.508 observed slow operation indications in BlueStore osd.593 observed slow operation indications in BlueStore I've tried to run ceph tell osd.XXX compact with no result. Bluefs stats: ceph tell osd.110 bluefs stats 1 : device size 0x14b33fe000 : using 0x202c00000(8.0 GiB) 2 : device size 0x8e8ffc00000 : using 0x5d31d150000(5.8 TiB) RocksDBBlueFSVolumeSelector >>Settings<< extra=0 B, l0_size=1 GiB, l_base=1 GiB, l_multi=8 B DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 16 MiB 0 B 0 B 0 B 15 MiB 1 WAL 0 B 18 MiB 0 B 0 B 0 B 6.3 MiB 1 DB 0 B 8.0 GiB 0 B 0 B 0 B 8.0 GiB 140 SLOW 0 B 0 B 4.5 GiB 0 B 0 B 4.5 GiB 78 TOTAL 0 B 8.0 GiB 4.5 GiB 0 B 0 B 0 B 220 MAXIMUMS: LOG 0 B 25 MiB 0 B 0 B 0 B 21 MiB WAL 0 B 118 MiB 0 B 0 B 0 B 93 MiB DB 0 B 8.2 GiB 0 B 0 B 0 B 8.2 GiB SLOW 0 B 0 B 14 GiB 0 B 0 B 14 GiB TOTAL 0 B 8.2 GiB 14 GiB 0 B 0 B 0 B >> SIZE << 0 B 79 GiB 8.5 TiB Help with what to do next will, be much appreciated

5 15

[erasure-code] limits on mirroring
by garcetto 20 May '25

20 May '25

good morning, i would like to know if there are some limits/best practices or still not production ready support for ceph mirroring/journailng between two different pools types, say replica --> erasure-code pool or vice-versa. thank you.

1 0

Questions about managing RGW multisite
by Adam Prycki 19 May '25

19 May '25

Hello, I have few question about managing rgw multisite deployments. A) zonegroups and zone endpoints can have multiple URLs/addresses. Is it better to pass just one url to load balancer or configure separate address for each RGW daemon in the zone. B) Rook documentation suggest that having more than 1 RGW running replication is bad for performance. I've also seen some IBM blog posts discussing rgw multisite with graphics showing only single RGW running replication traffic. I don't think there are any warnings about this in official documentation. Is this something you consider when planning your multisite deployments? How many RGW dedicated to replication do you usually deploy per zone? https://www.rook.io/docs/rook/v1.17/Storage-Configuration/Object-Storage-RG… C) Afaik most radosgw-admin command should be executed on ceph cluster with master zonegroup. When I commit changes after these operation period doesn't seem to update on secondary zones. Is this supposed to work like this? D) How are we supposed to keep period on secondary zone in sync? If I need to change secondary zone endpoint I would have always pull the current realm from master zone? 1. pull the latest realm/period from master zone. 2. modifying the secondary zone zone 3. run radosgw-admin period update --commit from secondary zone. E) Do radosgw-admin commands modifying zone on secondary cluster require committing the period afterwards? (for example zone storage class pool change doesn't directly affect zonegroup ) Do they always require rgw restart? Example with enabling compression doesn't commit the period and doesn't mention restarting rgw. https://docs.ceph.com/en/latest/radosgw/compression/ Placement/Storage Classes example mentions that you have to commit period and restart rgw after change. https://docs.ceph.com/en/latest/radosgw/placement/#adding-a-storage-class F) Is it safe to restart RGW daemons during full zone sync? Can it interrupt or negatively affect zone synchronization? Best regards Adam Prycki

1 0

[cephadm] Questions regarding cephadm infra-as-code philosophy, containerization, and trixie availability
by Florent Carli 19 May '25

19 May '25

Hello ceph team, I’ve been working with ceph for some time and am currently evaluating cephadm as a successor to ceph-ansible. During this process, I’ve encountered a few aspects of the cephadm approach that raised some questions, and I’d greatly appreciate your insights to better understand the intended usage and philosophy. 1) Declarative workflows and Infrastructure as Code One of the advantages of ceph-ansible was the ability to define the cluster state declaratively in YAML files, which aligned well with Infrastructure-as-Code principles. With cephadm, the process appears more imperative and CLI-driven, which makes automation and reproducibility harder in comparison. Is there a recommended approach to achieving a fully declarative deployment model with cephadm? Or plans to support this more directly? 2) Containerization vs. local dependencies Cephadm’s move to full containerization makes sense in principle, especially to avoid system-level dependencies. However, in practice, many operations (e.g., using ceph-bluestore-tool, or the python modules for Rados/rbd) still seem to require installing packages on the host. Is this the expected model : containers for core daemons, but local packages for tooling ? It feels somewhat contradictory, and I wonder if there's a clearer pattern or guidance for those cases. 3) Ceph packages for Debian Trixie on download.ceph.com Since I'm using debian, I'm also in the process of anticipating the soon to come Debian 13 version (Trixie). "cephadm install" pulls packages from download.ceph.com, where the latest versions are published. Currently, it seems that there are no ceph packages yet for Trixie on download.ceph.com, which forces me to use the debian hosted packages (they are not even the same ceph version). Is there an ETA for when those packages will be made available? Or maybe there's another url for development packages? Thank you very much for your time and for your work on cephadm. I’m looking to better align our automation practices with the direction Cephadm is taking, and any clarification you can provide would be very helpful. Best regards, Florent.

8 8

Re: 19.2.1: filestore created when bluestore requested
by Konold, Martin 17 May '25

17 May '25

Am 2025-05-16 18:01, schrieb Anthony D'Atri: > I wouldn’t think blkdiscard would necessarily fully clean. I would try > sgdisk —zap-all or Ceph-volume lvm zap I gave this a try in addition to a reboot but no changes. Still not created a bluestore osd as intended. I guess this is the culprit: 2025-05-16T16:28:14.865+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] Maybe someone has a hint? Yours --martin ------------------------------------------------------------------------------ # ceph-volume lvm prepare --bluestore --data vgsdd/sdd1 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a5ae5940-26f2-44c8-bf16-bfdfa6de20ba Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -h ceph:ceph /dev/vgsdd/sdd1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-7 Running command: /usr/bin/ln -s /dev/vgsdd/sdd1 /var/lib/ceph/osd/ceph-1/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap stderr: got monmap epoch 20 --> Creating keyring file for osd.1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid a5ae5940-26f2-44c8-bf16-bfdfa6de20ba --setuser ceph --setgroup ceph stderr: 2025-05-16T16:28:14.865+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T16:28:14.865+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T16:28:14.866+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T16:28:14.867+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T16:28:14.867+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) No valid bdev label found stderr: 2025-05-16T16:28:15.127+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: vgsdd/sdd1 root@pve-03 (pve-03.t3):~# ceph-volume lvm activate --all Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key --> OSD ID 0 FSID a764e57e-ad3b-4823-8849-454f133526f6 process is active. Skipping activation --> Activating OSD ID 1 FSID a5ae5940-26f2-44c8-bf16-bfdfa6de20ba Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vgsdd/sdd1 --path /var/lib/ceph/osd/ceph-1 --no-mon-config Running command: /usr/bin/ln -snf /dev/vgsdd/sdd1 /var/lib/ceph/osd/ceph-1/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-7 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/systemctl enable ceph-volume@lvm-1-a5ae5940-26f2-44c8-bf16-bfdfa6de20ba stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-1-a5ae5940-26f2-44c8-bf16-bfdfa6de20ba.service → /lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@1 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)1.service → /lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@1 --> ceph-volume lvm activate successful for osd ID: 1 ceph-volume-systemd.log has: [2025-05-16 16:23:43,600][ceph_volume.process][INFO ] stderr --> RuntimeError: could not find osd.1 with osd_fsid a2a1cb42-006f-4659-a461-bb0ffbf190ff ps: ceph 11572 0.5 0.0 689688 83000 ? Ssl 16:29 0:02 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph # systemctl status ceph-osd(a)1.service ● ceph-osd(a)1.service - Ceph object storage daemon osd.1 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: enabled) Drop-In: /usr/lib/systemd/system/ceph-osd@.service.d └─ceph-after-pve-cluster.conf Active: active (running) since Fri 2025-05-16 16:29:11 UTC; 7min ago Process: 11567 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 1 (code=exited, status=0/SUCCESS) Main PID: 11572 (ceph-osd) Tasks: 68 Memory: 49.4M CPU: 2.239s CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd(a)1.service └─11572 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph Mai 16 16:29:11 pve-03 systemd[1]: Starting ceph-osd(a)1.service - Ceph object storage daemon osd.1... Mai 16 16:29:11 pve-03 systemd[1]: Started ceph-osd(a)1.service - Ceph object storage daemon osd.1. Mai 16 16:29:15 pve-03 ceph-osd[11572]: 2025-05-16T16:29:15.066+0000 7b3558e37880 -1 osd.1 0 log_to_monitors true Mai 16 16:29:16 pve-03 ceph-osd[11572]: 2025-05-16T16:29:16.351+0000 7b3554dd76c0 -1 osd.1 0 waiting for initial osdmap Mai 16 16:29:16 pve-03 ceph-osd[11572]: 2025-05-16T16:29:16.352+0000 7b354c6166c0 -1 osd.1 0 failed to load OSD map for epoch 32827, got 0 bytes Mai 16 16:29:16 pve-03 ceph-osd[11572]: 2025-05-16T16:29:16.657+0000 7b3548b646c0 -1 osd.1 33967 osdmap NOUP flag is set, waiting for it to clear Mai 16 16:29:17 pve-03 ceph-osd[11572]: 2025-05-16T16:29:17.354+0000 7b3548b646c0 -1 osd.1 33968 osdmap NOUP flag is set, waiting for it to clear Mai 16 16:29:18 pve-03 ceph-osd[11572]: 2025-05-16T16:29:18.362+0000 7b3548b646c0 -1 osd.1 33969 osdmap NOUP flag is set, waiting for it to clear -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

3 2

19.2.1: filestore created when bluestore requested
by Konold, Martin 17 May '25

17 May '25

Hi, I am working on a small 3 node ceph cluster which used to work as expected. When creating a new ceph osd the ceph-volume command throws some errors and filestore instead of bluestore is created. (The drive was clean using blkdiscard before and no traces left in /etc from previous attempts. Any hints? Regards --martin # ceph-volume lvm create --bluestore --data vgsdb/sdb1 --> Incompatible flags were found, some values may get ignored --> Cannot use None (None) with --bluestore (bluestore) Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 19959f26-f2ed-451d-acb7-a808d15510cf Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -h ceph:ceph /dev/vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/ln -s /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 AuthRegistry(0x772d200660d0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx stderr: got monmap epoch 20 --> Creating keyring file for osd.0 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 19959f26-f2ed-451d-acb7-a808d15510cf --setuser ceph --setgroup ceph stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found stderr: 2025-05-16T14:47:09.190+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vgsdb/sdb1 --path /var/lib/ceph/osd/ceph-0 --no-mon-config Running command: /usr/bin/ln -snf /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf.service → /lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@0 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)0.service → /lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@0 --> ceph-volume lvm activate successful for osd ID: 0 --> ceph-volume lvm create successful for: vgsdb/sdb1 -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

3 3

[reef] radosgw user has objects but no bucket
by Boris 16 May '25

16 May '25

Hi, I am in the process of cleaning things up and just saw that we have a user, that seems to have data but no buckets. # radosgw-admin user stats --uid 3853f960-6d72-4760-95ea-5e8f2a571840 --sync-stats { "stats": { "size": 189946686516, "size_actual": 189946687488, "size_kb": 185494812, "size_kb_actual": 185494812, "num_objects": 12069 }, "last_stats_sync": "2025-05-16T08:03:49.458563Z", "last_stats_update": "2018-02-09T14:17:26.334536Z" } # radosgw-admin bucket list --uid 3853f960-6d72-4760-95ea-5e8f2a571840 [] How can I find the users data, to verify if this can be removed? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

quincy 17.2.9 hotfix QE validation status
by Yuri Weinstein 15 May '25

15 May '25

Details of this release are summarized here: https://tracker.ceph.com/issues/71311#note-1 Release Notes - https://github.com/ceph/ceph/pull/63267 LRC upgrade - N/A Seeking approvals/reviews for: rados - Laura, Adam Kupczyk approved? This hotfix testing was limited to smoke and rados suites.

2 1

Scrubbing + Deep crubbing
by Devender Singh 15 May '25

15 May '25

Hello All Need some help to tune scrubbing. 1. How to control not run scrubbing and deep scrubbing together and which one should be started first? 124 active+clean+remapped 35 active+clean+scrubbing+deep 24 active+clean+scrubbing 4 active+clean+remapped+scrubbing+deep I have osd_max_scrubs =1 but still impacting IO’s special when both are running together. 2. When changing some priorities to not to impact IO’s, Setting higher number is lower priority or it works differently with different kind of priority parameters? What is the standard of defining high and low priority(higher number is lower priority or lower number is lower priory etc...) on CEPH? eg: osd_requested_scrub_priority - 170 (120 default) Eg: ceph config set osd osd_scrub_event_cost 8192 (4096 default) Regards Dev

2 2

OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors
by Alex 15 May '25

15 May '25

Hi everyone. Help me settle a debate. My coworker is seeing OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors. We both agree they are caused by bad drives. The fix is to mark the drive as out, replace it and add it back in. Whenever we see this error on Ceph we see corresponding read errors on the physical drive. I'm saying that even though the drive is bad since there are two more copies, only 1 of 3 drives has bad sectors preventing the data from being accessed which is what dmesg is showing ie critical medium error, dev sd..., sector 12345... There should not be OBJECT_UNFOUND since Ceph compares the remaining two copies and assuming the data matches, it should be able to recover on it's own and move the data to another PG or maybe OBJECT_UNFOUND and PG_DAMAGED are warnings not errors. My coworker is saying because the primary OSD responsible for coordinating the PG was the one which failed, and is the "source of truth" the cluster goes into error state. His argument doesn't make sense to me since there should be no single point of failure, but I'm also not sure about my argument since I don't know enough about how Ceph works under the hood. Thanks.

3 9

Repo name bug?
by Alex 15 May '25

15 May '25

Good morning everyone. Does the preflight playbook have a bug? https://github.com/ceph/cephadm-ansible/blob/devel/cephadm-preflight.yml Line 82: paths: "{{ ['noarch', '$basearch'] if ceph_origin == 'community' else ['$basearch'] }}" The yum repo file then gets named ceph_stable_$basearch. Shouldn't it be basearch without the $ ? - Alex

3 10

write latency increase after upgrade from pacific to quincy
by Nima AbolhassanBeigi 15 May '25

15 May '25

Hi dear ceph community We have encountered an issue with our ceph cluster after upgrading from v16.2.13 to v17.2.7. The issue is that the write latency on OSDs has increased significantly and doesn't seem to plummet back down. The average write latency has almost doubled, and this has happened since we upgraded the OSDs. If anybody could help figure this out. Kind regards Nima

2 4

18.2.6 : OSD(s) experiencing slow operations in BlueStore
by Michel Jouvin 15 May '25

15 May '25

Hi, Since our upgrade to 18.2.6 2 days ago, our cluster is reporting the warning "1 OSD(s) experiencing slow operations in BlueStore": [root@dig-osd4 bluestore-slow-ops]# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.247 observed slow operation indications in BlueStore I have never seen this warning before so I've the feeling it is somehow related to the upgrade and it doesn't seem related to the regression mentioned in another thread (that should result in an OSD crash). Googling quickly, I found this reported on 19.2.1 with SSD where in my case it is an HDD. I don't know if the workaround mentioned in the issue (bdev_xxx_discard=true) also applies to 18.2.6... Did somebody saw this in 18.2.x? Any recommandation? Our plan was, according to best practicies described recently in another thread to move from 18.2.2 to 18.2.6 and then from 18.2.6 to 19.2.2... Will 19.2.2 clear this issue (at the risk of others as it is probably not widely used)? Best regards, Michel

8 12

reef radosgw returns 503/slow down when creating a bucket with location
by Boris 14 May '25

14 May '25

Hi, we have a multi zonegroup setup and since after the upgrade from pacific to reef we encountered an issue with creating buckets. When you try to create a bucket in another zonegroup and set the `bucket_location` (Region to create bucket in) parameter in the s3cmd, the radosgw answer with (503 (ServiceUnavailable)) 'reason': 'Slow Down' After removing the bucket_location from the config it just works. I use s3cmd with a really bare bones config. [default] # bucket_location = zg-b host_base = zg-b.test.dev host_bucket = %(bucket)s.zg-b.test.dev access_key = xxx secret_key = yyy website_endpoint = https://%(bucket)s.s3-website-zg-b.test.dev Is this a bug or expected behavior. Cheers Boris

2 1

Re: check bucket index on large bucket leads to laggy index PGs
by Boris 13 May '25

13 May '25

> > resharding the bucket is indeed the solution. while resharding does > have to read all of the keys from the source index objects, it doesn't > read all of them at once. writing these keys to the target bucket > index objects is the more expensive part, but those are different > objects/pgs and should be better distributed Ah great. Will try that. Ensure the index pool is really on only SSDs. I’ve seen crush rules not > specifying device class. > Yes they are. On dedicated NVMEs that we use for the meta pools (the listing I've sent in slack). Do you have autoresharding disabled? Versioned objects? Can you do a > bilog trim? Could you preshard a new bucket and move the objects? > No autoresharding enabled. We wanted to test sharding in the multisite setting before we enable it (and reshard all the buckets that need it in a controlled way). Yes, the bucket has versioned objects. We are in the process of deleting it, but the deletion is now running since two days and the bucket index is still very large. I can try to do a bilog trim. Need to read up what it does and how to do it. I could move the data to a new preshareded pool, but it feels like the bucket is somehow broken, because deleting is now working as I expect it. I will try to reshard the bucket tonight and hope it will work out. The explanation from Casey sounds promising. As I have a lot more buckets with a lot more objects (according to the bucket index) this needs to be done anyway. Cheers Boris Am Fr., 9. Mai 2025 um 19:21 Uhr schrieb Anthony D'Atri < anthony.datri(a)gmail.com>: > Ensure the index pool is really on only SSDs. I’ve seen crush rules not > specifying device class. > > Do you have autoresharding disabled? Versioned objects? Can you do a > bilog trim? Could you preshard a new bucket and move the objects? > > > On May 9, 2025, at 12:54 PM, Boris <bb(a)kervyn.de> wrote: > > > > Hi, > > > > I have a bucket that got >20m index entries but only got 11 shards. > > > > When I try to run a radosgw-admin bucket check the PGs that hold the > index start to become laggy after a couple of seconds. I need to stop it > because it kills the whole object storage. > > > > This is a latest reef cluster and the master of a multisite which only > replicates the metadata (1 realm, multiple zonegroups, one zone per > zonegroup). > > > > Any ideas what I can do? > > I fear to reshard the bucket, because I am not sure if I can stop the > resharding if the PGs become laggy. > > > > Cheers > > Boris > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 2

CephPGImbalance warning for different OSD sizes
by Eugen Block 13 May '25

13 May '25

Hi *, since the question came up yesterday on this list, I decided to share our workaround. I created a tracker issue [0] as well as a blog post [1] with some more details. The current Prometheus module relies on uniform OSD sizes, which is not very common, at least not from our experience or the reports on this list. So we added a new metric to the mgr prometheus module and modified the alert expression which only compares OSDs of the same size (since the crush weight is calculated similarly, we just called the metric osd_crush_weight). This is a bit hacky, not persistent across updates etc., but it has worked great so far. It would be great if the mgr module could be improved. I'm sure there are more elegant ways to do that, but with this approach we didn't need to introduce anything new, just utilized what was already there. Best regards, Eugen [0] https://tracker.ceph.com/issues/71310 [1] https://heiterbiswolkig.blogs.nde.ag/2025/05/13/cephadm-pg-imbalance/

1 0

reef upgrade 2.2 to 2.7 - slow operations in bluestore
by Steven Vacaroaia 13 May '25

13 May '25

Hi, After a cephadm upgrade from 18.2.2 to 18.2.7 that worked perfectly, I am noticing lots ( 42 out of 161 OSDs ) "slow operations in bluestore" errors My cluster has all 3 types of OSDs ( NVME, SSD, HDD+ journaling) I found some articles mentioning below setting but it did not help Anyone else having this issue ? Thanks Steven bdev_enable_discard: "true" # quote bdev_async_discard_threads: "1" # quote

4 4

Ceph 19.2.2 ceph-exporter and lvm zap
by Kristaps Cudars 13 May '25

13 May '25

Hello, After upgrading to 19.2.2 ceph-exporter container stopped exposing date. As workaround implemented exclude_perf_counters true Is anyone else experiencing this? And it seems that this is true for 19.2.1/2 from cephadm shell > ceph-volume lvm zap is not working. If you install “dnf install ceph-volume” than from OS ceph-volume lvm zap is working. This is expected and must be performed from OS on 19.2 or is it a bug?

2 2

cephfs: Moving directories with snapshots
by Kyriazis, George 12 May '25

12 May '25

Hello, I have a question regarding how snapshots are being treated on a directory that is moved out of a snapshot’ed are. My example: Let’s say I have a directory structure as follows: A/B/C A/D I have created snapsho schedules on A/B, and I want to move (with ‘mv’) directory C so it becomes A/D/C. There are no snapshots on A/D (yet). Will the existing snapshots “move” with the directory, or will they stay under the A/B hierarchy? My concern is that the C directory is 35TB, and if I mv the directory under A/D I may end up duplicating space. Ie, will I end up with 35TB stored in the old snapshots under A/B, and the actual 35TB on the live data under A/D/C ? Thank you for the clarification! George

1 0

Slow osd + 30% deviates (toward higher pgs or lower pgs numbers on osd)
by Devender Singh 12 May '25

12 May '25

Hello all Seeking some help to solve below… 1. # ceph health detail HEALTH_WARN 2 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 2 OSD(s) experiencing slow operations in BlueStore osd.10 observed slow operation indications in BlueStore osd.21 observed slow operation indications in BlueStore Having some other OSD’s with same size with more PG’s but only complaining on below… # ceph osd df |sort +17|egrep -i "^10|^21" 21 ssd 6.98630 1.00000 7.0 TiB 2.7 TiB 2.6 TiB 17 MiB 27 GiB 4.3 TiB 38.22 0.85 103 up 10 ssd 6.98630 1.00000 7.0 TiB 4.0 TiB 4.0 TiB 21 MiB 35 GiB 2.9 TiB 57.97 1.28 123 up 2. What does deviate means towards? holding higher number of pgs or Lower? OSD osd.6 on host1 deviates by more than 30% from average PG count. OSD osd.5 onhost2 deviates by more than 30% from average PG count. # ceph osd df |sort +17|egrep -i "^ 5|^ 6” 5 ssd 6.98630 1.00000 7.0 TiB 2.8 TiB 2.8 TiB 11 MiB 32 GiB 4.2 TiB 39.91 0.88 97 up 6 ssd 6.98630 1.00000 7.0 TiB 3.2 TiB 3.1 TiB 19 MiB 48 GiB 3.8 TiB 45.65 1.01 103 up Regards Dev

2 1

Notes from Ceph Steering Committee Weekly 2025-05-12
by Patrick Donnelly 12 May '25

12 May '25

Hello, Here are the minutes from today's meeting. - [DavidG] Offer for Z hardware. Non-mandatory PR tests? - tl;dr - Sure if they're willing to manage it. Run on main branch once a day? Would need enough hardware to match what we have for ARM? If they want to build it, sure. IBM Cloud? Crunch numbers on bare minimum useful new architecture hardware. - [DavidG] GitHub Security Improvements - Enforcing 2FA - yes - Downgrade non to outside collaborators? - One of the recommendations here: https://docs.google.com/document/d/1tEPuFv_UDYUmYcejMNZgTgZALI7jTSSye4wlgHC… - Periodic user removal from GitHub org? What interval of non-contribution? - 1yr no PR or Review. Will need to craft some API calls. - [Patrick] new pending release notes format - https://tracker.ceph.com/issues/71198 - Will move forward with this. - [Yehuda] cephx changes update - [Matt] revisit ceph-dashboard tests, there appeared to be a majority opinion these could be moved to teuthology (where they already can run); but we learned last week that "it was decided" not to? could we get a formal vote on this? that would allow everyone a voice - Seeking a resolution on how to run tests of this nature in the future: - Pull out Github CI and make part of all component lead QA testing in teuthology. - Tabled for dedicated discussion/meetign with Ernesto and other stakeholders. There seems to be somewhat broad agreement that these tests would be better in the teuthology QA suite going forward. - [Yuri] next quincy release for bluestore bugfix - https://github.com/ceph/ceph/pull/63122 - TODO: archive current quincy HEAD and reset to last bug fix release. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

MANY_OBJECTS_PER_PG & TOO_MANY_PGS
by Torkil Svensgaard 12 May '25

12 May '25

Hi 19.2.2 Got a couple new warnings when I doubled pg_num for the cephfs.hdd.data pool: [root@lazy ~]# ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average; too many PGs per OSD (261 > max 250) [WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than average pool cephfs.cephfs.data objects per pg (4084905) is more than 161.862 times cluster average (25237) [WRN] TOO_MANY_PGS: too many PGs per OSD (261 > max 250) [root@lazy ~]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 4.1 PiB 1.6 PiB 2.5 PiB 2.5 PiB 61.12 nvme 210 TiB 58 TiB 152 TiB 152 TiB 72.49 nvmebulk 196 TiB 149 TiB 47 TiB 47 TiB 23.83 ssd 49 TiB 32 TiB 16 TiB 16 TiB 33.66 TOTAL 4.5 PiB 1.8 PiB 2.7 PiB 2.7 PiB 59.77 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL rbd 4 2048 111 TiB 29.39M 288 TiB 25.04 288 TiB libvirt 5 256 3.3 TiB 872.28k 6.8 TiB 37.73 3.7 TiB rbd_internal 6 2048 103 TiB 32.71M 243 TiB 21.96 288 TiB .mgr 8 1 4.9 GiB 1.26k 2.1 GiB 0.02 5.6 TiB rbd_ec 10 32 9.0 MiB 27 4.6 MiB 0 3.7 TiB rbd_ec_data 11 16384 1.1 PiB 284.37M 1.4 PiB 63.10 576 TiB rbd.nvme 23 2048 95 TiB 25.16M 147 TiB 77.45 21 TiB .nfs 25 32 20 KiB 73 286 KiB 0 3.7 TiB cephfs.cephfs.meta 31 128 15 GiB 3.05M 46 GiB 0.40 3.7 TiB cephfs.cephfs.data 32 512 449 B 130.72M 48 KiB 0 3.7 TiB cephfs.nvme.data 34 32 977 GiB 250k 122 GiB 0.28 21 TiB cephfs.ssd.data 35 32 853 GiB 1.10M 1.9 TiB 14.65 3.7 TiB cephfs.hdd.data 37 3098 208 TiB 175.22M 429 TiB 33.18 384 TiB rbd.ssd 39 64 1.6 TiB 433.36k 4.3 TiB 27.65 3.7 TiB rbd.ssd.ec 43 32 2.5 KiB 5 20 KiB 0 3.7 TiB rbd.ssd.ec.data 44 32 1.0 TiB 269.92k 2.0 TiB 14.89 5.0 TiB rbd.nvmebulk.ec 47 32 3.0 MiB 6 6.1 MiB 0 8.3 TiB rbd.nvmebulk.data 48 512 23 TiB 6.00M 46 TiB 64.98 11 TiB MANY_OBJECTS_PER_PG: The ceph.cephfs.data pool stores 449B in 130.72M objects? That is weird? We don't actually use that pool as as far as I know, as we use media specific data pools for cephfs, but is evidently being used for something regardless? It was created as part of the original cephfs file system where the command created both .meta and .data. TOO_MANY_PGS: [root@lazy ~]# ceph config dump | grep mon| grep max mon advanced mon_max_pg_per_osd 1000 osd advanced mon_max_pg_per_osd 1000 Well above 250. There used to be a mon_pg_warn_max_per_osd key but that was removed back in Luminous so what controls this warning? [root@lazy ~]# ceph config get mon mon_pg_warn_max_per_osd Error ENOENT: unrecognized key 'mon_pg_warn_max_per_osd' Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 1

Staggered upgrade or not
by Kasper Rasmussen 12 May '25

12 May '25

Hi Upgrading from Pacific 16.2.15 to Quincy 17.2.5 Environment: Production Size: 9PB RAW capacity 5 mon/mgr/mds host. 56 storage nodes I’m looking for advice to help me decide between - Staggered upgrade or not. Thanks

5 5

We lost the stability of the cluster, 18.2.2 -> 18.2.6 -> 19.2.1 Chain of upgrade failure
by Senol COLAK 12 May '25

12 May '25

Hello, After upgrading from ceph reef 18.2.6 to ceph squid 19.2.1 I restarted the osds and they remained down. The events contain the following records: root@cmt6770:~# ceph health detail HEALTH_WARN 1 filesystem is degraded; 1 MDSs report slow metadata IOs; mon cmt6770 is low on available space; 9 osds down; 1 host (3 osds) down; 5 nearfull osd(s); Reduced data availability: 866 pgs inactive, 489 pgs down, 5 pgs incomplete, 60 pgs stale; Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull; Degraded data redundancy: 432856/3408880 objects degraded (12.698%), 191 pgs degraded, 181 pgs undersized; 12 pool(s) nearfull; 255 slow ops, oldest one blocked for 2417 sec, daemons [osd.10,osd.12,osd.21,osd.22,osd.23] have slow ops. [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.cmt5923(mds.0): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 38201 secs [WRN] MON_DISK_LOW: mon cmt6770 is low on available space mon.cmt6770 has 28% avail [WRN] OSD_DOWN: 9 osds down osd.0 (root=default,host=cmt6770) is down osd.1 (root=default,host=cmt6770) is down osd.2 (root=default,host=cmt6461) is down osd.7 (root=default,host=cmt7773) is down osd.8 (root=default,host=cmt5923) is down osd.9 (root=default,host=cmt5923) is down osd.14 (root=default,host=cmt6770) is down osd.17 (root=default,host=cmt6461) is down osd.24 (root=default,host=cmt6461) is down [WRN] OSD_HOST_DOWN: 1 host (3 osds) down host cmt6461 (root=default) (3 osds) is down [WRN] OSD_NEARFULL: 5 nearfull osd(s) osd.3 is near full osd.12 is near full osd.16 is near full osd.21 is near full osd.23 is near full [WRN] PG_AVAILABILITY: Reduced data availability: 866 pgs inactive, 489 pgs down, 5 pgs incomplete, 60 pgs stale pg 7.1c5 is down, acting [3] pg 7.1c7 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1c8 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1cb is stuck inactive for 3h, current state unknown, last acting [] pg 7.1cd is down, acting [15,3] pg 7.1ce is stuck inactive for 3h, current state unknown, last acting [] pg 7.1cf is stuck inactive for 3h, current state unknown, last acting [] pg 7.1d0 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1d1 is down, acting [29,13] pg 7.1d2 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1d3 is down, acting [23] pg 7.1d4 is down, acting [16] pg 7.1d5 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1d6 is down, acting [3] pg 7.1d9 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1da is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e0 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e1 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e2 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e4 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1e5 is down, acting [12,29] pg 7.1e7 is stuck stale for 7m, current state stale+down, last acting [9] pg 7.1e8 is down, acting [12] pg 7.1e9 is stuck stale for 31m, current state stale, last acting [2] pg 7.1eb is stuck inactive for 3h, current state unknown, last acting [] pg 7.1ed is down, acting [3] pg 7.1ee is stuck inactive for 3h, current state unknown, last acting [] pg 7.1ef is down, acting [12] pg 7.1f0 is down, acting [10] pg 7.1f1 is down, acting [12,29] pg 7.1f2 is down, acting [16] pg 7.1f3 is stuck stale for 7m, current state stale, last acting [9] pg 7.1f4 is down, acting [22] pg 7.1f5 is down, acting [22] pg 7.1f8 is down, acting [29] pg 7.1f9 is stuck inactive for 3h, current state unknown, last acting [] pg 7.1fb is stuck inactive for 3h, current state unknown, last acting [] pg 7.1fc is stuck stale for 15m, current state stale+down, last acting [29] pg 7.1fd is down, acting [3] pg 7.1fe is down, acting [3,15] pg 7.1ff is down, acting [3] pg 7.201 is down, acting [12] pg 7.204 is down, acting [10] pg 7.205 is down, acting [13] pg 7.207 is down, acting [11] pg 7.20a is down, acting [3] pg 7.20b is down, acting [22] pg 7.20d is stuck inactive for 3h, current state unknown, last acting [] pg 7.210 is stuck inactive for 3h, current state unknown, last acting [] pg 7.211 is stuck inactive for 3h, current state unknown, last acting [] pg 7.21b is down, acting [16] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull pg 7.20 is active+undersized+degraded+remapped+backfill_toofull, acting [3] pg 19.2b is active+remapped+backfill_toofull, acting [12,3] pg 19.6b is active+remapped+backfill_toofull, acting [12,3] pg 20.55 is active+remapped+backfill_toofull, acting [29,3] pg 24.6 is active+undersized+degraded+remapped+backfill_toofull, acting [22] pg 24.b is active+undersized+degraded+remapped+backfill_toofull, acting [21] pg 24.13 is active+undersized+degraded+remapped+backfill_toofull, acting [23] pg 24.16 is active+undersized+degraded+remapped+backfill_toofull, acting [21] pg 24.1d is active+undersized+degraded+remapped+backfill_toofull, acting [21] [WRN] PG_DEGRADED: Degraded data redundancy: 432856/3408880 objects degraded (12.698%), 191 pgs degraded, 181 pgs undersized pg 7.3f is stuck undersized for 9m, current state active+undersized+degraded, last acting [16] pg 7.56 is stuck undersized for 21m, current state active+undersized+degraded, last acting [13] pg 7.61 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.66 is stuck undersized for 7m, current state active+undersized+degraded, last acting [29] pg 7.6b is stuck undersized for 25m, current state active+undersized+degraded, last acting [21] pg 7.102 is active+undersized+degraded, acting [23] pg 7.118 is stuck undersized for 14m, current state active+undersized+degraded, last acting [23] pg 7.11c is stuck undersized for 9m, current state active+undersized+degraded, last acting [3] pg 7.133 is stuck undersized for 7m, current state active+undersized+degraded, last acting [12] pg 7.139 is stuck undersized for 7m, current state active+undersized+degraded, last acting [13] pg 7.143 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.155 is stuck undersized for 25m, current state active+undersized+degraded, last acting [3] pg 7.156 is active+undersized+degraded, acting [12] pg 7.15e is stuck undersized for 31m, current state active+undersized+degraded, last acting [3] pg 7.15f is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.168 is stuck undersized for 67m, current state active+undersized+degraded, last acting [22] pg 7.17f is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.180 is stuck undersized for 14m, current state active+undersized+degraded, last acting [21] pg 7.18e is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.193 is active+undersized+degraded, acting [16] pg 7.197 is stuck undersized for 14m, current state active+undersized+degraded, last acting [21] pg 7.1a6 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.1b7 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 7.1c6 is stuck undersized for 14m, current state active+undersized+degraded, last acting [22] pg 7.1ca is stuck undersized for 14m, current state active+undersized+degraded, last acting [22] pg 7.1d7 is stuck undersized for 9m, current state active+undersized+degraded, last acting [22] pg 7.1df is active+undersized+degraded, acting [21] pg 7.1e6 is stuck undersized for 10h, current state active+undersized+degraded, last acting [23] pg 7.200 is active+undersized+degraded, acting [29] pg 7.202 is stuck undersized for 7m, current state active+undersized+degraded, last acting [13] pg 7.20c is stuck undersized for 10h, current state active+undersized+degraded, last acting [16] pg 7.20e is stuck undersized for 47m, current state active+undersized+degraded, last acting [23] pg 7.20f is stuck undersized for 47m, current state active+undersized+degraded, last acting [23] pg 7.217 is stuck undersized for 7m, current state active+undersized+degraded, last acting [21] pg 15.35 is active+undersized+degraded, acting [22] pg 16.2a is stuck undersized for 10h, current state active+undersized+degraded, last acting [21] pg 19.43 is stuck undersized for 31m, current state active+undersized+degraded, last acting [23] pg 19.44 is stuck undersized for 8m, current state active+undersized+degraded, last acting [29] pg 19.4e is stuck undersized for 14m, current state active+undersized+degraded, last acting [16] pg 19.52 is active+undersized+degraded+wait, acting [3] pg 19.55 is stuck undersized for 25m, current state active+undersized+degraded, last acting [23] pg 19.61 is stuck undersized for 25m, current state active+undersized+degraded, last acting [21] pg 19.72 is stuck undersized for 31m, current state active+undersized+degraded, last acting [3] pg 20.42 is stuck undersized for 7m, current state active+undersized+degraded, last acting [23] pg 20.48 is stuck undersized for 67m, current state active+undersized+degraded, last acting [16] pg 20.5b is stuck undersized for 21m, current state active+undersized+degraded, last acting [12] pg 20.5f is stuck undersized for 10h, current state active+undersized+degraded, last acting [12] pg 20.65 is stuck undersized for 10m, current state active+undersized+degraded, last acting [23] pg 20.6a is active+undersized+degraded, acting [13] pg 20.71 is stuck undersized for 31m, current state active+undersized+degraded, last acting [13] pg 20.7d is stuck undersized for 7m, current state active+undersized+degraded, last acting [29] [WRN] POOL_NEARFULL: 12 pool(s) nearfull pool '.mgr' is nearfull pool 'DataStore' is nearfull pool 'cephfs_data' is nearfull pool 'cephfs_metadata' is nearfull pool 'OS' is nearfull pool 'cloud' is nearfull pool 'DataStore_2' is nearfull pool 'DataStore_3' is nearfull pool 'MGMT' is nearfull pool 'DataStore_4' is nearfull pool 'DataStore_5' is nearfull pool 'fast' is nearfull [WRN] SLOW_OPS: 255 slow ops, oldest one blocked for 2417 sec, daemons [osd.10,osd.12,osd.21,osd.22,osd.23] have slow ops. root@cmt6770:~# ceph -s cluster: id: 9319dafb-3408-46cb-9b09-b3d381114545 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs mon cmt6770 is low on available space 9 osds down 1 host (3 osds) down 5 nearfull osd(s) Reduced data availability: 866 pgs inactive, 489 pgs down, 5 pgs incomplete, 60 pgs stale Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull Degraded data redundancy: 432856/3408880 objects degraded (12.698%), 191 pgs degraded, 181 pgs undersized 12 pool(s) nearfull 255 slow ops, oldest one blocked for 2422 sec, daemons [osd.10,osd.12,osd.21,osd.22,osd.23] have slow ops. services: mon: 2 daemons, quorum cmt6770,cmt5923 (age 70m) mgr: cmt6770(active, since 3h) mds: 1/1 daemons up, 1 standby osd: 25 osds: 11 up (since 14s), 20 in (since 9m); 182 remapped pgs data: volumes: 0/1 healthy, 1 recovering pools: 12 pools, 1589 pgs objects: 1.70M objects, 6.2 TiB usage: 9.1 TiB used, 6.2 TiB / 15 TiB avail pgs: 28.760% pgs unknown 34.991% pgs not active 432856/3408880 objects degraded (12.698%) 388136/3408880 objects misplaced (11.386%) 466 down 457 unknown 209 active+clean 185 active+undersized+degraded 157 active+clean+remapped 62 stale 20 stale+down 9 active+undersized+remapped 6 active+undersized+degraded+remapped+backfill_toofull 5 incomplete 3 active+remapped+backfill_toofull 2 active+clean+scrubbing+deep 2 active+clean+remapped+scrubbing+deep 2 down+remapped 1 stale+creating+down 1 active+remapped+backfilling 1 active+remapped+backfill_wait 1 active+undersized+remapped+wait io: recovery: 12 MiB/s, 3 objects/s root@cmt6770:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 34.05125 root default -3 5.38539 host cmt5923 3 ssd 1.74660 osd.3 up 0.79999 1.00000 8 ssd 1.81940 osd.8 down 1.00000 1.00000 9 ssd 1.81940 osd.9 down 1.00000 1.00000 -15 4.40289 host cmt6461 24 nvme 0.90970 osd.24 down 0.79999 1.00000 2 ssd 1.74660 osd.2 down 1.00000 1.00000 17 ssd 1.74660 osd.17 down 1.00000 1.00000 -5 5.35616 host cmt6770 0 ssd 0.87329 osd.0 down 1.00000 1.00000 1 ssd 0.87329 osd.1 down 1.00000 1.00000 4 ssd 1.86299 osd.4 down 0 1.00000 14 ssd 0.87329 osd.14 down 1.00000 1.00000 15 ssd 0.87329 osd.15 up 1.00000 1.00000 -9 7.24838 host cmt7773 5 nvme 1.81940 osd.5 down 0 1.00000 19 nvme 1.81940 osd.19 down 0 1.00000 7 ssd 1.74660 osd.7 down 1.00000 1.00000 29 ssd 1.86299 osd.29 up 1.00000 1.00000 -13 7.93245 host dc2943 22 nvme 0.90970 osd.22 up 1.00000 1.00000 23 nvme 0.90970 osd.23 up 1.00000 1.00000 6 ssd 1.74660 osd.6 down 0 1.00000 10 ssd 0.87329 osd.10 up 1.00000 1.00000 11 ssd 0.87329 osd.11 up 1.00000 1.00000 12 ssd 0.87329 osd.12 up 1.00000 1.00000 13 ssd 0.87329 osd.13 up 1.00000 1.00000 16 ssd 0.87329 osd.16 up 0.79999 1.00000 -11 3.72598 host dc3658 20 nvme 1.86299 osd.20 down 0 1.00000 21 nvme 1.86299 osd.21 up 0.90002 1.00000 root@cmt6770:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 3 ssd 1.74660 0.79999 1.7 TiB 1.5 TiB 1.5 TiB 228 KiB 2.9 GiB 251 GiB 85.96 1.44 214 up 8 ssd 1.81940 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 9 ssd 1.81940 1.00000 1.8 TiB 776 MiB 745 MiB 8 KiB 31 MiB 1.8 TiB 0.04 0 46 down 24 nvme 0.90970 0.79999 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 2 ssd 1.74660 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 4 down 17 ssd 1.74660 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 0 ssd 0.87329 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 3 down 1 ssd 0.87329 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 4 ssd 1.86299 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 14 ssd 0.87329 1.00000 894 GiB 792 MiB 752 MiB 14 KiB 40 MiB 893 GiB 0.09 0.00 25 down 15 ssd 0.87329 1.00000 894 GiB 232 GiB 231 GiB 14 KiB 1.4 GiB 662 GiB 25.98 0.44 83 up 5 nvme 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 19 nvme 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 7 ssd 1.74660 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 29 ssd 1.86299 1.00000 1.9 TiB 1.5 TiB 1.5 TiB 323 KiB 2.8 GiB 354 GiB 81.44 1.37 222 up 22 nvme 0.90970 1.00000 932 GiB 689 GiB 687 GiB 181 KiB 1.6 GiB 243 GiB 73.96 1.24 139 up 23 nvme 0.90970 1.00000 932 GiB 820 GiB 818 GiB 138 KiB 2.0 GiB 112 GiB 87.98 1.48 144 up 6 ssd 1.74660 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 10 ssd 0.87329 1.00000 894 GiB 237 GiB 235 GiB 1 KiB 1.2 GiB 658 GiB 26.46 0.44 82 up 11 ssd 0.87329 1.00000 894 GiB 264 GiB 263 GiB 1 KiB 1.4 GiB 630 GiB 29.54 0.50 67 up 12 ssd 0.87329 1.00000 894 GiB 780 GiB 778 GiB 123 KiB 1.8 GiB 114 GiB 87.26 1.46 113 up 13 ssd 0.87329 1.00000 894 GiB 684 GiB 682 GiB 170 KiB 1.9 GiB 210 GiB 76.48 1.28 98 up 16 ssd 0.87329 0.79999 894 GiB 779 GiB 777 GiB 149 KiB 1.8 GiB 116 GiB 87.06 1.46 86 up 20 nvme 1.86299 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 21 nvme 1.86299 0.90002 1.9 TiB 1.7 TiB 1.7 TiB 430 KiB 3.5 GiB 194 GiB 89.84 1.51 314 up TOTAL 15 TiB 9.1 TiB 9.1 TiB 1.7 MiB 22 GiB 6.2 TiB 59.60 MIN/MAX VAR: 0/1.51 STDDEV: 44.89 also osd start logs: May 10 15:38:32 cmt5923 systemd[1]: ceph-osd(a)9.service: Failed with result 'signal'. May 10 15:38:37 cmt5923 ceph-osd[2383902]: 2025-05-10T15:38:37.579+0300 764caf13f880 -1 osd.8 100504 log_to_monitors true May 10 15:38:37 cmt5923 ceph-osd[2383902]: 2025-05-10T15:38:37.791+0300 764c8c64b6c0 -1 log_channel(cluster) log [ERR] : 7.26a past_intervals [96946,100253) start interval does not contain the required bound [93903,100253) start May 10 15:38:37 cmt5923 ceph-osd[2383902]: 2025-05-10T15:38:37.791+0300 764c8c64b6c0 -1 osd.8 pg_epoch: 100377 pg[7.26a( empty local-lis/les=0/0 n=0 ec=96946/96946 lis/c=96236/93898 les/c/f=96237/93903/91308 sis=100253) [3,1] r=-1 lpr=100376 pi=[96946,100253)/3 crt=0'0 mlcod 0'0 unknown mbc={}] PeeringState::check_past_interval_bounds 7.26a past_intervals [96946,100253) start interval does not contain the required bound [93903,100253) start We appreciate any support and guidance, Thanks in advance

5 6

Ceph Day Seattle is almost here!
by Anthony D'Atri 11 May '25

11 May '25

There’s still time to register to attend this community event sponsored by OSNexus and Clyso. Seattle, Washington USA May 15 2025 The schedule and registration link can be found here: https://ceph.io/en/community/events/2025/ceph-day-seattle/ Ceph Days Seattle 2025 ceph.io https://ceph.io/en/community/events/2025/ceph-day-seattle/

1 0

failing to enable disk failure prediction
by Jens Galsgaard 10 May '25

10 May '25

Hello, I’ve a cluster built with cephadm running on Debian 12/Bookworm. Ceph 18.2.5. I want to enable disk failure prediction and run this command: ceph mgr module enable diskprediction_local Then the cluster goes into ERROR state and the logs shows: 2025-04-14T08:55:37.073118+0200 mgr.host01.eqvsde [ERR] Unhandled exception from module 'diskprediction_local' while running on mgr.host01.eqvsde: No module named 'sklearn.svm.classes' 2025-04-14T08:55:38.388289+0200 mon.host03 [ERR] Health check failed: Module 'diskprediction_local' has failed: No module named 'sklearn.svm.classes' (MGR_MODULE_ERROR) How to add sklearn to the container as it is obviously missing? Venlig hilsen - Mit freundlichen Grüßen - Kind Regards, Jens Galsgaard Gitservice.dk v/ Jens Galsgaard Sparresmindevej 7 4780 Stege Danmark Tlf: +45 46551100 Mob: +45 28864340 CVR/VAT: DK30128540

2 5

Full OSD with almost empty pools
by Adam Prycki 10 May '25

10 May '25

Hello, we have just discovered new issue with our cephfs cluster. All of our SSD osd have been flagged as nearful. Ceph df on our cluster looks very wrong. Raw storage usage shows that SSDs are almost full, but SSD pools are hardly using any space. ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 27 PiB 16 PiB 11 PiB 11 PiB 40.57 ssd 71 TiB 10 TiB 60 TiB 60 TiB 85.49 TOTAL 27 PiB 16 PiB 11 PiB 11 PiB 40.68 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 16 47 GiB 12.12k 142 GiB 2.48 1.8 TiB cephfs.arm-vol.meta 2 4096 19 GiB 2.02M 57 GiB 1.01 1.8 TiB cephfs.arm-vol.data 4 16384 7.7 PiB 2.14G 10 PiB 58.29 5.5 PiB .rgw.root 24 32 16 KiB 24 264 KiB 0 2.4 PiB bst-ro-backup.rgw.log 25 32 343 MiB 2.90k 1.0 GiB 0 2.4 PiB bst-ro-backup.rgw.control 26 32 0 B 8 0 B 0 2.4 PiB bst-ro-backup.rgw.meta 27 32 198 KiB 440 5.0 MiB 0 2.4 PiB .nfs 32 32 8.2 KiB 10 99 KiB 0 2.4 PiB cephfs.arm-vol.meta pool seems to store 19GiB of data. rados -p cephfs.arm-vol.meta | wc -l confirms that there are about 2.02M objects in that pool. Our SSD OSDs are almost full ceph osd df ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 807 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 395 MiB 1.3 GiB 64 GiB 85.78 1.00 85 up 812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.0 GiB 66 GiB 85.23 1.00 74 up 569 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.0 GiB 66 GiB 85.24 1.00 72 up 577 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 334 MiB 1.5 GiB 67 GiB 85.12 1.00 72 up 571 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 369 MiB 2.3 GiB 66 GiB 85.30 1.00 86 up 578 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 329 MiB 1.8 GiB 66 GiB 85.19 1.00 68 up 1775 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 379 MiB 1.2 GiB 64 GiB 85.70 1.00 76 up 1838 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 2.3 GiB 66 GiB 85.29 1.00 74 up 1783 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 443 MiB 2.5 GiB 62 GiB 86.05 1.01 98 up 1846 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.7 GiB 66 GiB 85.18 1.00 85 up 1854 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 384 MiB 2.4 GiB 66 GiB 85.33 1.00 80 up 1855 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.6 GiB 65 GiB 85.37 1.00 78 up 1761 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 368 MiB 1.7 GiB 63 GiB 85.86 1.00 79 up 1826 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 331 MiB 2.2 GiB 66 GiB 85.29 1.00 73 up 1773 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 324 MiB 2.0 GiB 66 GiB 85.24 1.00 73 up 1836 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 390 MiB 2.3 GiB 66 GiB 85.30 1.00 82 up 1771 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB 66 GiB 85.31 1.00 75 up 1834 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 376 MiB 2.1 GiB 66 GiB 85.27 1.00 79 up 2111 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 396 MiB 2.0 GiB 63 GiB 85.90 1.01 84 up 2112 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 338 MiB 2.2 GiB 66 GiB 85.27 1.00 77 up 2272 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 374 MiB 2.0 GiB 66 GiB 85.24 1.00 76 up 2279 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 332 MiB 2.5 GiB 66 GiB 85.34 1.00 77 up 2276 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 360 MiB 2.4 GiB 66 GiB 85.34 1.00 82 up 2285 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 371 MiB 2.2 GiB 66 GiB 85.27 1.00 79 up 2275 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 368 MiB 2.3 GiB 66 GiB 85.31 1.00 79 up 2282 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.0 GiB 66 GiB 85.23 1.00 75 up 2271 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 1.6 GiB 66 GiB 85.15 1.00 81 up 2278 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB 63 GiB 85.85 1.00 77 up 2277 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 334 MiB 1.3 GiB 64 GiB 85.70 1.00 72 up 2286 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 351 MiB 2.3 GiB 66 GiB 85.30 1.00 74 up 2347 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 362 MiB 2.6 GiB 65 GiB 85.38 1.00 81 up 2348 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.0 GiB 66 GiB 85.23 1.00 76 up 573 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 404 MiB 2.7 GiB 65 GiB 85.40 1.00 85 up 580 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 320 MiB 2.3 GiB 60 GiB 86.65 1.01 70 up 169 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.0 GiB 66 GiB 85.24 1.00 88 up 171 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 323 MiB 2.5 GiB 66 GiB 85.34 1.00 71 up 1216 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 364 MiB 2.5 GiB 66 GiB 85.35 1.00 76 up 1218 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 334 MiB 1.9 GiB 63 GiB 85.84 1.00 70 up 811 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 377 MiB 2.0 GiB 66 GiB 85.24 1.00 80 up 815 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 419 MiB 2.6 GiB 65 GiB 85.38 1.00 91 up 810 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.2 GiB 66 GiB 85.27 1.00 67 up 813 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 340 MiB 2.4 GiB 63 GiB 85.93 1.01 72 up 1760 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 388 MiB 974 MiB 67 GiB 85.01 0.99 77 up 1823 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 366 MiB 2.5 GiB 66 GiB 85.35 1.00 73 up 1762 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 361 MiB 1.4 GiB 67 GiB 85.10 1.00 75 up 1824 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 278 MiB 2.6 GiB 62 GiB 86.06 1.01 65 up 1766 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 377 MiB 1.9 GiB 60 GiB 86.58 1.01 82 up 1830 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 339 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1755 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 388 MiB 2.5 GiB 65 GiB 85.36 1.00 77 up 1818 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 2.3 GiB 66 GiB 85.31 1.00 75 up 1774 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 2.2 GiB 66 GiB 85.26 1.00 67 up 1837 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 311 MiB 2.5 GiB 66 GiB 85.33 1.00 68 up 1748 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 339 MiB 2.4 GiB 66 GiB 85.31 1.00 76 up 1812 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 336 MiB 2.2 GiB 66 GiB 85.28 1.00 77 up 1337 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 312 MiB 2.3 GiB 66 GiB 85.30 1.00 64 up 1338 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 331 MiB 1.9 GiB 63 GiB 85.89 1.00 76 up 1091 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 329 MiB 2.5 GiB 66 GiB 85.34 1.00 69 up 1096 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 373 MiB 1.6 GiB 63 GiB 85.85 1.00 82 up 1093 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 379 MiB 2.6 GiB 65 GiB 85.38 1.00 84 up 1098 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.3 GiB 66 GiB 85.30 1.00 79 up 1752 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 350 MiB 1.5 GiB 67 GiB 85.11 1.00 80 up 1815 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB 66 GiB 85.30 1.00 70 up 1754 ssd 0.43660 1.00000 447 GiB 387 GiB 384 GiB 362 MiB 2.0 GiB 60 GiB 86.48 1.01 75 up 1816 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 346 MiB 2.5 GiB 66 GiB 85.35 1.00 72 up 1753 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.6 GiB 64 GiB 85.80 1.00 75 up 1817 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB 66 GiB 85.33 1.00 75 up 1739 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.3 GiB 67 GiB 85.09 1.00 75 up 1804 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 365 MiB 2.6 GiB 62 GiB 86.03 1.01 78 up 1763 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 369 MiB 2.6 GiB 62 GiB 86.04 1.01 76 up 1825 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 379 MiB 2.2 GiB 63 GiB 85.91 1.01 83 up 1738 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 354 MiB 1.9 GiB 63 GiB 85.84 1.00 75 up 1802 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 339 MiB 1.6 GiB 63 GiB 85.83 1.00 77 up 1661 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 320 MiB 2.4 GiB 66 GiB 85.33 1.00 65 up 1666 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.3 GiB 66 GiB 85.31 1.00 80 up 1663 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 444 MiB 1.9 GiB 63 GiB 85.91 1.01 98 up 1669 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 375 MiB 2.2 GiB 66 GiB 85.29 1.00 76 up 1665 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 341 MiB 2.1 GiB 66 GiB 85.26 1.00 75 up 1671 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 363 MiB 2.4 GiB 66 GiB 85.33 1.00 76 up 1726 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 356 MiB 1.7 GiB 64 GiB 85.78 1.00 77 up 1790 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 349 MiB 2.3 GiB 66 GiB 85.29 1.00 80 up 1765 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 437 MiB 2.5 GiB 62 GiB 86.07 1.01 91 up 1828 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 356 MiB 2.3 GiB 66 GiB 85.31 1.00 78 up 1744 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 355 MiB 2.5 GiB 62 GiB 86.03 1.01 77 up 1807 ssd 0.43660 1.00000 447 GiB 387 GiB 385 GiB 327 MiB 1.9 GiB 60 GiB 86.51 1.01 74 up 1768 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 415 MiB 2.2 GiB 66 GiB 85.29 1.00 88 up 1831 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 332 MiB 2.0 GiB 66 GiB 85.23 1.00 71 up 1756 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 339 MiB 1.3 GiB 67 GiB 85.07 1.00 65 up 1819 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 323 MiB 2.3 GiB 66 GiB 85.28 1.00 67 up 1772 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 407 MiB 1.8 GiB 66 GiB 85.21 1.00 87 up 1835 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 340 MiB 1.5 GiB 67 GiB 85.11 1.00 74 up 1879 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 342 MiB 2.5 GiB 66 GiB 85.34 1.00 70 up 1882 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 310 MiB 2.2 GiB 66 GiB 85.28 1.00 66 up 1880 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 344 MiB 2.6 GiB 65 GiB 85.36 1.00 79 up 1883 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 319 MiB 2.6 GiB 65 GiB 85.36 1.00 65 up 1881 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 313 MiB 2.1 GiB 66 GiB 85.25 1.00 64 up 1884 ssd 0.43660 1.00000 447 GiB 390 GiB 388 GiB 387 MiB 2.1 GiB 57 GiB 87.27 1.02 87 up 1747 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 426 MiB 2.6 GiB 65 GiB 85.38 1.00 87 up 1809 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 359 MiB 1.6 GiB 66 GiB 85.14 1.00 77 up 1751 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 340 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1813 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 367 MiB 2.2 GiB 63 GiB 85.93 1.01 81 up 1785 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 403 MiB 2.5 GiB 66 GiB 85.35 1.00 81 up 1848 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.5 GiB 66 GiB 85.34 1.00 74 up 1749 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.5 GiB 67 GiB 85.12 1.00 74 up 1811 ssd 0.43660 1.00000 447 GiB 388 GiB 385 GiB 335 MiB 2.5 GiB 59 GiB 86.70 1.01 74 up 1725 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 361 MiB 2.5 GiB 65 GiB 85.35 1.00 73 up 1788 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB 65 GiB 85.36 1.00 74 up 1779 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 363 MiB 1.1 GiB 67 GiB 85.03 0.99 77 up 1843 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 387 MiB 2.4 GiB 66 GiB 85.34 1.00 78 up 1967 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.2 GiB 66 GiB 85.28 1.00 66 up 1970 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 382 MiB 1.9 GiB 66 GiB 85.21 1.00 78 up 1968 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 337 MiB 1.9 GiB 66 GiB 85.22 1.00 71 up 1971 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 354 MiB 2.1 GiB 66 GiB 85.26 1.00 79 up 1966 ssd 0.43660 1.00000 447 GiB 384 GiB 381 GiB 352 MiB 1.8 GiB 64 GiB 85.79 1.00 82 up 1969 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 315 MiB 2.3 GiB 63 GiB 85.99 1.01 73 up 1781 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 381 MiB 1.8 GiB 66 GiB 85.20 1.00 81 up 1844 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 330 MiB 2.1 GiB 66 GiB 85.25 1.00 71 up 1743 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 360 MiB 2.1 GiB 63 GiB 85.95 1.01 79 up 1806 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 385 MiB 2.4 GiB 66 GiB 85.34 1.00 78 up 1778 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 354 MiB 2.5 GiB 66 GiB 85.35 1.00 74 up 1839 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 348 MiB 1.6 GiB 66 GiB 85.15 1.00 79 up 1757 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 409 MiB 2.5 GiB 65 GiB 85.36 1.00 82 up 1820 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.1 GiB 66 GiB 85.26 1.00 73 up 1758 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 403 MiB 2.5 GiB 63 GiB 86.02 1.01 81 up 1821 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.1 GiB 66 GiB 85.27 1.00 80 up 1787 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 281 MiB 2.1 GiB 66 GiB 85.24 1.00 64 up 1850 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 260 MiB 2.6 GiB 65 GiB 85.36 1.00 59 up 2054 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 318 MiB 1.6 GiB 66 GiB 85.14 1.00 68 up 2057 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 364 MiB 2.3 GiB 66 GiB 85.31 1.00 78 up 2055 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 343 MiB 2.1 GiB 66 GiB 85.26 1.00 79 up 2058 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 289 MiB 2.5 GiB 63 GiB 85.99 1.01 65 up 2053 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 335 MiB 2.0 GiB 66 GiB 85.23 1.00 74 up 2056 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 362 MiB 2.3 GiB 63 GiB 85.93 1.01 77 up 1735 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 363 MiB 1.8 GiB 66 GiB 85.20 1.00 76 up 1799 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 360 MiB 2.1 GiB 66 GiB 85.27 1.00 74 up 1784 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 361 MiB 1.8 GiB 66 GiB 85.19 1.00 74 up 1847 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 370 MiB 2.4 GiB 66 GiB 85.33 1.00 74 up 1740 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 368 MiB 2.4 GiB 66 GiB 85.33 1.00 72 up 1801 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 362 MiB 2.1 GiB 66 GiB 85.26 1.00 74 up 1776 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 367 MiB 2.1 GiB 66 GiB 85.25 1.00 80 up 1841 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 393 MiB 2.1 GiB 66 GiB 85.26 1.00 86 up 1742 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 358 MiB 1.6 GiB 66 GiB 85.14 1.00 74 up 1805 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 330 MiB 2.6 GiB 62 GiB 86.05 1.01 75 up 1764 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.4 GiB 66 GiB 85.32 1.00 74 up 1827 ssd 0.43660 1.00000 447 GiB 385 GiB 382 GiB 434 MiB 2.3 GiB 63 GiB 86.00 1.01 88 up 2301 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 355 MiB 2.3 GiB 66 GiB 85.30 1.00 79 up 2309 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 315 MiB 1.7 GiB 66 GiB 85.17 1.00 69 up 2302 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 314 MiB 2.5 GiB 66 GiB 85.33 1.00 68 up 2315 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 333 MiB 2.3 GiB 66 GiB 85.30 1.00 70 up 2308 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 356 MiB 2.4 GiB 66 GiB 85.33 1.00 79 up 2316 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 366 MiB 2.3 GiB 66 GiB 85.29 1.00 76 up 1733 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 348 MiB 1.9 GiB 63 GiB 85.86 1.00 74 up 1796 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 352 MiB 2.6 GiB 65 GiB 85.38 1.00 74 up 1782 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 313 MiB 1.4 GiB 64 GiB 85.77 1.00 70 up 1845 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 379 MiB 2.2 GiB 66 GiB 85.29 1.00 81 up 1730 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 347 MiB 2.2 GiB 66 GiB 85.29 1.00 74 up 1794 ssd 0.43660 1.00000 447 GiB 384 GiB 382 GiB 377 MiB 2.1 GiB 63 GiB 85.94 1.01 80 up 1786 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 381 MiB 2.6 GiB 65 GiB 85.37 1.00 76 up 1849 ssd 0.43660 1.00000 447 GiB 383 GiB 382 GiB 337 MiB 963 MiB 64 GiB 85.63 1.00 76 up 1741 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 358 MiB 1.4 GiB 67 GiB 85.09 1.00 75 up 1803 ssd 0.43660 1.00000 447 GiB 382 GiB 379 GiB 383 MiB 2.7 GiB 65 GiB 85.40 1.00 78 up 1750 ssd 0.43660 1.00000 447 GiB 381 GiB 379 GiB 353 MiB 2.1 GiB 66 GiB 85.26 1.00 70 up 1814 ssd 0.43660 1.00000 447 GiB 380 GiB 379 GiB 351 MiB 772 MiB 67 GiB 84.95 0.99 81 up Both ssd pools run with basic 3x replication and rack failure domain. cephfs.arm-vol.data pool is EC pool on HDD OSDs. I've checked few OSDs and pools are not overlapping over. Cluster runs on 19.2.0 and OSD nodes are ARM64. In emergency I'm planning to just move these pool to HDDs OSD. Do you have any idea what could have filled these SSDs and how to clean it up? Today I've Best regards Adam Prycki

7 20

Network traffic with failure domain datacenter
by Torkil Svensgaard 10 May '25

10 May '25

Hi We are looking at a cluster split between two DCs with the DCs as failure domains. Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

7 13

Re: *** Spam *** Re: v18.2.7 Reef released
by Erwin Bogaard 09 May '25

09 May '25

Yeah, In think this is a difference between Centos 9 Stream and Almalinux 9 (which we use): openssl-libs-3.5.0 is only available for Centos 9 Stream, not for Almalinux. So we probably need to wait a little bit longer until this dependency is available at Almalinux we can install 18.2.7.

3 2

v18.2.7 Reef released
by Yuri Weinstein 09 May '25

09 May '25

We're happy to announce the seventh backport (hotfix) release in the Reef series. https://ceph.io/en/news/blog/2025/v18-2-7-reef-released/ Notable Changes This release fixes a critical bluestore regression in versions 18.2.5 and 18.2.6 (https://github.com/ceph/ceph/pull/61653) Users running either of those releases are recommended to upgrade at the earliest convenience. This release also includes several other important BlueStore fixes: * https://github.com/ceph/ceph/pull/62840 * https://github.com/ceph/ceph/pull/62054 * https://github.com/ceph/ceph/pull/62152 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad

3 2

check bucket index on large bucket leads to laggy index PGs
by Boris 09 May '25

09 May '25

Hi, I have a bucket that got >20m index entries but only got 11 shards. When I try to run a radosgw-admin bucket check the PGs that hold the index start to become laggy after a couple of seconds. I need to stop it because it kills the whole object storage. This is a latest reef cluster and the master of a multisite which only replicates the metadata (1 realm, multiple zonegroups, one zone per zonegroup). Any ideas what I can do? I fear to reshard the bucket, because I am not sure if I can stop the resharding if the PGs become laggy. Cheers Boris

1 0

Ceph reef ingress service - v4v6_flag undefined
by Steven Vacaroaia 08 May '25

08 May '25

Hi, I am unable to deploy ingress service because "v4v6_flag" is undefined I couldn't find any information about this flag The ingress.yaml file used is similar with this one Any help would be greatly appreciated Steven service_type: ingress service_id: rgw placement: hosts: - ceph-node-06.cephlab.com - ceph-node-07.cephlab.com spec: backend_service: rgw.client virtual_ip: 192.168.122.152/24 frontend_port: 80 monitor_port: 1967 use_tcp_mode_over_rgw: True

3 8

Request for Recommendations: Tiering/Archiving from NetApp to Ceph (with stub file support)
by sacawulu 07 May '25

07 May '25

Hi all, We're exploring solutions to offload large volumes of data (on the order of petabytes) from our NetApp all-flash storage to our more cost-effective, HDD-based Ceph storage cluster, based on criteria such as: last access time older than X years. Ideally, we would like to leave behind a 'stub' or placeholder file on the NetApp side to preserve the original directory structure and potentially enable some sort of transparent access or recall if needed. This kind of setup is commonly supported by solutions like DataCore/FileFly, but as far as we can tell, FileFly doesn’t support Ceph as a backend and instead favors its own Swarm object store. Has anyone here implemented a similar tiering/archive/migration solution involving NetApp and Ceph? We’re specifically looking for: * Enterprise-grade tooling * Stub file support or similar metadata-preserving offload * Support and reliability (given the scale, we can’t afford data loss or inconsistency) * Either commercial or well-supported open source solutions Any do’s/don’ts, war stories, or product recommendations would be greatly appreciated. We’re open to paying for software or services if it brings us the reliability and integration we need. Thanks in advance! MJ

5 5

Update of MDS (non-cephadm cluster)
by Massimo Sgaravatto 07 May '25

07 May '25

I have done the update of the ceph software several times, but this is the first time I have to do an update (from quincy to reef) considering also ceph-fs I am not using cephadm I have 3 MDSs (2 active and one in standby) I'm looking at point 5 of: https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/ If I have got it right It says to temporarily set max_mds to 1 and when you have only one active: 1 - turn off the MDS service on the 2 inactive MDS nodes 2- update and restart MDS on the active instance 3- restart (after updating the software: this is not written in the doc but it seems obvious to me) MDS on the other 2 instances 4 put max_mds back to 2 Did I understand correctly? But doesn't this mean to have a short period of time when the cluster is in error state (in 2, during the restart) and therefore with possible problems on the cephfs clients ? Thanks, Massimo

9 24

Need help with MDS stuck in readonly state
by Maxim Goncharov 07 May '25

07 May '25

Hi Everyone, I have a CephFS cluster that has MDS stuck in read only mode, and I do not know what can be done to return the cluster into the writable mode: health: HEALTH_WARN 1 MDSs are read only 1 MDSs behind on trimming How I got there: we have an FS where we store data in EC pool. We have one subdirectory that contains a lot of transitory data (constant write/read/delete). I wanted to put this subdirectory (let's call it /temp) into a separate replicated pool. I did: created replicated pool, added it to the FS, switched /temp to that replicated pool. All worked for about 24 hours and then became obvious it was not a good idea, so I reversed all steps: switched /temp to the original EC pool, removed replicated pool from FS, deleted the replicated pool. That is when MDS went into read only state. I can see inodes that were created in /temp when it belonged to the replicated pool. I tried many different things that I found in documentation, nothing works. What should I do in order for the FS become writable again? Thank you, --Max

4 10

CFS IO500 ISC'25 Deadline 2 June 2025
by IO500 Committee 07 May '25

07 May '25

Call for Submission Submission Deadline: June 2nd, 2025 AoE The IO500 is now accepting and encouraging submissions for the upcoming 16th semi-annual IO500 Production and Research lists, in conjunction with ISC25. We are also accepting submissions to both the Production and Research 10 Client Node Challenges to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Webpage. The new ranked lists will be announced at the BoF [1]. We look forward to seeing many new results! Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance and the documentation of how the numbers are achieved. Support small to extreme-scale Research and Production HPC systems using flexible storage APIs Maximizing simplicity in running the benchmark suite while offering tunable parameters Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes runs with highly prescribed parameters in an attempt to determine a lower performance bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported Storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. The goals of the community are also multi-fold: Gather historical data for the sake of analysis and to aid predictions of storage futures Collect tuning information to share valuable performance optimizations across the community Encourage vendors and designers to optimize for workloads beyond "hero runs" Establish bounded expectations for users, procurers, and administrators Understand and be able to reproduce performance on storage systems Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. Benchmark Changes This year, we have added a 4KiB random read test phase at the end of the benchmark suite. By default, this phase will also run, but will not be included in the score nor affect if the score is valid or not. If this causes problems, it is possible to turn this off in the configuration file. Entries without this result will not be penalized. We will have the traditional two-week stabilization period. Unless a serious flaw is discovered, results submitted during stabilization will be considered for this event. The detailed information about the new phase can be read here: 4KiB Random Read [3] 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather We encourage you to submit [2] to join our community, and to attend the ISC'25 BoF [1] on Tuesday, Nov 19, 2024 at 12:15pm - 1:15pm EST, where we will announce the new IO500 Production and Research lists and their 10 Client Node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. We will also highlight new and interesting results with invited talk(s) at the BoF. [1] https://io500.org/pages/bof-isc25 [2] https://io500.org/submission [3] https://docs.google.com/document/d/1zr88byIhNIhVXp6KN6v4zGB828oNs2LDZSbIj-3… -- The IO500 Committee

1 0

reef 18.2.7 hotfix QE validation status
by Yuri Weinstein 06 May '25

06 May '25

Details of this release are summarized here: https://tracker.ceph.com/issues/71166 Release Notes - https://github.com/ceph/ceph/pull/63090 LRC upgrade - N/A Seeking approvals/reviews for: smoke - same as in 18.2.5 rados - Laura, Mark N, Igor approved? upgrade/quincy-x (reef) - Laura approved? upgrade/reef-p2p - unexpected failure in `test_librbd_python.sh` Ilya PTL (it did pass in 18.2.5 https://tracker.ceph.com/issues/70563 and was not ran in 18.2.6)

3 3

BlueFS race condition between truncate() and unlink()
by Frédéric Nass 06 May '25

06 May '25

Hi everyone, I've recently become aware of this bug [1] that affects Reef and Squid. When the bug is triggered, OSD will fail to start with "bluefs mount failed to replay log: (5) Input/output error" after crashing with "bluestore/AvlAllocator.cc: 173: FAILED ceph_assert(rs->start <= start)". The Red Hat KB solution [2] recommends redeploying the OSD as soon as possible to avoid having to deal with multiple crashed OSDs in the cluster. Reef v18.2.7 will fix this issue via PR #62840 [3]. Squid v19.2.3 should fix this issue via PR #62839 [4]. Thought it might be worth sharing. Regards, Frédéric. [1] https://tracker.ceph.com/issues/70747 [2] https://access.redhat.com/solutions/7113657 [3] https://github.com/ceph/ceph/pull/62840 [4] https://github.com/ceph/ceph/pull/62839

1 0

Reef: Dashboard still freezing on RGW overview after upgrade to 18.2.6
by Eugen Block 06 May '25

06 May '25

Hi, I just updated these two trackers: https://tracker.ceph.com/issues/69755 https://tracker.ceph.com/issues/67009 But I'm not sure if that has any impact on closed trackers. I just want to raise awareness that the issue still persists. Shoud I create a new tracker or will one of those above be reopened? Thanks! Eugen

2 2

Grafana vulnerability - cephadm deployment
by Michael Worsham 06 May '25

06 May '25

With cephadm reef (v18.2.4), is there a way to upgrade the Grafana that is incorporated into the docker build? Currently it's showing Grafana v9.4.7 (quay.io/ceph/ceph-grafana:9.4.7) and that is actually an open vulnerability per CVE-2023-1387 (Grafana 9.1.0 < 9.2.17, 9.3.x < 9.3.13, 9.4.x < 9.4.9 Information Disclosure Vulnerability). Grafana shows that it's fixed with version 9.4.9 and above (https://grafana.com/security/security-advisories/cve-2023-1387/) -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

7 14

Heads up -- 18.2.6 regression
by Dan van der Ster 06 May '25

06 May '25

Hi all, Just a quick heads up -- 18.2.6 has a bluefs regression [1] which was not present in 18.2.4. Please avoid upgrading until further notice. Regards, Dan [1] Introduced in https://tracker.ceph.com/issues/65356 and fixed in https://tracker.ceph.com/issues/69764 -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO https://clyso.com | dan.vanderster(a)clyso.com

9 18

ceph NFS reef - cephadm reconfigure haproxy
by Steven Vacaroaia 05 May '25

05 May '25

Hi I am testing NFS trying to make sure that, deploying it as below will give me redundancy ceph nfs cluster create main "1 ceph-01,ceph-02" --port 2049 --ingress --virtual-ip 10.90.0.90 For what I have read so far ( and some of the posts on this list) the only way to get the NFS "surviving " a node reboot is to actually reconfigure haproxy since the config is missing the second entry under "backend" and the "check" The method to do that is as below using the template from the github link ceph config-key set mgr/cephadm/services/ingress/haproxy.cfg -i <custom-haproxy-config> ceph orch redeploy <ingress-service-name> ceph orch restart ingress-service-name https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/templates/ser… However, after doing the above , I was checking the actual haproxy.cfg on the main node (/var/lib/ceph/FSID/SERVICENAME/haproxy/haproxy.cfg ) and it does not change restart the NFS service (ceph orch restart servicename) did not help either What am I missing ?

1 0

Process to update images used by cephadm
by Joshua Blanch 05 May '25

05 May '25

Hello All, Is there a process to update images used by cephadm such as HAproxy? For example on main right now it is pointing to v2.3 tag https://github.com/ceph/ceph/blob/94876852b462c2ac15b447becf4d78071c529fb1/… and on quay, there is a more recent (albeit outdated) image version https://quay.io/repository/ceph/haproxy?tab=tags There are more recent releases for this image, is there a process to push a new image to quay and update which one cephadm pulls the default image. This question also goes for other images like grafana, etc. -- Joshua Blanch Clyso GmbH p: +49 89 215252722 | a: Victoria, Canada w: https://clyso.com | e: joshua.blanch(a)clyso.com

2 1

Notes from Ceph Steering Committee Weekly 2025-05-05
by Dan van der Ster 05 May '25

05 May '25

Hi all, Here are the summary notes from today's Ceph Steering Committee Weekly meeting: * CI/Testing: Push to reduce CI load by disabling flaky tests, parallelizing checks, and possibly allowlisting known infra issues; Will be coordinated further in the Infrastructure calls. * 18.2.7 Hotfix: Test summary sent, awaiting core team review and approval. Quincy will get a hotfix with minimal / critical PRs. Coordinating on Slack. * Major Point-Release Release Process: Clarified the process (<release> copied to <release>-release at the start, and that post-fork fix PRs need manual inclusion; proposals for better coordination via milestones and automation. * Telemetry Bot: Crash reports are not being reported in Redmine; the team is working on it: the collected reports will be ticketed soon. As usual the full notes are available at https://pad.ceph.com/p/csc-weekly-minutes -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO https://clyso.com | dan.vanderster(a)clyso.com

1 0

singular PG is snaptrimming for ever since pacific -> reef 18.2.6 upgrade
by Boris 05 May '25

05 May '25

Hi, we have one cluster where one PG that seems to be snaptrimming forever. It sometimes seem to hinder other PGs to on the same main OSD to snaptrim as well. We've synced out the OSD but it still keeps happening. It was gone sometime in between but now it is back again with the same PG. And the constant factor is always PG 3.c1c. Anyone got an idea what I can do? Is it possible to "sync out and recreate" a PG, like I can do with an OSD? Here is some output from the pg list Cheers Boris # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:12:21 PM UTC 2025 3.1b active+clean+snaptrim_wait 3m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 3m [156,117,32]p156 3.65d active+clean+snaptrim_wait 3m [156,263,9]p156 3.721 active+clean+snaptrim_wait 3m [156,23,190]p156 3.849 active+clean+snaptrim_wait 3m [156,135,31]p156 3.abb active+clean+snaptrim_wait 3m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 3m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 3m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 3m [156,266,281]p156 3.103f active+clean+snaptrim_wait 3m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 3m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 3m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 3m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 3m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 3m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 3m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 3m [156,212,134]p156 3.1e74 active+clean+snaptrim 3m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 3m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 3m [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 02:47:38 PM UTC 2025 3.1b active+clean+snaptrim_wait 2m [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 2m [156,117,32]p156 3.65d active+clean+snaptrim_wait 2m [156,263,9]p156 3.721 active+clean+snaptrim_wait 2m [156,23,190]p156 3.849 active+clean+snaptrim_wait 2m [156,135,31]p156 3.abb active+clean+snaptrim_wait 2m [156,70,262]p156 3.c1c active+clean+snaptrim_wait 2m [156,111,11]p156 3.f83 active+clean+snaptrim_wait 2m [156,119,97]p156 3.f8a active+clean+snaptrim_wait 2m [156,266,281]p156 3.103f active+clean+snaptrim_wait 2m [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 2m [156,135,280]p156 3.14ea active+clean+snaptrim_wait 2m [156,118,103]p156 3.1688 active+clean+snaptrim_wait 2m [156,221,98]p156 3.1848 active+clean+snaptrim_wait 2m [156,268,68]p156 3.18ca active+clean+snaptrim_wait 2m [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 2m [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 2m [156,212,134]p156 3.1e74 active+clean+snaptrim 2m [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 2m [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 2m [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Tue Apr 29 04:24:55 PM UTC 2025 3.1b active+clean+snaptrim_wait 55s [156,115,48]p156 3.2f6 active+clean+snaptrim_wait 55s [156,117,32]p156 3.65d active+clean+snaptrim_wait 55s [156,263,9]p156 3.721 active+clean+snaptrim_wait 55s [156,23,190]p156 3.849 active+clean+snaptrim_wait 55s [156,135,31]p156 3.abb active+clean+snaptrim_wait 55s [156,70,262]p156 3.c1c active+clean+snaptrim 34s [156,111,11]p156 3.f83 active+clean+snaptrim_wait 55s [156,119,97]p156 3.f8a active+clean+snaptrim_wait 55s [156,266,281]p156 3.103f active+clean+snaptrim_wait 55s [156,262,215]p156 3.11e9 active+clean+snaptrim_wait 55s [156,135,280]p156 3.14ea active+clean+snaptrim_wait 55s [156,118,103]p156 3.1688 active+clean+snaptrim_wait 55s [156,221,98]p156 3.1848 active+clean+snaptrim_wait 55s [156,268,68]p156 3.18ca active+clean+snaptrim_wait 55s [156,39,94]p156 3.1b6e active+clean+snaptrim_wait 55s [156,274,31]p156 3.1e27 active+clean+snaptrim_wait 55s [156,212,134]p156 3.1e74 active+clean+snaptrim_wait 34s [156,96,12]p156 3.1f11 active+clean+snaptrim_wait 55s [156,253,120]p156 3.1f7b active+clean+snaptrim_wait 55s [156,190,283]p156 # date; ceph pg ls| grep snaptrim | awk '{print $1, $11, $12, $15}' | column -t Mon May 5 06:32:53 AM UTC 2025 3.c1c active+clean+snaptrim 41h [165,111,25]p165 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 3

Ceph usage doubled after update
by Yunus Emre Sarıpınar 05 May '25

05 May '25

I have a ceph cluster created with nautilus and I upgraded to octopus later. I was have 24x node and I added 8x new nodes to the cluster. Balancer is upmap enabled. I increased my PG count from 8192 to 16384. I had to set reweight 0.8 on new OSD's to solve full usage (There was only 1TB left because of the faulted balance) Now cluster health is OK and distribution is balanced by my manual fix. data: pools: 4 pools, 16545 pgs objects: 173.32M objects, 82 TiB usage: 436 TiB used, 235 TiB / 671 TiB avail pgs: 15930 active+clean 579 active+clean+snaptrim_wait 36 active+clean+snaptrim Before adding 8 nodes to the cluster, the usage was 215 TiB used. I did the update about 2 months ago and my usage area still hasn't decreased. Why did the usage double, how can I solve it?

3 2

Excited to Support the Ceph User Community!
by Anthony Middleton 03 May '25

03 May '25

Hello Ceph Users, I'm Anthony Middleton, and I'm excited to introduce myself as the new Ceph Community Manager. I'm based in Chicago, IL, and bring seven years of experience supporting users across tech communities, including Confluent and Hedera Hashgraph, focusing on continually improving user experience and engagement. My goal is to ensure that every Ceph user feels heard, supported, and connected. I'll be working closely with you to gather feedback, strengthen support systems, and build programs that make the community even stronger. Outside of work, I'm active in the Chicago Gay Men's Chorus and the Chicago Dragons Rugby Club — two amazing communities that remind me every day of the value of connection and teamwork. Let's connect — I'd love to hear your ideas and experiences. Connect with me on LinkedIn <https://www.linkedin.com/in/middleton-anthony/>and Bluesky <https://bsky.app/profile/anthonymicmidd.bsky.social>! Looking forward to growing with you, Best, *Anthony Middleton*

2 1

Adding crush rules and pool crush rules to prometheus
by Boris 02 May '25

02 May '25

Hi, is it possible to have the crush rules, the one you get from ceph osd crush rule dump, and which pool uses which crush rule into the prometheus exporter? It would help us to monitor this. Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Optimal number of placement groups per OSD
by gagan tiwari 01 May '25

01 May '25

Hi Guys, I have 7 OSD nodes with 10X15T NVME disk on each OSD node. To start with , I want to use only 8X15T disk on each osd node and keep 2X15 disk spare in case of any disk failure and recovery event. I am going to use the 4X2 EC CephFS data pool to store data. So, with the above set-up, what will be the optimal number of placement groups per OSD. As per the PG calculator :- ( 8X7X100 ) / 6 = 933.33 nearest power of 2 is 1024. With 1024 placement groups distributed across 56 OSDs, that evaluates to approximately 18 placement groups per OSD. I don't think its optimal as Ceph doc recommends 50-100 PG per OSD. So, am I doing something wrong? or missing something white calculating the number of PG per OSD. Also, will it be best practice to keep 2X15T disk spare on each OSD or should I use all of them. Also, I am going to deploy 7 OSD nodes across 4 Racks and will be using the failure domain as "rack" to have Ceph handle the entire rack failure. I hope this will provide more protection. Please advise. Thanks, Gagan Thanks, Gagan.

2 4

Doubled numbers of PGs from 8192 to 16384 - backfill bottlenecked
by Torkil Svensgaard 01 May '25

01 May '25

Hi 19.2.2 with WPQ, osd-max-backfills and osd-recovery-max-active are set to 1. In order to get our PG sizes better aligned we doubled the number of PGs on the pool with the largest PG size. The pool is HDD with DB/WAL on SATA SSD and HDD sizes between 2TB and 20TB and PG size was ~140GB before the doubling. All went well but looking at this: " cluster: id: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-flash1,ceph-flash2,ceph-flash3,ceph-e3s2,ceph-e3s3 (age 2d) mgr: ceph-flash2.utlhuz(active, since 4d), standbys: ceph-flash3.ciudre, ceph-flash1.erhakb mds: 1/1 daemons up, 2 standby osd: 576 osds: 576 up (since 2h), 576 in (since 3d); 8767 remapped pgs data: volumes: 1/1 healthy pools: 18 pools, 25249 pgs objects: 683.85M objects, 1.6 PiB usage: 2.7 PiB used, 1.9 PiB / 4.5 PiB avail pgs: 842769842/3951610673 objects misplaced (21.327%) 16481 active+clean 8762 active+remapped+backfill_wait 6 active+remapped+backfilling io: client: 374 MiB/s rd, 14 MiB/s wr, 2.86k op/s rd, 410 op/s wr recovery: 153 MiB/s, 38 objects/s " The balancer was running and seemingly making very small changes: " [root@lazy ~]# ceph balancer status { "active": true, "last_optimize_duration": "0:00:01.012679", "last_optimize_started": "Mon Apr 28 10:01:24 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Optimization plan created successfully", "plans": [] } " I tried the pgremapper and that gave a nice throughput in the beginning but now it seems to be about back to scratch: " cluster: id: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-flash1,ceph-flash2,ceph-flash3,ceph-e3s2,ceph-e3s3 (age 7h) mgr: ceph-flash2.utlhuz(active, since 5d), standbys: ceph-flash3.ciudre, ceph-flash1.erhakb mds: 1/1 daemons up, 2 standby osd: 577 osds: 577 up (since 5h), 577 in (since 31h); 7392 remapped pgs data: volumes: 1/1 healthy pools: 18 pools, 25249 pgs objects: 683.96M objects, 1.6 PiB usage: 2.7 PiB used, 1.9 PiB / 4.5 PiB avail pgs: 459889823/3952312686 objects misplaced (11.636%) 17857 active+clean 7384 active+remapped+backfill_wait 8 active+remapped+backfilling io: client: 70 MiB/s rd, 15 MiB/s wr, 689 op/s rd, 378 op/s wr recovery: 294 MiB/s, 75 objects/s " Balancer is still running and making small changes. This is going to take a while, any tips on how to escape the apparent bottleneck? Is having many PGs misplaced actually counter productive, so it would be better to throttle via target_max_misplaced_ratio? I was thinking it was better to let the balancer balance all it could, as that would make all the moves available and decrease the risk of bottlenecking. Thanks. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

6 13