January 2025 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

constant increase in osdmap epoch
by Frank Schilder 15 Apr '25

15 Apr '25

Hi all, we observe a problem that has been reported before, but I can't find a resolution. This is related to an earlier thread "failed to load OSD map for epoch 2898146, got 0 bytes" (https://www.spinics.net/lists/ceph-users/msg84485.html) We run an octopus latest cluster and observe a constant increase in osdmap epoch every few seconds. There is no change in the contents between two successive epochs: # diff map.3075085.txt map.3075086.txt 1c1 < epoch 3075085 --- > epoch 3075086 4c4 < modified 2024-11-18T15:38:45.512100+0100 --- > modified 2024-11-18T15:38:47.858092+0100 This is exactly what others reported too, for example, "steady increasing of osd map epoch since octopus" (https://www.spinics.net/lists/ceph-users/msg69443.html) Its a real problem since it dramatically shortens the time window an OSD can be down before its latest OSD map is purged from the cluster. This, in turn, leads to serious follow-up problems with OSD restart as reported in the thread I'm referring to at the beginning. Related to that I also see the mgrs increasing the pgmap version constantly every 2 seconds. However, I believe this is intentional. I don't see this redundant pgp_num_actual setting by the mgrs reported here: https://tracker.ceph.com/issues/51433 . I can't find a resolution anywhere. Any help would be very much appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 7

Ceph Day Silicon Valley 2025 - Call for Proposals Now Open!
by Neha Ojha 14 Apr '25

14 Apr '25

Dear Ceph Community, We are very excited to announce that Ceph Day is returning to Silicon Valley after seven years! On March 25, 2025, we will be hosting Ceph Day at IBM Almaden Research Center in San Jose, CA. We are now accepting talk proposals! For more details, visit https://ceph.io/en/community/events/2025/ceph-days-almaden/. We look forward to bringing the Ceph community together once again! Cheers, Neha

2 2

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

RGW multisite metadata sync issue
by Vahideh Alinouri 11 Mar '25

11 Mar '25

Hi guys, My Ceph release is Quincy 17.2.5. I need to change the master zone to decommission the old one and upgrade all zones. I have separated the client traffic and sync traffic in RGWs, meaning there are separate RGW daemons handling the sync process. I encountered an issue when trying to sync one of the zones in the zonegroup. The data sync is proceeding fine, but I have an issue with the metadata sync. It gets stuck behind on a shard. Here is the output from radosgw-admin sync status: metadata sync syncing full sync: 1/64 shards full sync: 135 entries to sync incremental sync: 63/64 shards metadata is behind on 1 shard behind shards: [0] In the RGW log, I see this error: 2024-12-15T21:30:59.641+0000 7f6dff472700 1 beast: 0x7f6d2f1cf730: 172.19.66.112 - s3-cdn-user [15/Dec/2024:21:30:59.641 +0000] "GET /admin/log/?type=data&id=56&marker=00000000000000000000%3A00000000000000204086&extra-info=true&rgwx-zonegroup=7c01d60f-88c6-4192-baf7-d725260bf05d HTTP/1.1" 200 44 - - - latency=0.000000000s 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 I’ve tried the following steps: - Changed the PG number of the metadata pool to force a rebalance, but everything was fine. - Ran metadata sync init and tried to run it again. - Restarted RGW services in both the zone and the master zone. - Created a user in the master zone to ensure metadata sync works, which was successful. - Checked OSD logs but didn’t see any specific errors. - Attempted to list metadata in the pool using rados ls -p s3-cdn-dc07.rgw.meta, but got an empty result. - Compared the code for listing OMAP keys between Quincy and Squid versions; there were no specific changes. I’m looking for any advice or suggestions to resolve this issue.

3 6

Move block.db to new ssd
by Roland Giesler 11 Mar '25

11 Mar '25

I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions. The new 4TB SSD's I have split into volumes with: # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 & osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this: # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami I then take osd4 out of the cluster and stop it. Then I do this: # ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami Thereafter I do this to load the changes: # ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 --path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami After that the configuration change has reverted to what it was before. I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 What am I missing to get these changes to be permanent? regards Roland

6 13

When 18.2.5 will be released?
by wojiaowugen 04 Mar '25

04 Mar '25

Hi, everyone, It's an honor to ask questions here. can I ask when 18.2.5 will be released?

2 1

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

512e -> 4Kn hdd
by Marc 12 Feb '25

12 Feb '25

I used to have a tool from hgst/wd to 'convert' drives to 4Kn, but after a mentally challenging interaction with some wd support, I decided to try and use other solutions mentioned on the internet this seems to produce output similar to the previously used tool sg_format --format --size=4096 /dev/sdd Anyone have experience with this? This is first time the drives are already in the nodes, so I don't have a power cycle on them.

3 3

RGW - S3 bucket browser and/or S3 explorer
by notna＠ist-einmalig.de 12 Feb '25

12 Feb '25

Hi all. Is anyone aware of a ("S3 bucket browser" | "S3 explorer"), which works with RadosGW static websites? I've played around with those below, but none of them works (in my environment): - https://github.com/qoomon/aws-s3-bucket-browser - https://github.com/guimou/ceph-js-s3-explorer - https://github.com/awslabs/aws-js-s3-explorer - https://github.com/awslabs/aws-js-s3-explorer/tree/v2-alpha Any hints are highly appreciated!

4 9

Measuring write latency (ceph osd perf)
by Jan Kasprzak 07 Feb '25

07 Feb '25

Hello, ceph users, TL;DR: how can I look into ceph cluster write latency issues? Details: we have a HDD-based cluster (with NVMe for metadata), about 20 hosts, 2 OSD per host, mostly used as RBD storage for QEMU/KVM virtual machines. From time to time our users complain about write latencies inside their VMs. I would like to be able to see when the cluster is overloaded or when the write latency is bad. What did I try so far: 1) fio inside the KVM virtual machine: fio --ioengine=libaio --direct=1 --rw=write --numjobs=1 --bs=1M --iodepth=16 --size=5G --name=/var/tmp/fio-test [...] write: IOPS=63, BW=63.3MiB/s (66.4MB/s)(5120MiB/80863msec); 0 zone resets - I am usually getting about 60 to 150 IOPS for 1MB writes 2) PostgreSQL from the KVM virtual machine, running many tiny INSERTs as separate transactions for about 10 seconds. This is where I clearly see latency spikes: Wed Dec 18 09:20:21 PM CET 2024 406.062 txn/s Wed Dec 18 09:25:21 PM CET 2024 318.974 txn/s Wed Dec 18 09:30:21 PM CET 2024 285.591 txn/s Wed Dec 18 09:35:21 PM CET 2024 191.804 txn/s Wed Dec 18 09:40:22 PM CET 2024 246.679 txn/s Wed Dec 18 09:45:22 PM CET 2024 201.005 txn/s Wed Dec 18 09:50:22 PM CET 2024 153.206 txn/s Wed Dec 18 09:55:22 PM CET 2024 124.546 txn/s Wed Dec 18 10:00:23 PM CET 2024 33.094 txn/s Wed Dec 18 10:05:23 PM CET 2024 82.659 txn/s Wed Dec 18 10:10:23 PM CET 2024 292.544 txn/s Wed Dec 18 10:15:24 PM CET 2024 453.366 txn/s The drawback of both fio and postgresql benchmark is that I am unnecessarily loading the cluster with additional work, just to measure latency. And I am not covering the whole cluster, just the OSDs on which that VM happens to have its own data. 3) ceph osd perf I don't see any single obviously overloaded OSD here, but the latencies vary nevertheless. Here are statistics computed across all OSDs from the "ceph osd perf" output: Fri Jan 3 10:12:41 AM CET 2025 average 9 9 median 5 5 3rd-q 13 13 max 70 70 Fri Jan 3 10:13:42 AM CET 2025 average 5 5 median 3 3 3rd-q 10 10 max 31 31 Fri Jan 3 10:14:42 AM CET 2025 average 3 3 median 2 2 3rd-q 4 4 max 19 19 Fri Jan 3 10:15:42 AM CET 2025 average 5 5 median 1 1 3rd-q 3 3 max 63 63 However, I am not sure what exactly these numbers actually mean - what timespan do they cover? I would like to have something like "in the last 5 minutes, 99 % of all writes committed under XXX ms". Can Ceph tell me that? What else apart from buying faster hardware can I try in order to improve the write latency for QEMU/KVM-based VMs with RBD images? Thanks for any hints. -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

CephFS subdatapool in practice?
by Otto Richter (Codeberg e.V.) 05 Feb '25

05 Feb '25

Hello everyone, I am maintaining a cost-optimized Ceph cluster at Codeberg.org, that is serving a lot of small random IO (Git operations). Currently, the data is stored in a single CephFS pool on mixed SSD and HDD cluster, where the HDDs have a primary affinity of 0 to shift the reads to the SSDs. In a quest to further optimize our setup, I discovered the subdatapool feature described at https://github.com/TheJJ/ceph-cheatsheet#subdatapool (assigning multiple pools to a filesystem and using setfattr to allow writes in a certain folder to a separate pool). It feels like this would allow us to cut cost by moving certain directories with large files to a cheaper pool than that used for Git operations easily, without doing data migrations such as splitting data into multiple filesystems. Is anyone using this feature in production and can comment on it? Are there best practices regarding the usage of this feature? Any caveats to be aware of, such as overhead, latency, long-term maintainability? Since certain "interesting" features have been deprecated in the past, such as inline_data, I thought it would be better to ask here for feedback first. Thank you for the comments and have a nice day! Kind Regards Otto -- https://codeberg.org Codeberg e.V. – Arminiusstraße 2-4 – 10551 Berlin – Germany Registered at registration court Amtsgericht Charlottenburg VR36929.

3 2

Orphaned rbd_data Objects
by Stolte, Felix 05 Feb '25

05 Feb '25

Hi guys, we have a rbd pool we used for images exported via ceph-iscsi on a 17.2.7 cluster. The pool uses 10 times the diskspace i would suppose it should and after investigating we noticed a lot of rbd_data Objects which images are no longer present. I assume that the original images were deleted using the gwcli but not all Objects have been removed properly. What would be the best/most secure way to get rid of these orphaned objects and reclaim the diskspace? Best regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

4 11

ceph-osd segmentation fault on arm64 quay.io/ceph/ceph:v18.2.4
by Adam Prycki 05 Feb '25

05 Feb '25

Hi, today we attempted to upgrade our ceph cluster from 18.2.0 to 18.2.4. OSDs updated to 18.2.4 couldn't start. I've discovered that ceph-osd in segfaults. It happens always on our setup. I've tested it on ubunut 20.04 with docker and ubuntu 22.04 with podman Example: root@ktsh02:~# podman run --rm -it quay.io/ceph/ceph:v18.2.4 /bin/bash [root@20af860446bd /]# ceph-osd Segmentation fault (core dumped) 19.2.0 container seems to be fine. Do we have other users with arm64 ceph deployments? Do 18.2.4 containers work on your machines? Best regards Adam Prycki

2 1

Squid: RGW overview freezes (this.dataArray[W] is undefined)
by Eugen Block 04 Feb '25

04 Feb '25

Hi, I couldn't find an existing tracker yet, so I'm asking here if this already has been reported (or even fixed). Until now, I've only been playing around with virtual test clusters on Squid, and I noticed the dashboard freezes when I'm on the /rgw/overview page. This is what I get in the mgr log: ---snip--- Jan 31 13:16:37 host2 ceph-mgr[2799]: [dashboard ERROR frontend.error] (https://IP:8443/#/rgw/overview) this.dataArray[W] is undefined updateChartData@https://IP:8443/main.5d76eec705c379ed.js:3:1175137 ngOnChanges@https://IP:8443/main.5d76eec705c379ed.js:3:1174390 ... 99140/ye/v.prototype.invoke@https://IP:8443/ ---snip--- The output is lengthy (around 100 lines), just wanted to give an impression. Clicking anywhere else doesn't really improve anything, only reloading the page lets you browse again. This seems to be related to "Multi-Site Sync Status" (which I don't have enabled), it seems to freeze while loading. But sometimes it does load successfully, showing the info ( Multi-site needs to be configured in order to see the multi-site sync status. Please consult the documentation on how to configure and enable the multi-site functionality.). But I thought, maybe that's my lab environment or something. Last week I upgraded a customer cluster to 19.2.0 as well, and it shows the same behavior. Should I create a tracker issue? Thanks! Eugen

3 3

[ceph-ansible][radosgw]: s3 key regenration issue
by Danish Khan 04 Feb '25

04 Feb '25

Dear All, I am getting 500 internal server error after deleting and creating new access and secret keys. I am able to access my buckets and and its data using new keys from S3cmd utility. but it seems ceph MGR keep cache and throwing 500 internal server error I am trying to get meta data from CEPH dashboard. But when I do ceph mgr fail and move the ceph dashboard to another MGR then it starts working fine again. Is there a way I can clear cache of MGR or is the any bug in ceph for regeneration of S3 access keys? Regards, Danish

2 4

CephFS: EC pool with "leftover" objects
by Robert Sander 03 Feb '25

03 Feb '25

Hi, there is an old cluster (9 years) that gets constantly upgraded and is currently running version 17.2.7. 3 years ago (when running version 16) a new EC pool was added to the existing CephFS to be used with the directory layout feature. Now it was decided to remove that pool again. On the filesystem level all files have been copied to the original replicated pool and then deleted. No files or directories have the EC pool in their extended attributes referenced. But still this EC pool has appr 242 million objects with a total of ca 600 TB data stored. This shows up in "ceph df" and "ceph pg dump". The objects can be listed with "rados ls" but a "rados stat" or "rados get" will yield an error: error stat-ing cephfs_data_ec/1001275b3fe.00000241: (2) No such file or directory How can this be? Are these artifacts from not properly removed snapshots? Is it really save to remove this pool from CephFS and delete it? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

4 6

cephadm: Move DB/WAL from HDD to SSD
by Alan Murrell 03 Feb '25

03 Feb '25

Hello, I posted this on the Reddit sub, but it was suggested I might get better responses here. I am running a 5-node Ceph cluster (v18.2.2) installed using "cephadm". I am trying to migrate the DB/WAL on our slower HDDs to NVME; I am following this article: https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-fo… I have a 1TB NVME in each node, and there are four HDDs. I have created the VG ("cephdbX", where "X" is the node number) and four equal-sized LVs ("cephdb1", "cephdb2", "cephdb3", "cephdb4"). On the node I am trying to move the DB/WAL first, I have stopped the systemd OSD service for the OSD I am doing this first to. I have switched into the cephadm shell so I can run the ceph-volume commands, but when I run: ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/cephdb1 I get the following error: --> Target path cephdb03/cephdb1 is not a Logical Volume Unable to attach new volume : cephdb03/cephdb1 If I run 'lvs' in the cephadm shell, I can see the LVs (sorry about the formatting; hopefully you can view it in a way that it isn't a big "jumble"): LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-f85a57a8-e2f5-4bda-bc3b-e99d8b70768b ceph-341561e6-da91-4678-b6c8-0f0281443945 -wi-ao---- <1.75t osd-block-f1fd3d53-4ed9-4492-82a0-4686231d57e1 ceph-65ebde73-28ac-4dac-b0cb-4cf8df18bd4b -wi-ao---- 16.37t osd-block-3571394c-3afa-4177-904a-17550f8e902c ceph-6c8de2ed-cae3-4dd9-9ea8-49c94b746878 -wi-a----- 16.37t osd-block-41d44327-3df7-4166-a675-d9630bde4867 ceph-703962c7-6f28-4d8b-b77f-a6eba39da6b2 -wi-ao---- <1.75t osd-block-438c7681-ee6b-4d29-91f5-d487377c3ac9 ceph-71cc35c4-436d-42b7-a704-b21c2d22b43b -wi-ao---- 16.37t osd-block-2ebf78e8-1de1-464e-9125-14a8b7e6796f ceph-7c1fe149-8500-4a41-9052-64f27b2cb70b -wi-ao---- <1.75t osd-block-ca347144-eb84-4e9f-bfb5-81d60659f417 ceph-92595dfe-dc70-47c7-bcab-65b26d84448c -wi-ao---- 16.37t osd-block-2d338a42-83ce-4281-9762-b268e74f83b3 ceph-e9b51fa2-2be1-40f3-b96d-fb0844740afa -wi-ao---- <1.75t cephdb1 cephdb03 -wi-a----- 232.00g cephdb2 cephdb03 -wi-a----- 232.00g cephdb3 cephdb03 -wi-a----- 232.00g cephdb4 cephdb03 -wi-a----- 232.00g lv_root cephnode03-20240110 -wi-ao---- <468.36g lv_swap cephnode03-20240110 -wi-ao---- <7.63g All the official docs I read about it seem to assume the Ceph components are installed directly on the host, rather than in containers (which is what 'cephadm' does) Any advice for migrating the DB/WAL to the SSDs when using 'cephadm'? (I could probably destroy the OSD and manually re-create it with the options for pointing the DB/WAL to the SSD, but I would rather do it without forcing a data migration, otherwise I would have to wait for that with each OSD I am migrating) Followup question: if I do seem to have to go the route of destroying the OSD and re-creating it with the flags for point the DB/WAL to the SSD partition, the syntax I have seen is this: ceph-volume lvm prepare --bluestore --block.db --block.wal --data VOLUME_GROUP/LOGICAL_VOLUME but the above command seems to only specify the location for the data? Should the syntax be more like: ceph-volume lvm prepare --bluestore --block.db VG2/LV1 --block.wal VG2/LV1 --data /dev/sda (assuming my HDD is /dev/sda)? Thanks! :-)

3 2

SMB Support in Squid
by Tarrago, Eli (RIS-BCT) 31 Jan '25

31 Jan '25

Hello Ceph-users, I may be in the process of upgrading to Squid to add SMB support to our ceph clusters. Has anyone used this feature in a production environment and what has been your experience? Thank you. ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

2 1

RGW Exporter for Storage Class Metrics
by Preisler, Patrick 31 Jan '25

31 Jan '25

Hi, we would like to have a detailed usage report for our S3 Buckets. I already installed this rgw exporter https://github.com/blemmenes/radosgw_usage_exporter and it does provide useful information about the buckets and the amount of objects etc. However we have configured different storage-classes in our placement-targets and we would like to know how much data inside each bucket is placed in the STANDARD or STANDARD_IA Storage Class. Is there an exporter that does provide this data? Thanks and kind regards Patrick Preisler

2 1

RGW S3 Compatibility
by Anthony Fecarotta 31 Jan '25

31 Jan '25

Hello, Our company has a five node Proxmox cluster that recently implemented Ceph. We utilize multiple open-source applications that prefer (not require) S3 storage. Now that we are running Ceph, are we able to simply create a Ceph Object Gateway with the S3 API? I have read that not all S3 buckets share the same compatibility. Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

4 6

slow backfilling and recovering
by Jaemin Joo 31 Jan '25

31 Jan '25

Hi, all I am suffering from slow backfilling and recovering. I increased to "max_backfills" = 64, "recovery_max_actives" = 64. I know it's unnessary to increase more in my ceph cluster. When I checked pg query, I found the number of "backfills_in_flight"&"recovering" is 4. I set "osd_recovery_max_single_start = 4" so I think "osd_recovery_max_single_start" affects to the number of "backfills_in_flight"&"recovering". I increased to "osd_recovery_max_single_start = 16" but the number of "backfills_in_flight"&"recovering" in pg query was not changed. I know "osd_recovery_max_single_start" runtime is true. Should I restart the osds to increase the number of "backfills_in_flight"&"recovering"? or Is there the other way?

4 5

pgmap version increasing like every second ok or excessive?
by Andreas Elvers 31 Jan '25

31 Jan '25

Hello, I see this in my logs: 2025-01-22T09:14:43.063966+0000 mgr.node1.joznex (mgr.584732) 337151 : cluster [DBG] pgmap v300985: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:45.066685+0000 mgr.node1.joznex (mgr.584732) 337154 : cluster [DBG] pgmap v300986: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:47.070458+0000 mgr.node1.joznex (mgr.584732) 337155 : cluster [DBG] pgmap v300987: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:49.074664+0000 mgr.node1.joznex (mgr.584732) 337158 : cluster [DBG] pgmap v300988: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:51.079225+0000 mgr.node1.joznex (mgr.584732) 337159 : cluster [DBG] pgmap v300989: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:53.081633+0000 mgr.node1.joznex (mgr.584732) 337160 : cluster [DBG] pgmap v300990: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:55.084216+0000 mgr.node1.joznex (mgr.584732) 337163 : cluster [DBG] pgmap v300991: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:57.087873+0000 mgr.node1.joznex (mgr.584732) 337164 : cluster [DBG] pgmap v300992: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:59.092225+0000 mgr.node1.joznex (mgr.584732) 337175 : cluster [DBG] pgmap v300993: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:15:01.096803+0000 mgr.node1.joznex (mgr.584732) 337176 : cluster [DBG] pgmap v300994: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail Cluster is healthy. 3 nodes with 2 HDD OSDs each. pgmap version is increasing like every second. Is this excessive or is it deemed normal?

2 1

Squid Grafana Certificates
by Frank Frampton 31 Jan '25

31 Jan '25

Nothing I try will successfully deploy a Grafana daemon with the certificates I choose. The only way I can get it to use my certs is by manually over writing the certs placed on the host node in the binds section of the container and restarting the existing daemons. Replace file contents here '/var/lib/ceph/05aa38e6-8944-11ee-a398-f8f21e2ffe70/grafana.cephc01gw04/etc/grafana/certs/cert_file|cert_key' will result in Grafana using the desired certs. Using ceph config-key commands will add cert|key, however deployed containers do not use the cert|key defined. This process has worked from v16-v18, but now nothing, has something changed in v19? There are no errors stating that the certificates are invalid or that the json is invalid. Scanned By Microsoft EOP

2 1

Re: Grafana certificates storage and Squid
by Thorsten Fuchs 30 Jan '25

30 Jan '25

In the meantime we did update the `mgr/cephadm/cert_store.cert.grafana_cert` and `mgr/cephadm/cert_store.key.grafana_key` variables by hand and that worked out for us. I also tested the python script and it worked well. Makes the creation and management a much nicer experience. -- Thorsten Fuchs abaut GmbH Agnes-Pockels-Bogen 1, 80992 München

1 0

OSD latency
by Devender Singh 29 Jan '25

29 Jan '25

Hello Friends Seeking some help on OSD write latency in "osd performance graph" whereas "osd graph on dashboard" is not showing any high latency. So, is dashboard graph showing only read latency as I can see low latency in osd perf graph which matches the dashboard osd perf graph? OSD per Graph… Dashboard OSD perf Graph CEPH health… # ceph -s cluster: id: 384d7590-d018-11ee-b74c-5b2acfe0b35c health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 (age 17h) mgr: node01.cvknae(active, since 18h), standbys: node2.foomun mds: 2/2 daemons up, 2 standby osd: 119 osds: 119 up (since 17h), 119 in (since 17h) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 2/2 healthy pools: 18 pools, 817 pgs objects: 6.02M objects, 19 TiB usage: 33 TiB used, 383 TiB / 416 TiB avail pgs: 817 active+clean io: client: 129 KiB/s rd, 19 MiB/s wr, 42 op/s rd, 826 op/s wr Regards Dev

2 2

Cephfs mds not trimming after cluster outage
by Adam Prycki 29 Jan '25

29 Jan '25

Hello, we are having issues with cephfs cluster. Any help would be appreciated. We are running still on 18.2.0. During holidays we had outage caused by filling up rootfs. OSDs started randomly dying and we had time when not all PGs were active. This issue is already solved and all OSDs work fine but we're stuck with some MDS issues. warnings we are concerned about: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.arm-vol.k02r04nvm01.zaqebs(mds.0): 29 slow metadata IOs are blocked > 30 secs, oldest blocked for 1899 secs [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.arm-vol.k02r04nvm01.zaqebs(mds.0): Behind on trimming (4851/128) max_segments: 128, num_segments: 4851 1. Out MDSs are not trimming. 2. our active MDS has metadata slow ops which we cannot understand Cephfs status look ok, main MDS is active. All metadata pool PGs are active and working, there are not laggy PGs. Trying to dump ops from mds also doesn't help ceph daemon ./ceph-mds.arm-vol.k02r04nvm01.zaqebs.asok dump_ops_in_flight { "ops": [], "num_ops": 0 } MDS failover or MDS restart also doesn't help. Metadata slow ops always return after MDS restart. (all MDSs have this issue) After failover main MDS is stuck in rejoin state for a long time. We've used mds_wipe_sessions config option to bring it quickly into active state. I'm guessing slow metadata ops are stopping MDS from trimming but we cannot figure out what is causing these slow ops. Best regards Adam Prycki

2 1

ceph-14.2.22 OSD crashing - PrimaryLogPG::hit_set_trim on unfound object
by Nikola Ciprich 29 Jan '25

29 Jan '25

Hello dear ceph users and developers, today, we've hit issue on one of our legacy clusters running 14.2.22. we've apparently lost two objects containing cache tier hit_set history. when hit_set_trim hits missing object, it causes OSD to crash: ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x141) [0x55714332faad] 2: (()+0x4e8ca8) [0x55714332fca8] 3: (PrimaryLogPG::hit_set_trim(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >&, unsigned int)+0xcd6) [0x557143626136] 4: (PrimaryLogPG::hit_set_remove_all()+0x2d5) [0x5571436264b5] 5: (PrimaryLogPG::on_activate()+0x55a) [0x5571436360aa] 6: (PG::RecoveryState::Active::react(PG::AllReplicasActivated const&)+0x130) [0x557143512b50] 7: (boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::react_impl(boost::statechart:: 8: (boost::statechart::simple_state<PG::RecoveryState::Activating, PG::RecoveryState::Active, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na 9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x15d) [0x55714353c7ed] 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x211) [0x5571434555d1] 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x52) [0x557143723352] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x582) [0x557143443122] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3eb) [0x557143ac2f4b] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x557143ac5d30] 15: (()+0x7ea5) [0x7f22383e5ea5] 16: (clone()+0x6d) [0x7f22372a8b0d] I've found old report of similar problem here: https://www.spinics.net/lists/ceph-users/msg48660.html however using the same patch: diff -Naur ceph-14.2.22/src/osd/PrimaryLogPG.cc ceph-14.2.22-fix-hit_set_trim/src/osd/PrimaryLogPG.cc --- ceph-14.2.22/src/osd/PrimaryLogPG.cc 2024-12-29 12:34:17.000000000 +0100 +++ ceph-14.2.22-fix-hit_set_trim/src/osd/PrimaryLogPG.cc 2024-12-29 19:42:31.527632776 +0100 @@ -13932,11 +13932,13 @@ updated_hit_set_hist.history.pop_front(); ObjectContextRef obc = get_object_context(oid, false); - ceph_assert(obc); - --ctx->delta_stats.num_objects; - --ctx->delta_stats.num_objects_hit_set_archive; - ctx->delta_stats.num_bytes -= obc->obs.oi.size; - ctx->delta_stats.num_bytes_hit_set_archive -= obc->obs.oi.size; + //ceph_assert(obc); + if(obc){ + --ctx->delta_stats.num_objects; + --ctx->delta_stats.num_objects_hit_set_archive; + ctx->delta_stats.num_bytes -= obc->obs.oi.size; + ctx->delta_stats.num_bytes_hit_set_archive -= obc->obs.oi.size; + }; } } causes OSD to crash later: /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: In function 'virtual void PrimaryLogPG::op_applied(const eversion_t&)' thread 7f91e146d700 time 2024-12-29 19:58:13.880029 /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: 10457: FAILED ceph_assert(applied_version <= info.last_update) /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: In function 'virtual void PrimaryLogPG::op_applied(const eversion_t&)' thread 7f91df469700 time 2024-12-29 19:58:13.882629 /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: 10457: FAILED ceph_assert(applied_version <= info.last_update) ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x122) [0x55e07e95b815] 2: (()+0x49899a) [0x55e07e95b99a] 3: (PrimaryLogPG::op_applied(eversion_t const&)+0x1f2) [0x55e07eb9fef2] 4: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x720) [0x55e07ed411b0] 5: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xdc5) [0x55e07eba0cd5] 6: (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >)+0x9b) [0x55e07eba2acb] 7: (PrimaryLogPG::hit_set_remove_all()+0x2e5) [0x55e07ebdcbf5] 8: (PrimaryLogPG::on_pool_change()+0xeb) [0x55e07ebddc7b] 9: (PG::handle_advance_map(std::shared_ptr<OSDMap const>, std::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, int, std::vector<int, std::allocator<int> >&, int, PG::RecoveryCtx*)+0x34f) [0x55e07eaf8c4f] 10: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x2f6) [0x55e07ea482e6] 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x55e07ea50a24] 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x4e) [0x55e07ecde60e] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xfea) [0x55e07ea5527a] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x415) [0x55e07f032345] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55e07f034740] 16: (()+0x7ea5) [0x7f9203024ea5] 17: (clone()+0x6d) [0x7f9201ee7b0d] unfortunately here I don't have any clue how to proceed further. I've increased hit_set_count to 32 and hit_set_period to 36000 thus hopefully gaining some time (now OSD seem to be running). Any ideas on how to safely get from this mess? I can't easily get rid of cache tier now since it's used by running VMs and what is worse, I'm not sure I won't hit same problem when deleting cache pool anyways - OSDs are shared by cache pool and NVME data pool so it's pretty uncomfortable situation :-( I'm using bluestore for all OSDs, so can't even try copying other hit_set within filestore.. I'll be very grateful for any help with best regards nikola ciprich

2 1

link to grafana dashboard with osd / host % usage
by Marc 29 Jan '25

29 Jan '25

Is there an existing grafana dashboard/panel that sort of shows the % used on disks and hosts?

3 2

18.2.5 reediness for QE Validation
by Yuri Weinstein 28 Jan '25

28 Jan '25

Happy New Year! We are getting closer to start testing this point release. Almost all PRs marked with "milestone:v18.2.5" were merged. Dev Leads pls tag PRs that must be included appropriately, so we can test and merge them soon. TIA

3 3

AWS SDK change (CRC32 checksums on multiple objects)
by Dave Holland 28 Jan '25

28 Jan '25

Hi, a colleague brought this to my attention: | AWS recently updated their SDKs to enable CRC32 checksums on multiple | object operations by default. | R2 does not currently support CRC32 checksums, and the default | configurations will return header related errors such as Header | 'x-amz-checksum-algorithm' with value 'CRC32' not implemented. https://www.cloudflarestatus.com/incidents/t5nrjmpxc1cj Is this likely to affect Ceph/radosgw users, if they update their clients to the new AWS SDKs? Or does radosgw already support this? thanks, Dave -- ** Dave Holland ** Systems Support -- Informatics Systems Group ** ** dh3(a)sanger.ac.uk ** Wellcome Sanger Institute, Hinxton, UK ** ---------------------------------------------------------------------- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.

5 4

Grafana certificates storage and Squid
by Thorsten Fuchs 28 Jan '25

28 Jan '25

We recently migrated our cluster from 18.2.4 to 19.2.0 and started having issues with Grafana. Ceph gives out the warning "CEPHADM_CERT_ERROR: Invalid grafana certificate on host cc-1: Invalid certificate key: [('PEM routines', '', 'no start line')]. Looking at the certificates they contain a line '# generated by cephadm' and are not the certificates we stored in the config, e.g. 'mgr/cephadm/cc-1/grafana_crt'. After some investigation we found that there was a change to how certificates are stored. Yet I could find no documentation on how to setup per hosts certificates with the new way. The commit that changed the certificate store is https://github.com/ceph/ceph/commit/bb7e715320e41f5d6b6291769e2b6d230eec74cc Maybe anyone can point us in the right direction on how to get our own certs back into Grafana. -- Thorsten Fuchs abaut GmbH Agnes-Pockels-Bogen 1, 80992 München

3 2

Cephfs bug in default Debian 12 (bookworm) kernel v6.1
by Frédéric Nass 28 Jan '25

28 Jan '25

Hello, I just wanted to share here that the default Debian 12 (bookworm) kernel v6.1 is affected by a bug that leads to cephfs client "failing to respond to capability release". This bug has been investigated here [1] and there [2]. To work around this, make sure you're using a Ceph version above v16.2.15 or v17.2.6 or v18.2.1 and an upstream kernel client above v6.5 (like kernel v6.12.9 from bookworm-backports repo) that includes this patch [3]. Cheers, Frédéric. [1] https://tracker.ceph.com/issues/57244 [2] https://tracker.ceph.com/issues/67221 [3] https://patchwork.kernel.org/project/ceph-devel/patch/20230618231011.9077-1…

1 0

Slow initial boot of OSDs in large cluster with unclean state
by Thomas Byrne - STFC UKRI 28 Jan '25

28 Jan '25

Hi all, On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today... Looking at the logs of a new OSD when it's starting: 2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs. I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling. The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me. Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers? (There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.) So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation. Thanks, Tom

8 21

Seeking Participation! Take the new Ceph User Stores Survey!
by Laura Flores 28 Jan '25

28 Jan '25

Hi all, The Ceph User Council is conducting a survey to gather insights from community members who actively use production Ceph clusters. We want to hear directly from you: *What is the use case of your production Ceph cluster?* Since its official Argonaut release in 2012, Ceph has grown significantly in features and user adoption. By learning about your use cases, we aim to understand Ceph’s strengths and limitations in performance, scalability, and usability. This feedback will help inform future improvements to Ceph. Our ultimate goal is to enhance the Ceph community's shared knowledge by publishing real-world user stories on our website. These stories will serve as valuable resources for both current and future users, demonstrating the diversity and potential of Ceph in production environments. All responses are anonymous unless the participant willingly shares their contact information. The Ceph User Council will contact participants who opt to share their contact information before anything is officially published. We have not yet determined when the survey will close, but we will give a heads up before doing so. Take the survey here! https://docs.google.com/forms/d/e/1FAIpQLSe66NedXh4gHLgk9G45eqP5V2wHlz4IKqR… Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

4 12

Mix NVME's in a single cluster
by Bruno Gomes Pessanha 28 Jan '25

28 Jan '25

I have a Ceph Reef cluster with 10 hosts with 16 nvme slots but only half occupied with 15TB (2400 KIOPS) drives. 80 drives in total. I want to add another 80 to fully populate the slots. The question: What would be the downside if I expand the cluster with 80 x 30TB (3300 KIOPS) drives? Thank you! Bruno

4 6

radosgw daemons with "stuck ops"
by Reid Guyett 27 Jan '25

27 Jan '25

Hello, We are experiencing slowdowns on one of our radosgw clusters. We restart the radosgw daemons every 2 hours and things start getting slow after an hour and a half. The avg get/put latencies go from 20ms/400ms to 1s/5s+ according to the metrics. When I stop traffic to one of the radosgw daemon by setting it to DRAIN in HAProxy and HAProxy reports 0 sessions to the daemon, I still see the objecter_requests still going up and down 15 min later. linger_ops seems to stay at a constant 10 the whole time. > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 46 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 211 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 427 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 26 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 190 > The requests say they are "call rgw.bucket_list in=234b" and mostly reference ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.N" where N is 1-149 (the shard count for the bucket). > { > "tid": 2583119, > "pg": "8.579482b0", > "osd": 208, > "object_id": > ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.124", > "object_locator": "@8", > "target_object_id": > ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.124", > "target_object_locator": "@8", > "paused": 0, > "used_replica": 0, > "precalc_pgid": 0, > "last_sent": "1796975.496664s", > "age": 0.350012616, > "attempts": 1, > "snapid": "head", > "snap_context": "0=[]", > "mtime": "1970-01-01T00:00:00.000000+0000", > "osd_ops": [ > "call rgw.bucket_list in=234b" > ] > }, > I don't think it should be other rgw processes because we have this daemon's ceph.conf set to disable the other threads. > rgw enable gc threads = false > rgw enable lc threads = false > rgw dynamic resharding = false > When I restart the service while it is still in a DRAINED state in HAProxy, checking the objecter_requests yields 0 even a few minutes after it has been up. > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > Any thoughts on why these ops appear to be stuck/recurring until restarting the daemon? I think this is related to our performance issues but I don't know what the fix is. The rgws are 18.2.4 running as containers in Podman on Debian 11. Our other clusters do not exhibit this behavior. Thanks! Reid

2 2

Understanding how crush works
by Andre Tann 27 Jan '25

27 Jan '25

Hi list, I have a problem understanding on how crush works when the crush map changes. Let's take a pool with some data in it, and a crush map that enables a client to calculate itself where a particular chunk is stored. Now we add more OSDs, which means, the crush map changes. Now most objects are misplaced, given the new crush map. If the clients wants a particular chunk, it takes the modified map, but as the chunk is misplaced, it won't find it where the crush algorithm points to. How can the client know which crush map to consider when doing the calculation? Do the clients keep several versions of the map, and try them one after the other? Thanks for some hints on this. -- Andre Tann

2 1

Ceph Steering Committee Notes 2025-01-27
by Dan van der Ster 27 Jan '25

27 Jan '25

Hi all, Here is a summary of the CSC meeting Jan 27, 2025. Full notes are available at https://pad.ceph.com/p/csc-weekly-minutes Component Leads Poll re: Workload and Bottlenecks Dan proposes an informal poll for component leads to identify workload distribution, bottlenecks, and areas where community contributions are most needed. Venky shares that the CephFS team uses GitHub's "Assigned" field to track PR reviews and suggests exploring good and bad aspects of roles to involve more community support. LRC On-Call Assistance DavidG seeks improved mechanisms to involve developers and leads during LRC outages. In a recent case, an outage was related (again) to iSCSI, which is missing a lead maintainer. It was agreed to have a follow up discussion on David's plans for modernizing the upstream infra in the next meeting. Tentacle Roadmap Tentacle development freeze and release candidate timelines are not finalized. Historically, dev freeze occurs by the end of January and RC by the end of March. Discussions are to continue at the next CSC meeting. BOTO3 Issue A BOTO3 change caused S3 client issues, with backports and fixes in progress. Relevant links and PRs for tracking the problem are shared. Strategy Working Group Yehuda proposes forming a strategy working group to address high-level Ceph planning over a 3–5 year horizon, as existing forums like CSC and Dev Summit are too technical for such discussions. Discussion to continue next meeting. Cephalocon 2024 Recordings Most recordings from Cephalocon 2024 are now available on YouTube. Announcements to the mailing list are planned.

1 0

Cephalocon 2024 Recordings Available
by Matt Vandermeulen 27 Jan '25

27 Jan '25

Hi folks! The Cephalocon 2024 recordings are available on the YouTube channel! - Channel: https://www.youtube.com/@Cephstorage/videos - Cephalocon 2024 playlist: https://www.youtube.com/watch?v=ECkgu2zZzeQ&list=PLrBUGiINAakPfVfFfPQ5wLMQJ… Thanks, Matt

1 0

unmatched rstat rbytes on single dirfrag
by Frank Schilder 27 Jan '25

27 Jan '25

Hi all, I see error messages like these in the logs every now and then: 10:14:44 [ERR] unmatched rstat rbytes on single dirfrag 0x615, inode has n(v2211 rc2038-01-18T21:22:13.000000+0100 b2506575730 9676264=3693+9672571), dirfrag has n(v2211 rc2025-01-24T10:14:44.628760+0100 b30517 102=3+99) 10:14:44 [ERR] unmatched fragstat size on single dirfrag 0x615, inode has f(v67 m2025-01-24T10:14:44.628760+0100 9676556=3819+9672737), dirfrag has f(v67 m2025-01-24T10:14:44.628760+0100 102=3+99) 10 /var/log/ceph/ceph.log-20250123:2025-01-22T17:56:18.060011+0100 mds.ceph-11 (mds.6) 4 : cluster [ERR] unmatched fragstat on 0x641, inode has f(v14 m2025-01-18T11:59:21.676346+0100 2801=913+1888), dirfrags have f(v0 m2025-01-18T11:59:21.676346+0100 427=9+418) /var/log/ceph/ceph.log-20250123:2025-01-22T17:56:18.076061+0100 mds.ceph-11 (mds.6) 5 : cluster [ERR] inconsistent rstat on inode 0x641, inode has n(v194 rc2038-01-07T22:42:17.000000+0100 b46230814357 2802=913+1889), directory fragments have n(v0 rc2032-04-29T16:39:38.000000+0200 b46204251164 428=9+419) How critical are these? Everything seems to work normal. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 5

Error ENOENT: Module not found
by Devender Singh 27 Jan '25

27 Jan '25

Hello all Any quick fix for … root@sea-devnode1:~# ceph orch ls Error ENOENT: Module not found Regards Dev

6 11

Watcher Issue
by Devender Singh 25 Jan '25

25 Jan '25

Hello Seeking some help if I can clean the client mounting my volume? rbd status pool/image Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088 Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used It shows above client, but there is no such volume… Another similar issue… on dashboard… CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate. Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message.. CEPH version: 18.2.4 Regards Dev

4 13

No recovery after removing node - active+undersized+degraded-- removed osd using purge...
by Devender Singh 25 Jan '25

25 Jan '25

Hello all Urgent help needed. No recovery happening. Tried repairing pg and redeploy or create. Rebooted cluster but no luck.. data: volumes: 2/2 healthy pools: 18 pools, 817 pgs objects: 6.06M objects, 20 TiB usage: 30 TiB used, 302 TiB / 332 TiB avail pgs: 2846742/29091911 objects degraded (9.785%) 2654404/29091911 objects misplaced (9.124%) 516 active+clean 105 active+undersized+degraded Regards Dev

2 1

FS design question around subvolumes vs dirs
by Jesse Galley 24 Jan '25

24 Jan '25

Hello, We are building a ~2PiB 200 OSD cluster that will be used entirely by an FS for hosting data storage. I have a question regarding whether there is any specific reason/advantage to use subvolumes vs plain directories for segregating different departments, envs, etc. Currently, the plan is to create subtrees like: /web/home/env1 /web/logs/env1 /web/logs/env2 /mail/home/env1 ..etc Clients will be path-restricted to the bottom directory with MDS caps, and namespace restricted with OSD caps. Namespace layouts will be set on the mounted dir when new users/dirs are provisioned. Snapshots will be managed by storage admins from the top of the trees. Static MDS pinning will be used throughout. As far as I understand, if I create subvolume groups and subvolumes, I would have to flatten the <department>/<service>/<env> structure a bit to make it work, but it may simplify the manual management of snapshots, caps, and namespaces. Are there any other reasons to use subvolumes via manual dir management? Are subvolumes treated any differently by the cluster than dirs? I am especially concerned with io performance, as well as snap mirroring, as we are building an identical cluster to receive mirrors. Appreciate any insight. Thank you. *Jesse Galley* *Director, Mail Platform & Migrations, Hostopia* *C: 416.816.0866* *jesse.galley(a)hostpapa.com <jesse.galley(a)hostpapa.com>*

1 0

January Ceph Science Virtual User Group
by Kevin Hrpcek 23 Jan '25

23 Jan '25

Hey All, We will be having a Ceph science/research/big cluster call on Tuesday January 28th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. A change for this year is that we will start each meeting with a brief 10-15 minute talk given by a community member. If you're interested in signing up to give a short talk for a meeting please let me know! This month Mattia will present regarding balancing CephFS performance and reliability. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20250128 Virtual event details: January 28th, 2025 15:00 UTC 4pm Central European 9am Central US Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Ceph Community Calendar. Go to Jan 28 and you can quickly add it to your calendar as well https://ceph.io/en/community/meetups/ Kevin

1 0

Help needed, ceph fs down due to large stray dir
by Frank Schilder 23 Jan '25

23 Jan '25

Hi all, we seem to have a serious issue with our file system, ceph version is pacific latest. After a large cleanup operation we had an MDS rank with 100Mio stray entries (yes, one hundred million). Today we restarted this daemon, which cleans up the stray entries. It seems that this leads to a restart loop due to OOM. The rank becomes active and then starts pulling in DNS and INOS entries until all memory is exhausted. I have no idea if there is at least progress removing the stray items or if it starts from scratch every time. If it needs to pull as many DNS/INOS into cache as there are stray items, we don't have a server at hand with enough RAM. Q1: Is the MDS at least making progress in every restart iteration? Q2: If not, how do we get this rank up again? Q3: If we can't get this rank up soon, can we at least move directories away from this rank by pinning it to another rank? Currently, the rank in question reports .mds_cache.num_strays=0 in perf dump. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

7 34

MDS hung in purge_stale_snap_data after populating cache
by Frank Schilder 23 Jan '25

23 Jan '25

Hi all, this post is related to "Help needed, ceph fs down due to large stray dir" (https://www.spinics.net/lists/ceph-users/msg85394.html) We finally got the MDS up and could record what it is doing before and after getting hung. We opened tracker https://tracker.ceph.com/issues/69547 . The uploads were too large to attach. For a quick overview of the key findings see the 3 screen shots posted here: https://imgur.com/a/RF7ExSP . Apparently, the MDS gets hung in CInode::purge_stale_snap_data after populating the cache. I hope that the data uploaded will give someone a clue what is happening and how we can get out of it. Putting my hopes on tomorrow after sleeping a bit (well, at least trying to). Thanks for all your help with this critical situation. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 16

malformed osd ID
by Joffrey 23 Jan '25

23 Jan '25

Hi, After a failed OSD during the upgrade to 19.2.0, I renamed a folder. Now, I have an OSD with the name osd.60.oldf. This bug impacts various commands and services. The host is offline (but it is not — another OSD, osd.67, is up and in). Some commands return the following error: cephadm shell -- ceph cephadm get-ssh-config: Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: invalid literal for int() with base 10: '60.oldf' ceph osd status: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1864, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/status/module.py", line 337, in handle_osd_status assert metadata AssertionError I tried to remove the daemon: ceph orch daemon rm osd.60.oldf --force but received: TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' because the host is offline, and get-ssh-config does not work... I have removed all data related to osd.60 and osd.60.oldf on the host hosting this OSD. How can I remove this daemon entry osd.60.oldf from Ceph's store? osd.60.oldf only appears in orch ps, not in osd ls... Thank you.

2 1

Bad/strange performance on a new cluster
by Jan 22 Jan '25

22 Jan '25

Hi all, we have a new Ceph cluster and have some very bad/strange performance behavior. I really don't understand what I'm doing wrong here and would be more than happy if anyone has an idea. Even a hint on what to look at would be helpful. Some Information: Machines (8 Nodes) per Node: - CPU 2x Intel(R) Xeon(R) Gold 6258R CPU @ 2.70GHz (28 Cores) - 384 GB RAM - 20x Dell Ent NVMe AGN RI U.2 7.68TB (for OSDs) - 4x 25G LACP Backend - 2x 25G LACP Frontend - OS: - Ubuntu 22.04 - Kernel: 5.15.0 - Ceph: - Version 18.2.4 - 160 osds - 4096 PGs for the VM pool I took some fio benchmarks from the Proxmox Ceph Performance Paper: https://www.proxmox.com/images/download/pve/docs/Proxmox-VE_Ceph-Benchmark-… The First test should have about 1500 IOPS (Proxmox Paper: 1806). We only get 170. root@ceph001:/mnt# fio --ioengine=psync --filename=test_fio --size=9G --time_based --name=fio --group_reporting --runtime=60 --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.28 Starting 1 process fio: Laying out IO file (1 file / 9216MiB) Jobs: 1 (f=1): [W(1)][100.0%][w=680KiB/s][w=170 IOPS][eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=174797: Wed Jan 22 20:19:19 2025 write: IOPS=202, BW=811KiB/s (831kB/s)(47.5MiB/60003msec); 0 zone resets clat (usec): min=2185, max=20081, avg=4925.43, stdev=931.63 lat (usec): min=2186, max=20082, avg=4926.19, stdev=931.63 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 3818], 10.00th=[ 3982], 20.00th=[ 4293], | 30.00th=[ 4490], 40.00th=[ 4686], 50.00th=[ 4817], 60.00th=[ 5014], | 70.00th=[ 5211], 80.00th=[ 5407], 90.00th=[ 5800], 95.00th=[ 6063], | 99.00th=[ 8586], 99.50th=[ 9503], 99.90th=[12256], 99.95th=[13304], | 99.99th=[19006] bw ( KiB/s): min= 672, max= 1000, per=100.00%, avg=813.11, stdev=73.18, samples=119 iops : min= 168, max= 250, avg=203.28, stdev=18.29, samples=119 lat (msec) : 4=10.24%, 10=89.43%, 20=0.32%, 50=0.01% cpu : usr=0.25%, sys=2.57%, ctx=36503, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12167,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=811KiB/s (831kB/s), 811KiB/s-811KiB/s (831kB/s-831kB/s), io=47.5MiB (49.8MB), run=60003-60003msec Disk stats (read/write): rbd0: ios=0/24296, merge=0/2, ticks=0/56351, in_queue=56351, util=99.97% Bandwith and IOPs with more IO depth look ok form me: fio --filename=/mnt/testingfio1 --size=50GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=150 --numjobs=1 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1 iops-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256 fio-3.28 iops-test-job: (groupid=0, jobs=1): err= 0: pid=146931: Wed Jan 22 19:43:14 2025 read: IOPS=20.0k, BW=78.0MiB/s (81.8MB/s)(11.4GiB/150006msec) slat (nsec): min=1245, max=7415.7k, avg=22636.97, stdev=224525.03 clat (usec): min=238, max=32714, avg=5620.53, stdev=2255.97 lat (usec): min=243, max=32721, avg=5643.32, stdev=2258.28 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2311], 10.00th=[ 2671], 20.00th=[ 3654], | 30.00th=[ 4146], 40.00th=[ 4752], 50.00th=[ 5342], 60.00th=[ 6128], | 70.00th=[ 6915], 80.00th=[ 7635], 90.00th=[ 8717], 95.00th=[ 9896], | 99.00th=[10683], 99.50th=[10945], 99.90th=[11863], 99.95th=[12649], | 99.99th=[14615] bw ( KiB/s): min=63254, max=98432, per=100.00%, avg=79914.04, stdev=6290.69, samples=299 iops : min=15813, max=24608, avg=19978.36, stdev=1572.70, samples=299 write: IOPS=19.9k, BW=77.9MiB/s (81.7MB/s)(11.4GiB/150006msec); 0 zone resets slat (nsec): min=1349, max=8871.7k, avg=23250.80, stdev=225370.53 clat (usec): min=629, max=81108, avg=7160.58, stdev=2338.33 lat (usec): min=633, max=81114, avg=7183.98, stdev=2348.98 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3982], 10.00th=[ 4293], 20.00th=[ 5014], | 30.00th=[ 5735], 40.00th=[ 6325], 50.00th=[ 6980], 60.00th=[ 7570], | 70.00th=[ 8225], 80.00th=[ 9110], 90.00th=[10421], 95.00th=[11207], | 99.00th=[13435], 99.50th=[14353], 99.90th=[16581], 99.95th=[17957], | 99.99th=[21365] bw ( KiB/s): min=61755, max=98813, per=100.00%, avg=79877.64, stdev=6336.82, samples=299 iops : min=15438, max=24703, avg=19969.22, stdev=1584.22, samples=299 lat (usec) : 250=0.01%, 500=0.03%, 750=0.07%, 1000=0.08% clat (usec): min=238, max=32714, avg=5620.53, stdev=2255.97 [141/1761] lat (usec): min=243, max=32721, avg=5643.32, stdev=2258.28 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2311], 10.00th=[ 2671], 20.00th=[ 3654], | 30.00th=[ 4146], 40.00th=[ 4752], 50.00th=[ 5342], 60.00th=[ 6128], | 70.00th=[ 6915], 80.00th=[ 7635], 90.00th=[ 8717], 95.00th=[ 9896], | 99.00th=[10683], 99.50th=[10945], 99.90th=[11863], 99.95th=[12649], | 99.99th=[14615] bw ( KiB/s): min=63254, max=98432, per=100.00%, avg=79914.04, stdev=6290.69, samples=299 iops : min=15813, max=24608, avg=19978.36, stdev=1572.70, samples=299 write: IOPS=19.9k, BW=77.9MiB/s (81.7MB/s)(11.4GiB/150006msec); 0 zone resets slat (nsec): min=1349, max=8871.7k, avg=23250.80, stdev=225370.53 clat (usec): min=629, max=81108, avg=7160.58, stdev=2338.33 lat (usec): min=633, max=81114, avg=7183.98, stdev=2348.98 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3982], 10.00th=[ 4293], 20.00th=[ 5014], | 30.00th=[ 5735], 40.00th=[ 6325], 50.00th=[ 6980], 60.00th=[ 7570], | 70.00th=[ 8225], 80.00th=[ 9110], 90.00th=[10421], 95.00th=[11207], | 99.00th=[13435], 99.50th=[14353], 99.90th=[16581], 99.95th=[17957], | 99.99th=[21365] bw ( KiB/s): min=61755, max=98813, per=100.00%, avg=79877.64, stdev=6336.82, samples=299 iops : min=15438, max=24703, avg=19969.22, stdev=1584.22, samples=299 lat (usec) : 250=0.01%, 500=0.03%, 750=0.07%, 1000=0.08% lat (msec) : 2=0.65%, 4=15.15%, 10=75.45%, 20=8.55%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=7.36%, sys=18.55%, ctx=155494, majf=0, minf=9200 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=2993949,2992429,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=78.0MiB/s (81.8MB/s), 78.0MiB/s-78.0MiB/s (81.8MB/s-81.8MB/s), io=11.4GiB (12.3GB), run=150006-150006msec WRITE: bw=77.9MiB/s (81.7MB/s), 77.9MiB/s-77.9MiB/s (81.7MB/s-81.7MB/s), io=11.4GiB (12.3GB), run=150006-150006msec Disk stats (read/write): rbd0: ios=2989470/2987987, merge=0/1, ticks=9760844/13043854, in_queue=22804699, util=100.00% We have 4096 PG on the tested pool. root@ceph001:/mnt# ceph -s cluster: id: health: HEALTH_OK services: mon: 5 daemons, quorum ceph001,ceph002,ceph003,ceph005,ceph006 (age 52m) mgr: ceph002.hgppdu(active, since 2d), standbys: ceph001.ooznoq osd: 160 osds: 160 up (since 5w), 160 in (since 5M) rgw: 2 daemons active (2 hosts, 1 zones) data: pools: 11 pools, 8449 pgs objects: 7.80M objects, 19 TiB usage: 56 TiB used, 1.0 PiB / 1.1 PiB avail root@ceph001:~# ceph config get osd osd_memory_target 4294967296 root@ceph001:~# ceph config get osd WHO MASK LEVEL OPTION VALUE ... osd advanced osd_memory_target_autotune true ... We would like to use the cluster with OpenStack Cinder. But the tests were made directly on the cluster nodes with rbd. The metrics on VMs are similar. Thanks in advance. Jan

1 0

osd won't restart
by Dominique Ramaekers 22 Jan '25

22 Jan '25

Hi, Strange thing just happened (ceph v19.2.0). I added two disks to a host. Kernel recognized nicely the two disks and they appeared as available devices in ceph. After 15 minutes osd's were not created, so I looked at the logs: /usr/bin/docker: stderr --> Creating keyring file for osd.36 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/keyring /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/ /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 36 --monmap /var/lib/ceph/osd/ceph-36/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd-data /var/lib/ceph/osd/ceph-36/ --osd-uuid 41675779-943d-4dca-baa3-3a4f6ace004a --setuser ceph --setgroup ceph /usr/bin/docker: stderr stderr: 2025-01-20T08:21:01.979+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/docker: stderr stderr: 2025-01-20T08:21:01.979+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/docker: stderr stderr: 2025-01-20T08:21:01.980+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/docker: stderr stderr: 2025-01-20T08:21:01.980+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.075+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.076+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.506+0000 79e9fea34640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.506+0000 79e9fea34640 -1 bdev(0x566a21f14a80 /var/lib/ceph/osd/ceph-36//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.506+0000 79e9fea34640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/docker: stderr stderr: 2025-01-20T08:21:02.506+0000 79e9fea34640 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-36/: (13) Permission denied /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes As it sad "Permission denied" and I have already osd's running, I thought it was an issue that docker may be updated but not restarted. So I did 'systemcrl restart docker.service'. Now none of the managed osd's are going back online!!! 'systemctl start ceph-dd4b0610-b4d2-11ec-bb58-d1b32ae31585(a)osd.18.service<mailto:ceph-dd4b0610-b4d2-11ec-bb58-d1b32ae31585@osd.18.service>' fails with not much explanation... Only unmanaged osd have no issue... I didn't payed much attention to the log entry '_read_fsid unparsable uuid'... So I think there is more going on. Permission denied is logical if the path is wrong... see dubbel slash in '_read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block' Is this a bug like https://github.com/rook/rook/issues/10219 ? Can I get around this without recreating these osd as unmanaged? Thanks in advance. Dominique.

2 4

shell fautly command
by Dominique Ramaekers 22 Jan '25

22 Jan '25

Hi, I was removing config keys of osd's that were purged... I accidently entered the following command: #ceph config-key rm osd Feedback "key deleted" :-( Did I do something very stupid??? The output of my osd keys: # ceph config-key ls | grep 'osd' | grep -v history "config/global/osd_pool_default_pg_autoscale_mode", "config/osd.0/osd_mclock_max_capacity_iops_hdd", "config/osd.0/osd_mclock_max_capacity_iops_ssd", "config/osd.1/osd_mclock_max_capacity_iops_hdd", "config/osd.1/osd_mclock_max_capacity_iops_ssd", "config/osd.10/osd_mclock_max_capacity_iops_ssd", "config/osd.11/osd_mclock_max_capacity_iops_ssd", "config/osd.12/osd_mclock_max_capacity_iops_ssd", "config/osd.13/osd_mclock_max_capacity_iops_ssd", "config/osd.14/osd_mclock_max_capacity_iops_ssd", "config/osd.15/osd_mclock_max_capacity_iops_ssd", "config/osd.16/osd_mclock_max_capacity_iops_ssd", "config/osd.17/osd_mclock_max_capacity_iops_ssd", "config/osd.2/osd_mclock_max_capacity_iops_hdd", "config/osd.2/osd_mclock_max_capacity_iops_ssd", "config/osd.20/osd_mclock_max_capacity_iops_ssd", "config/osd.21/osd_mclock_max_capacity_iops_ssd", "config/osd.22/osd_mclock_max_capacity_iops_ssd", "config/osd.23/osd_mclock_max_capacity_iops_ssd", "config/osd.24/osd_mclock_max_capacity_iops_ssd", "config/osd.25/osd_mclock_max_capacity_iops_ssd", "config/osd.26/osd_mclock_max_capacity_iops_ssd", "config/osd.27/osd_mclock_max_capacity_iops_ssd", "config/osd.28/osd_mclock_max_capacity_iops_ssd", "config/osd.29/osd_mclock_max_capacity_iops_ssd", "config/osd.3/osd_mclock_max_capacity_iops_hdd", "config/osd.3/osd_mclock_max_capacity_iops_ssd", "config/osd.30/osd_mclock_max_capacity_iops_ssd", "config/osd.31/osd_mclock_max_capacity_iops_ssd", "config/osd.32/osd_mclock_max_capacity_iops_ssd", "config/osd.33/osd_mclock_max_capacity_iops_ssd", "config/osd.34/osd_mclock_max_capacity_iops_ssd", "config/osd.35/osd_mclock_max_capacity_iops_ssd", "config/osd.37/osd_mclock_max_capacity_iops_ssd", "config/osd.38/osd_mclock_max_capacity_iops_ssd", "config/osd.4/osd_mclock_max_capacity_iops_hdd", "config/osd.4/osd_mclock_max_capacity_iops_ssd", "config/osd.5/osd_mclock_max_capacity_iops_hdd", "config/osd.5/osd_mclock_max_capacity_iops_ssd", "config/osd.6/osd_mclock_max_capacity_iops_ssd", "config/osd.7/osd_mclock_max_capacity_iops_ssd", "config/osd.8/osd_mclock_max_capacity_iops_ssd", "config/osd.9/osd_mclock_max_capacity_iops_ssd", "config/osd/host:hvs001/osd_memory_target", "config/osd/host:hvs002/osd_memory_target", "config/osd/host:hvs003/osd_memory_target", "config/osd/host:hvs005/osd_memory_target", "config/osd/osd_memory_target_autotune", "mgr/cephadm/osd_remove_queue", "mgr/cephadm/spec.osd.all-available-devices",

2 1

January Ceph Science Virtual User Group
by Kevin Hrpcek 22 Jan '25

22 Jan '25

Hey All, We will be having a Ceph science/research/big cluster call on Tuesday January 28th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. A change for this year is that we will start each meeting with a brief 10-15 minute talk given by a community member. If you're interested in signing up to give a short talk for a meeting please let me know! This month Mattia will present regarding balancing CephFS performance and reliability. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20250128 Virtual event details: January 28th, 2025 15:00 UTC 4pm Central European 9am Central US Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Ceph Community Calendar. Go to Jan 28 and you can quickly add it to your calendar as well https://ceph.io/en/community/meetups/ Kevin

1 0

Changing crush map result in > 100% objects degraded
by Kasper Rasmussen 21 Jan '25

21 Jan '25

Hi community Please help me understand what is going on. I have a ceph (Reef) test cluster with the following crushmap ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 12.00000 root default -7 3.00000 host ksr-ceph-osd1 0 hdd 1.00000 osd.0 6 hdd 1.00000 osd.6 10 hdd 1.00000 osd.10 -9 3.00000 host ksr-ceph-osd2 3 hdd 1.00000 osd.3 7 hdd 1.00000 osd.7 11 hdd 1.00000 osd.11 -5 3.00000 host ksr-ceph-osd3 2 hdd 1.00000 osd.2 5 hdd 1.00000 osd.5 9 hdd 1.00000 osd.9 -3 3.00000 host ksr-ceph-osd4 1 hdd 1.00000 osd.1 4 hdd 1.00000 osd.4 8 hdd 1.00000 osd.8 -11 0 rack rack1 -13 0 rack rack2 -15 0 rack rack3 -17 0 rack rack4 Ceph status is like: cluster: id: 8a174287-42f8-43b6-9973-f174110b508b health: HEALTH_OK services: mon: 5 daemons, quorum ksr-ceph-mon2,ksr-ceph-mon3,ksr-ceph-mon1,ksr-ceph-mon5,ksr-ceph-mon4 (age 3h) mgr: ksr-ceph-mon1(active, since 3h), standbys: ksr-ceph-mon2, ksr-ceph-mon3 mds: 2/2 daemons up, 3 standby osd: 12 osds: 12 up (since 3h), 12 in (since 6d) data: volumes: 2/2 healthy pools: 5 pools, 129 pgs objects: 578 objects, 154 MiB usage: 1.0 GiB used, 599 GiB / 600 GiB avail pgs: 129 active+clean I then run: ceph osd set norecover ceph osd set nobackfill ceph osd set norebalance ceph osd crush move ksr-ceph-osd1 rack=rack1 ceph osd crush move ksr-ceph-osd2 rack=rack2 ceph osd crush move ksr-ceph-osd3 rack=rack3 ceph osd crush move ksr-ceph-osd4 rack=rack4 resulting in the following crush tree ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 12.00000 root default -11 3.00000 rack rack1 -7 3.00000 host ksr-ceph-osd1 0 hdd 1.00000 osd.0 6 hdd 1.00000 osd.6 10 hdd 1.00000 osd.10 -13 3.00000 rack rack2 -9 3.00000 host ksr-ceph-osd2 3 hdd 1.00000 osd.3 7 hdd 1.00000 osd.7 11 hdd 1.00000 osd.11 -15 3.00000 rack rack3 -5 3.00000 host ksr-ceph-osd3 2 hdd 1.00000 osd.2 5 hdd 1.00000 osd.5 9 hdd 1.00000 osd.9 -17 3.00000 rack rack4 -3 3.00000 host ksr-ceph-osd4 1 hdd 1.00000 osd.1 4 hdd 1.00000 osd.4 8 hdd 1.00000 osd.8 And ceph status is like: cluster: id: 8a174287-42f8-43b6-9973-f174110b508b health: HEALTH_WARN nobackfill,norebalance,norecover flag(s) set Degraded data redundancy: 2701/1734 objects degraded (155.767%), 55 pgs degraded services: mon: 5 daemons, quorum ksr-ceph-mon2,ksr-ceph-mon3,ksr-ceph-mon1,ksr-ceph-mon5,ksr-ceph-mon4 (age 3h) mgr: ksr-ceph-mon1(active, since 3h), standbys: ksr-ceph-mon2, ksr-ceph-mon3 mds: 2/2 daemons up, 3 standby osd: 12 osds: 12 up (since 3h), 12 in (since 6d); 22 remapped pgs flags nobackfill,norebalance,norecover data: volumes: 2/2 healthy pools: 5 pools, 129 pgs objects: 578 objects, 154 MiB usage: 1.0 GiB used, 599 GiB / 600 GiB avail pgs: 2701/1734 objects degraded (155.767%) 479/1734 objects misplaced (27.624%) 70 active+clean 34 active+recovery_wait+degraded 20 active+recovery_wait+undersized+degraded+remapped 3 active+recovering 1 active+recovery_wait+remapped 1 active+recovery_wait+degraded+remapped No crush rules have been changed on the pools. All pools has the default replicated_rule ceph osd crush rule dump replicated_rule { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } Questions 1 - Why do this result in such a high - objects degraded - percentage? 2 - Why do PGs get undersized? All in all this behavior does not make sense to me - I expect nothing to happen basically since I've only implemented some buckets in the map, no rules have changed - so I reach out in hope that someone can explain me the logic.

4 6

Emergency support request for ceph MDS trouble shooting
by Frank Schilder 20 Jan '25

20 Jan '25

Dear all, this is a request to companies/consultants with development experience on ceph for a contract to help us out of our current file system outage. If you can offer help, please send a PM directly back to me. Short description with links to what we found out already: We experience a total outage of access to our ceph file system due to MDS rank 2 getting hung on startup. It does not crash. This rank has .mds_cache.num_strays=99446248 and the initial processing of stray items leads to a producer-consumer deadlock that prevents this rank from becoming responsive. On startup approximately 37Mio stray items can be scheduled for purging before the dead lock occurs. The cluster is healthy otherwise. Ceph version is 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable). We have deployed a rescue server with large amount of RAM and swap that we can make accessible remotely. On this rescue server we have an MDS up on rank 2 for trouble shooting and this is all working fine. Related bug reports with more details: https://tracker.ceph.com/issues/69547 https://www.spinics.net/lists/ceph-users/msg85394.html https://www.spinics.net/lists/ceph-users/msg85480.html Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

Notes from CSC Weekly 2025-01-20
by Dan van der Ster 20 Jan '25

20 Jan '25

Hi all, The CSC Weekly (formerly known as the CLT Weekly) was shortened today due to a US holiday. All planned agenda items were postponed to next week. There was however one technical point to be communicated today: Radek would like all component leads to review PR https://github.com/ceph/ceph/pull/60104, which touches bufferlists across the stack. Otherwise, also please note that the etherpad for the CSC Weekly has been moved to https://pad.ceph.com/p/csc-weekly-minutes Cheers, Dan

1 0

ceph orch ls --refresh
by Alex Hussein-Kershaw (HE/HIM) 20 Jan '25

20 Jan '25

Hi Folks, Curious about the --refresh argument on this command (and similar). From experience so far, and a bit of a code read, it seems that: * Passing --refresh will kick the serve loop to run. * It won't wait for it to complete, so data returned may not be the latest. * Re-running the "ceph orch ls" command soon after may or may not return the latest data, depending on if the serve loop has completed. * The serve loop will run once every N seconds anyway (looks like 60 for me, which I assume is the default?) so data shouldn't be more than a minute out of date in normal operation. Is that an accurate understanding? I'm looking to use this command in some scripts so would like to be sure I understand what I am doing. Thanks, Alex

1 0

Help needed: s3cmd set ACL command possess S3 error: 400 (InvalidArgument) in squid ceph version.
by Saif Mohammad 20 Jan '25

20 Jan '25

Hello Community, We are trying to set ACL for one of the objects by s3cmd tool within the buckets to be public by using the command as follows but we are unable to set it in squid ceph version, however the same was done in the reef version, we were successfully able to set it public. Please let me know if we have any solutions for it. $ s3cmd setacl s3://<my-bucket>/logo.png --acl-public ERROR: S3 error: 400 (InvalidArgument): Invalid group uri Thanks, Mohammad Saif

2 2

Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience
by Kasper Rasmussen 20 Jan '25

20 Jan '25

I'm managing a ceph cluster with +1K OSDs distributed accross 56 host. Untill now the crush rule used is the default replicated rule, but I want to change that in order to implement failure domain on rack level. Ceph version: Pacific 16.2.15 All pools(RBD and CephFS) currently use the default replicated_rule All OSD hosts has 25G network, and spinning disks(HDD) MON DBs is on NVMEs Workload is 24x7x365 The builtin balancer is disabled, and has been for a long time - Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30 Current plan is to * Disable rebalancing and backfilling by executing - ceph osd set norebalance; ceph osd set nobackfill; * Add all 7 Rack to crushmap and distribute the hosts (8 in each.) by using the built in commands like - * ceph osd crush add-bucket rack<#> rack root=default * ceph osd crush move osd-host<#> rack=rack<#> * Create the new rack split rule with command * ceph osd crush rule create-replicated rack_split default rack * Set the rule across all my pools * for p in $(ceph osd lspools | cut -d' ' -f 2) ; do echo $p $(ceph osd pool set $p crush_rule rack_split) ; done * I will probably also be using upmap-remapped.py here.. * Finally enable rebalancing, and backfilling - ceph osd unset norebalance; ceph osd unset nobackfill; However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation. Also I've seen some weird behavior - where Pacific(16) seems to do something different from Quincy(17) - In a test cluster I've tested the plan - On Pasific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue) On Quincy: Data is marked as misplaced - which seems correct. All experience and/or input will be greatly appreciated.

4 7

weekend maintenance to bot+bridge slack/irc/discord
by Alvaro Soto 18 Jan '25

18 Jan '25

Hi All, This weekend, I'll be moving the bot from the primary node, so expect some hiccups in the bridge and log bot. Cheers. -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 1

squid 19.2.1 RC QE validation status
by Yuri Weinstein 17 Jan '25

17 Jan '25

Details of this release are summarized here: https://tracker.ceph.com/issues/69234#note-1 Release Notes - TBD LRC upgrade - TBD Gibba upgrade -TBD Please provide tracks for failures so we avoid duplicates. Seeking approvals/reviews for: rados - Radek, Laura rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume

14 33

Non existing host in maintenance
by Dominique Ramaekers 17 Jan '25

17 Jan '25

Hi, I have removed a host (hvs004) that was in maintenance. The system disk of this host had failed, so removed the host hvs004 in ceph; replaced the system disk; erased all the osd-disks and reinstalled the host as hvs005. Resulting a cluster status in waring that doesn’t goes away: health: HEALTH_WARN 1 host is in maintenance mode Removal is done by “ceph orch host rm hvs004 --offline –force” in cephadm shell. How can I correct this false warning? Some more info: root@hvs001:/# ceph orch host ls HOST ADDR LABELS STATUS hvs001 xxx.xxx.xxx.xxx _admin hvs002 xxx.xxx.xxx.xxx _admin hvs003 xxx.xxx.xxx.xxx _admin hvs005 xxx.xxx.xxx.xxx _admin 4 hosts in cluster root@hvs001:/# ceph health detail HEALTH_WARN 1 host is in maintenance mode [WRN] HOST_IN_MAINTENANCE: 1 host is in maintenance mode hvs004 is in maintenance :-/ Help is greatly apreciated…

2 4

Cephadm: Specifying RGW Certs & Keys By Filepath
by Alex Hussein-Kershaw (HE/HIM) 17 Jan '25

17 Jan '25

Hi Folks, Looking for some advice on RGW service specs and Cephadm. I've read the docs here: RGW Service — Ceph Documentation<https://docs.ceph.com/en/reef/cephadm/services/rgw/> Using a service spec I can deploy a RGW: service_type: rgw service_id: '25069123' service_name: rgw.25069123 placement: hosts: - raynor-sc-1 extra_container_args: - -v - /etc/pki:/etc/pki:ro - -v - /etc/ssl/:/etc/ssl:ro spec: rgw_frontend_port: 7480 rgw_realm: geored_realm rgw_zone: siteA rgw_zonegroup: geored_zg ssl: true rgw_frontend_extra_args: - "ssl_certificate=/etc/ssl/certs/server.crt" - "ssl_private_key=/etc/ssl/private/server.key" But it fails to start because my "rgw_frontends" config ends up as this. Notice the two entries of "ssl_certificate". beast ssl_port=7480 ssl_certificate=config://rgw/cert/rgw.25069123 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key That causes it to fail to start: debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid 7 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework: beast debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_port, val: 7480 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: config://rgw/cert/rgw.25069123 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: /etc/ssl/certs/server.crt debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_private_key, val: /etc/ssl/private/server.key debug 2025-01-16T15:26:54.520+0000 7f65bcfdc840 -1 LDAP not started since no server URIs were provided in the configuration. debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework: beast debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key debug 2025-01-16T15:26:54.597+0000 7f65bcfdc840 0 starting handler: beast debug 2025-01-16T15:26:54.601+0000 7f65bcfdc840 -1 ssl_certificate was not found: rgw/cert/rgw.25069123 debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 no ssl_certificate configured for ssl_port debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: failed initializing frontend debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: initialize frontend fail, r = 22 Correcting that config manually: $ ceph config set client.rgw.25069123.raynor-sc-1.qiatgr rgw_frontends "beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key" Then allows the RGW to start: debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid 7 debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework: beast debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_port, val: 7480 debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_certificate, val: /etc/ssl/certs/server.crt debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_private_key, val: /etc/ssl/private/server.key debug 2025-01-16T15:28:17.767+0000 7fba6e85a840 -1 LDAP not started since no server URIs were provided in the configuration. debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework: beast debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key debug 2025-01-16T15:28:17.860+0000 7fba6e85a840 0 starting handler: beast debug 2025-01-16T15:28:17.908+0000 7fba6e85a840 0 set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:28:17.912+0000 7fba6e85a840 1 mgrc service_daemon_register rgw.643295 metadata {arch=x86_64,ceph_release=squid,ceph_version=ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable),ceph_version_short=19.2.0,container_hostname=raynor-sc-1,container_image=ceph/squid:v19.2.0,cpu=Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz,distro=centos,distro_description=CentOS Stream 9,distro_version=9,frontend_config#0=beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key,frontend_type#0=beast,hostname=raynor-sc-1,id=25069123.raynor-sc-1.qiatgr,kernel_description=#1 SMP Thu Dec 19 20:58:14 EST 2024,kernel_version=5.4.288-1.el8.elrepo.x86_64,mem_swap_kb=0,mem_total_kb=12251748,num_handles=1,os=Linux,pid=7,realm_id=c99973d7-27cb-4abd-8391-bfbe4fa928be,realm_name=geored_realm,zone_id=30816d1d-e315-471b-bc84-6cdc4f6f1408,zone_name=siteA,zonegroup_id=c109a213-6638-4028-b21b-8bc9fcb49cd3,zonegroup_name=geored_zg} Specifying the certificate by file path I think is the right choice for me as: * I don't have a domain, I have a per host certificate covering the IP of the host. Each is different, signed by a self-signed trusted CA. * I don't want to put the hardcoded key into the service spec, as suggested in the docs, as it appears in the mgr logs. My org policy (sensibly) doesn't allow for secrets to appear in logs. * It's just generally more hassle to manage keys on a per VM basis, rather than a static file path. * It will also allow me to not manage RGW services on a per host basis, but use the same spec for all and a count=1 placement, which would be lovely. So my question - is there a way around this such that I can specify the key and certificate by file path and not have to manually change the config to make it work? Thanks, Alex

2 10

Re: More objects misplaced than exist?
by Andre Tann 16 Jan '25

16 Jan '25

Hi Anthony, answering also to the list... Am 16.01.25 um 15:52 schrieb Anthony D'Atri: > When I see anomalous status my first thought is to manually failover the mgr I stopped the active mgr, another took over, but still the status is the same: data: volumes: 1/1 healthy pools: 4 pools, 2081 pgs objects: 3.53M objects, 12 TiB usage: 36 TiB used, 226 TiB / 262 TiB avail pgs: 17669564/10601739 objects misplaced (166.667%) 2081 active+clean+remapped What I still don't understand is: why does Ceph report usage 100% for all pools, which is nonsense? Can that be related to the confusing numbers of misplaces objects? root@pve01:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 262 TiB 226 TiB 36 TiB 36 TiB 13.58 TOTAL 262 TiB 226 TiB 36 TiB 36 TiB 13.58 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 7.6 MiB 3 15 MiB 100.00 0 B ReplicationPool 2 1024 8.0 TiB 2.11M 24 TiB 100.00 0 B cephfs_data 7 1024 3.9 TiB 1.43M 12 TiB 100.00 0 B cephfs_metadata 8 32 268 MiB 143 804 MiB 100.00 0 B -- Andre Tann

1 1

More objects misplaced than exist?
by Andre Tann 16 Jan '25

16 Jan '25

Hi all, # ceph -w ... volumes: 1/1 healthy pools: 4 pools, 2081 pgs objects: 3.72M objects, 12 TiB usage: 36 TiB used, 226 TiB / 262 TiB avail pgs: 18590764/11154459 objects misplaced (166.667%) 2081 active+clean+remapped How can more objects be misplaced than exist in the pools? Also ceph reports for all pools 100% usage. But also it shows that 35 out of 250 TB are used. So there is sth weird. Any ideas what could be wrong here? -- Andre Tann

1 0

issue with new AWS cli when upload: MissingContentLength
by Szabo, Istvan (Agoda) 16 Jan '25

16 Jan '25

Hi, Amazon released a new version of their cli today https://github.com/aws/aws-cli/tags and seems to break our stuffs with the following error when PUT object happens: bash-4.2# /usr/local/bin/aws --endpoint=https://endpoint --no-verify-ssl s3 cp online.txt s3://bucket/ upload failed: ./online.txt to s3://bucket/online.txt An error occurred (MissingContentLength) when calling the PutObject operation: Unknown Wonder is there anything on ceph side that I can do to eliminate this? The amazon workaround response_checksum_validation = when_required did not work. https://github.com/aws/aws-cli/issues/9214 Thank you

2 1

Ceph symbols for v15_2_0 in pacific libceph-common
by Frank Schilder 16 Jan '25

16 Jan '25

Hi all, during debugging of an MDS problem we observed something that looks odd. The output of perf top seems to show symbols from v15 (octopus) on a pacific (v16) installation: 23.56% ceph-mds [.] std::_Rb_tree<inodeno_t, inodeno_t, std::_Identity<inodeno_t> 7.02% libceph-common.so.2 [.] ceph::buffer::v15_2_0::ptr::copy_out 4.99% ceph-mds [.] std::_Hashtable<inodeno_t, std::pair<inodeno_t const, CInode* 4.69% libceph-common.so.2 [.] ceph::buffer::v15_2_0::list::iterator_impl<true>::copy 2.53% ceph-mds [.] std::_Rb_tree<SnapRealm*, SnapRealm*, std::_Identity<SnapReal 2.46% libtcmalloc.so.4.5.3 [.] tcmalloc::CentralFreeList::Populate 2.31% [kernel] [k] _raw_spin_lock 2.11% ceph-mds [.] std::_Rb_tree<dentry_key_t, std::pair<dentry_key_t const, CDe 1.90% libstdc++.so.6.0.25 [.] std::_Rb_tree_insert_and_rebalance 1.86% [kernel] [k] copy_user_enhanced_fast_string 1.83% libtcmalloc.so.4.5.3 [.] operator new[] 1.83% libceph-common.so.2 [.] ceph::buffer::v15_2_0::list::iterator_impl<true>::operator+= 1.68% ceph-mds [.] MDCache::populate_mydir 1.56% ceph-mds [.] std::_Rb_tree<frag_t, std::pair<frag_t const, CDir*>, std::_S 1.38% [kernel] [k] clear_page_erms 1.06% [kernel] [k] native_irq_return_iret The pacific packages are installed from download.ceph.com and all package candidates claim to be v16: # yum provides "/*/libceph-common.so.2" Last metadata expiration check: 0:45:57 ago on Wed 15 Jan 2025 12:14:09 PM CET. librados2-2:16.2.15-0.el8.x86_64 : RADOS distributed object store client library Repo : @System Matched from: Filename : /usr/lib64/ceph/libceph-common.so.2 librados2-2:16.2.15-0.el8.x86_64 : RADOS distributed object store client library Repo : ceph Matched from: Filename : /usr/lib64/ceph/libceph-common.so.2 Why do we see v15 symbols here or am I interpreting the symbol name ceph::buffer::v15_2_0::list::iterator_impl<true>::copy incorrectly? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 1

Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.
by bruno.pessanha＠gmail.com 16 Jan '25

16 Jan '25

Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ``` I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ``` I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ``` 1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again? Thank you! Bruno

6 16

Ceph User + Dev Meeting Information
by Laura Flores 15 Jan '25

15 Jan '25

Hi all, It was brought to my attention that some users were unaware that the old User + Dev meetup day changed from Thursdays to Wednesdays. To stay informed about future User + Dev meetings, you can: 1. Join the Meetup group here: https://www.meetup.com/ceph-user-group/, which will allow you to RSVP to upcoming events. 2. Download the Ceph Community Calendar, which is accessible here: https://ceph.io/en/community/meetups/ 3. Follow updates about upcoming topics, etc. for these meetings in the #ceph-user-dev-meetup Slack channel at this link: https://ceph-storage.slack.com/archives/C072CRU63H9 Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

MDS crashing on startup
by Frank Schilder 15 Jan '25

15 Jan '25

Hi Dan, hi all, this is related to the thread "Help needed, ceph fs down due to large stray dir". We deployed a bare metal host for debugging ceph daemon issues, here, to run "perf top" to find out where our MDS becomes unresponsive. Unfortunately, we encounter a strange issue: The bare-metal MDS crashes very quickly during the initial reconnect phase: -61> 2025-01-14T08:59:47.202-0500 7f676e519700 3 mds.2.server not active yet, waiting -60> 2025-01-14T08:59:47.202-0500 7f676e519700 5 mds.2.server dispatch request in up:reconnect: client_request(client.425250501:594886 lookup #0x3001059de1e/02JanParetoR esults_n5_a7_m3 2025-01-13T15:25:32.427929-0500 RETRY=6 caller_uid=315104, caller_gid=315104{}) v2 -59> 2025-01-14T08:59:47.202-0500 7f676e519700 3 mds.2.server not active yet, waiting -58> 2025-01-14T08:59:47.202-0500 7f6771d20700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MAC HINE_SIZE/gigantic/release/16.2.15/rpm/el8/BUILD/ceph-16.2.15/src/msg/async/AsyncMessenger.cc: In function 'void Processor::accept()' thread 7f6771d20700 time 2025-01-14T08: 59:47.200795-0500 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.15/rpm/el8/BUILD/ceph -16.2.15/src/msg/async/AsyncMessenger.cc: 214: ceph_abort_msg("abort() called") ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f6776f6e904] 2: (Processor::accept()+0x862) [0x7f6777261502] 3: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xcb7) [0x7f67772b6b87] 4: /usr/lib64/ceph/libceph-common.so.2(+0x5c90bc) [0x7f67772bd0bc] 5: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6775380b23] 6: /lib64/libpthread.so.0(+0x81ca) [0x7f6775f4b1ca] 7: clone() -57> 2025-01-14T08:59:47.202-0500 7f676e519700 0 log_channel(cluster) log [DBG] : reconnect by client.425250501 v1:192.168.57.60:0/1003106369 after 1.01409 ... -3> 2025-01-14T08:59:47.202-0500 7f676e519700 5 mds.2.server dispatch request in up:reconnect: client_request(client.42561 2912:251430 lookup #0x10000f5568d/util-linux 2025-01-11T17:43:34.212128-0500 RETRY=8 caller_uid=298337, caller_gid=298337{}) v2 -2> 2025-01-14T08:59:47.202-0500 7f676e519700 3 mds.2.server not active yet, waiting -1> 2025-01-14T08:59:47.202-0500 7f676e519700 0 log_channel(cluster) log [DBG] : reconnect by client.425612912 v1:192.168. 58.11:0/4294630612 after 1.01409 0> 2025-01-14T08:59:47.203-0500 7f6771d20700 -1 *** Caught signal (Aborted) ** in thread 7f6771d20700 thread_name:msgr-worker-0 ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) 1: /lib64/libpthread.so.0(+0x12d10) [0x7f6775f55d10] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator< char> > const&)+0x1b6) [0x7f6776f6e9d5] 5: (Processor::accept()+0x862) [0x7f6777261502] 6: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xcb7) [0x7 f67772b6b87] 7: /usr/lib64/ceph/libceph-common.so.2(+0x5c90bc) [0x7f67772bd0bc] 8: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6775380b23] 9: /lib64/libpthread.so.0(+0x81ca) [0x7f6775f4b1ca] 10: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. All other log messages look normal. The strange thing is that the bare metal and the containerized MDSes are the same version, yet the containerized daemon does *not* crash. Versions are: bare-metal# ceph-mds --version ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) container# ceph-mds --version ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) Both binaries have the same md5 sum as well. The only possibly relevant difference might be the kernel version: bare-metal: 4.18.0-553.34.1.el8_10.x86_64 containerized: 5.14.13-1.el7.elrepo.x86_64 I also installed all sorts of debuginfo packages. Still, this symbol is not resolved: 7: /usr/lib64/ceph/libceph-common.so.2(+0x5c90bc) [0x7f67772bd0bc] Which package is it in? I did install ceph-common-debuginfo-16.2.15-0.el8.x86_64.rpm . For installing the MDS we followed the instructions for manual start here: https://docs.ceph.com/en/pacific/install/manual-deployment/#adding-mds . Thanks for any pointers! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 6

Multi Active MDS on old kernel client (<4.14)
by Jesse Galley 15 Jan '25

15 Jan '25

Hello! The documentation on kernel client features states: "The feature has been supported since the Luminous release. It is recommended to use Linux kernel clients >= 4.14 when there are multiple active MDS." My question is: what is the actual impact of using an old (3.10) client with a multi-mds FS? I used a 3.10 client to do some reads and writes of metadata pinned to different MDSs, and it all ostensibly worked fine. Is it purely a performance problem? Or is there a more serious issue lurking that my simple tests would not reveal. Thanks! *Jesse Galley* *Director, Mail Platform & Migrations, Hostopia* *C: 416.816.0866* *jesse.galley(a)hostpapa.com <jesse.galley(a)hostpapa.com>*

2 2

Re: [ceph-users] Installing Ceph on ARM fails
by filip Mutterer 14 Jan '25

14 Jan '25

Its Debian 12 On Tuesday, January 14, 2025 19:34 CET, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Which OS? > > > On Jan 14, 2025, at 1:29 PM, filip Mutterer <filip(a)3dds.io> wrote: > > > > hi, > > > > trying to install Ceph 19.2 on ARM. > > > > My commands are: > > > > for i in 192.168.1.{11,12,13};do ssh $i "apt install -y chrony curl";done > > for i in 192.168.1.{11,12,13};do ssh $i \ > > "curl --silent --remote-name --locationhttps://download.ceph.com/rpm-19.2.0/el9/noarch/cephadm && chmod +x cephadm";done > > for i in 192.168.1.{11,12,13};do ssh $i "./cephadm add-repo --release squid && ./cephadm install";done > > > > So far no problem, but if I want to check the version "/usr/sbin/cephadm version" i am getting no result, it just hangs. > > > > Ignoring this and continue with the installation: > > > > apt-get install -y ceph-common > > > > Warning: Permanently added '192.168.1.13' (ED25519) to the list of known hosts. > > Installing packages ['ceph-common']... > > Non-zero exit code 100 from apt-get install -y ceph-common > > apt-get: stdout Reading package lists... > > apt-get: stdout Building dependency tree... > > apt-get: stdout Reading state information... > > apt-get: stdout Some packages could not be installed. This may mean that you have > > apt-get: stdout requested an impossible situation or if you are using the unstable > > apt-get: stdout distribution that some required packages have not yet been created > > apt-get: stdout or been moved out of Incoming. > > apt-get: stdout The following information may help to resolve the situation: > > apt-get: stdout > > apt-get: stdout The following packages have unmet dependencies: > > apt-get: stdout ceph-common : Depends: python3-ceph-argparse (= 16.2.15+ds-0+deb12u1) but 19.2.0-1~bpo12+1 is to be installed > > apt-get: stdout Depends: python3-ceph-common (< 16.2.15+ds-0+deb12u1.1~) but 19.2.0-1~bpo12+1 is to be installed > > apt-get: stdout Depends: python3-cephfs (= 16.2.15+ds-0+deb12u1) but it is not going to be installed > > apt-get: stderr E: Unable to correct problems, you have held broken packages. > > Traceback (most recent call last): > > File "<frozen runpy>", line 198, in _run_module_as_main > > File "<frozen runpy>", line 88, in _run_code > > File "/root/./cephadm/__main__.py", line 5579, in <module> > > File "/root/./cephadm/__main__.py", line 5567, in main > > File "/root/./cephadm/__main__.py", line 4586, in command_install > > File "/root/./cephadm/cephadmlib/packagers.py", line 209, in install > > File "/root/./cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws > > RuntimeError: Failed command: apt-get install -y ceph-common: E: Unable to correct problems, you have held broken packages. > > > > On my x86 system this step was different, it just installed version 19.2.0 > > > > How can I get the version wanted? > > > > > > Greetings filip > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Installing Ceph on ARM fails
by filip Mutterer 14 Jan '25

14 Jan '25

hi, trying to install Ceph 19.2 on ARM. My commands are: for i in 192.168.1.{11,12,13};do ssh $i "apt install -y chrony curl";done for i in 192.168.1.{11,12,13};do ssh $i \ "curl --silent --remote-name --locationhttps://download.ceph.com/rpm-19.2.0/el9/noarch/cephadm && chmod +x cephadm";done for i in 192.168.1.{11,12,13};do ssh $i "./cephadm add-repo --release squid && ./cephadm install";done So far no problem, but if I want to check the version "/usr/sbin/cephadm version" i am getting no result, it just hangs. Ignoring this and continue with the installation: apt-get install -y ceph-common Warning: Permanently added '192.168.1.13' (ED25519) to the list of known hosts. Installing packages ['ceph-common']... Non-zero exit code 100 from apt-get install -y ceph-common apt-get: stdout Reading package lists... apt-get: stdout Building dependency tree... apt-get: stdout Reading state information... apt-get: stdout Some packages could not be installed. This may mean that you have apt-get: stdout requested an impossible situation or if you are using the unstable apt-get: stdout distribution that some required packages have not yet been created apt-get: stdout or been moved out of Incoming. apt-get: stdout The following information may help to resolve the situation: apt-get: stdout apt-get: stdout The following packages have unmet dependencies: apt-get: stdout ceph-common : Depends: python3-ceph-argparse (= 16.2.15+ds-0+deb12u1) but 19.2.0-1~bpo12+1 is to be installed apt-get: stdout Depends: python3-ceph-common (< 16.2.15+ds-0+deb12u1.1~) but 19.2.0-1~bpo12+1 is to be installed apt-get: stdout Depends: python3-cephfs (= 16.2.15+ds-0+deb12u1) but it is not going to be installed apt-get: stderr E: Unable to correct problems, you have held broken packages. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/./cephadm/__main__.py", line 5579, in <module> File "/root/./cephadm/__main__.py", line 5567, in main File "/root/./cephadm/__main__.py", line 4586, in command_install File "/root/./cephadm/cephadmlib/packagers.py", line 209, in install File "/root/./cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: apt-get install -y ceph-common: E: Unable to correct problems, you have held broken packages. On my x86 system this step was different, it just installed version 19.2.0 How can I get the version wanted? Greetings filip

1 0

Modify or override ceph_default_alerts.yml
by Devin A. Bougie 14 Jan '25

14 Jan '25

Hi, All. We are using cephadm to manage a 19.2.0 cluster on fully-updated AlmaLinux 9 hosts, and would greatly appreciate help modifying or overriding the alert rules in ceph_default_alerts.yml. Is the best option to simply update the /var/lib/ceph/<cluster_id>/home/ceph_default_alerts.yml file? In particular, we’d like to either disable the CephPGImbalance alert or change it to calculate averages per-pool or per-crush_rule instead of globally as in [1]. We currently have PG autoscaling enabled, and have two separate crush_rules (one with large spinning disks, one with much smaller nvme drives). Although I don’t believe it causes any technical issues with our configuration, our dashboard is full of CephPGImbalance alerts that would be nice to clean up without having to create periodic silences. Any help or suggestions would be greatly appreciated. Many thanks, Devin [1] https://github.com/rook/rook/discussions/13126#discussioncomment-10043490

3 14

Understanding filesystem size
by Nicola Mori 14 Jan '25

14 Jan '25

Dear Ceph users, I'd need some help with CephFS. Originally, my cluster consisted of 12 hosts with different total raw disk size, ranging from 16 to 32 TB; there were two pools, one 3x replicated metadata and one 6+2 EC data with host failure domain, and which I access via CephFS. In total there were 512 PGs. Naively, I would expect that the available size of the data pool would be about 75% of the total available space given the EC parameters (6/(6+2) = 75%), but I noticed that it was actually lower, around 66% (as reported by df -h, while for the total raw capacity I used the one reported by the Ceph dashboard). I guessed this could be due to smaller hosts limiting the total amount of available space for the filesystem, so I started a disk upgrade campaign to make the host sizes more even. For example, for a 16 TB host with 8x2TB I replaced two disks with 8 TB ones, for a new raw size of 28 TB. I did this and similar upgrades for other hosts in steps, every time waiting for backfill to finish and checking the new filesystem size before proceeding. And I noticed that the filesystem size reported by df -h always remained at 66% of the total raw capacity, despite host sizes are now all between 24 and 32 TB. So my guess about the limiting factor for the relative fs size is evidently wrong. I thought that for some reason maybe 66% is the actual limit and also about some possible explanation (e.g. some space being reserved for the metadata pool), and even considered the possibility that the size reported by df -h being meaningless, but I'd need some advice from an expert to sort this out. Thanks in advance for any help. Nicola

5 27

Issue with CopyObject version 18.2.4: Copied objects are not deleted below the pool
by tranthithuan180693＠gmail.com 14 Jan '25

14 Jan '25

I tried uploading without copying, the obj still deleted normally When using copy, obj cannot be deleted under the pool for storage-class COLD Here's how I do it: I created storage-class COLD radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id default-placement --storage-class COLD radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class COLD --data-pool default.rgw.cold.data and restart service rgw Create bucket name hn3 radosgw-admin bucket stats --bucket hn3 { "bucket": "hn3", "num_shards": 11, "tenant": "", "zonegroup": "39c9f30a-42dd-4037-8ef7-02942f6bfcc7", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3", "marker": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3", "index_type": "Normal", "versioned": false, "versioning_enabled": false, "object_lock_enabled": false, "mfa_enabled": false, "owner": "test", "ver": "0#1,1#1,2#1,3#1,4#4,5#1,6#1,7#1,8#1,9#1,10#1", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "2025-01-08T08:44:50.831879Z", "creation_time": "2025-01-08T08:44:50.804023Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": { "rgw.main": { "size": 37836, "size_actual": 40960, "size_utilized": 37836, "size_kb": 37, "size_kb_actual": 40, "size_kb_utilized": 37, "num_objects": 1 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } I upload any file with storage-class COLD I use mc (MinIO Client): mc cp test.txt s3/hn3/ Then copy that file with a different name using storage-class cold mc cp s3/hn3/test.txt s3/hn3/test.txt2 Check obj in pool default.rgw.cold.data rados -p default.rgw.cold.data ls | grep ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3 ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.ksiViAsAWgM7uyfn68yJiDNjLEXRCVc_0 Now i will delete this obj mc rm s3/hn3/test.txt mc rm s3/hn3/test.txt2 Check garbage collection radosgw-admin gc list --include-all | grep ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3 "tag": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.wubPUKz-g1B8XdzfiNXWM_rrD_oQeqM_1_01863ccdb47a110b5af24277107e76a9\u0000", "oid": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.wubPUKz-g1B8XdzfiNXWM_rrD_oQeqM_0", "oid": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.RWuk6chwz7Pkd8jvkBqY_eUibNDHnIq_0", Run manual gc radosgw-admin gc process --include-all Check again obj in pool , obj is not deleted rados -p default.rgw.cold.data ls | grep ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3 ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.ksiViAsAWgM7uyfn68yJiDNjLEXRCVc_0

2 1

Any idea why misplace recovery wont finish?
by Ml Ml 14 Jan '25

14 Jan '25

Hello List, i have this 3 node Setup with 17 hdds (new - spinning rust). After putting it all together and after resetting the manual Rack-Placements back to default it seems to recover forever. I also changed the pg placements at some time. The disks where very busy until this morning when i disabled scrubbing to see if this would speed up the recovery. Here is my current status: ---------------------------------------- root@ceph12:~# ceph -s cluster: id: 2ed9f8fb-1316-4ef1-996d-a3223a3dd594 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set services: mon: 3 daemons, quorum ceph10,ceph11,ceph12 (age 2h) mgr: ceph11(active, since 3h), standbys: ceph10, ceph12 osd: 17 osds: 17 up (since 3h), 17 in (since 11d); 18 remapped pgs flags noscrub,nodeep-scrub data: pools: 2 pools, 257 pgs objects: 9.62M objects, 37 TiB usage: 110 TiB used, 134 TiB / 244 TiB avail pgs: 1534565/28857846 objects misplaced (5.318%) 239 active+clean 16 active+remapped+backfill_wait 2 active+remapped+backfilling io: recovery: 25 MiB/s, 6 objects/s I set: ceph tell 'osd.*' injectargs --osd-max-backfills=6 --osd-recovery-max-active=9 My hourly recovery process seems not to really proceed (it decreeses and increases again): 5.410 5.036 5.269 5.008 5.373 5.769 5.555 5.103 5.067 5.135 5.409 5.417 5.373 5.197 5.090 5.458 5.204 5.339 5.164 5.425 5.692 5.383 5.726 5.492 6.694 6.576 6.362 6.243 6.011 5.880 5.589 5.433 5.846 5.378 5.184 5.647 5.374 5.513 root@ceph12:~# ceph osd perf (this is okay for spinning rust i guess) osd commit_latency(ms) apply_latency(ms) 17 0 0 13 0 0 11 57 57 9 4 4 0 0 0 1 79 79 14 43 43 2 0 0 3 0 0 16 43 43 4 33 33 5 0 0 6 0 0 7 0 0 10 0 0 12 0 0 8 48 48 iostat -xt 3 shows that they are busy but its not overloaded as i can tell. Any idea why it will only backfill 2? How could i speed this up? My other cluster with 7 nodes and very old hdds and 45 OSDs can make easily 76 objects/s Cheers, Michael

5 4

Snaptriming speed degrade with pg increase
by Szabo, Istvan (Agoda) 14 Jan '25

14 Jan '25

Hi, When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04) -osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2 We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours. Is there any good solution that we are missing to fix this? On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still. Thank you

4 9

Re: OSDs won't come back after upgrade
by Jorge Garcia 13 Jan '25

13 Jan '25

Actually, stupid mistake on my part. I had selinux mode as enforcing. Changed it to disabled, and everything works again. Thanks for the help!

2 2

Ceph orch commands failing with Error ENOENT: Module not found
by Frank Frampton 13 Jan '25

13 Jan '25

Recent upgrade from 18.2 to 19.2, upgrade went fine. Since the upgrade and a manager fail over, I can no longer run orchestrator commands. The only error I can find on an active manager daemon is the following, or it is the only one that stands out. 2025-01-06T18:48:41.698+0000 7fcf42b99640 -1 mgr load Failed to construct class in 'cephadm' 2025-01-06T18:48:41.698+0000 7fcf42b99640 -1 mgr load Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 667, in __init__ self.cert_key_store.load() File "/usr/share/ceph/mgr/cephadm/inventory.py", line 2073, in load self.known_certs[entity] = json.loads(v) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2025-01-06T18:48:41.702+0000 7fcf42b99640 -1 mgr operator() Failed to run module in active mode ('cephadm') For any thing to really work in the dashboard I must have the balancer off. While the balance is off I can make changes in dashboard to the orchestrator, and it doesn't give me any trouble. When trying different commands from a ceph node say "ceph cephadm config-check status" it returns "Error ENOTSUP: Module 'cephadm' is not enabled/loaded (required by command 'cephadm config-check status'): use `ceph mgr module enable cephadm` to enable it". Running "ceph mgr module enable cephadm" returns "module 'cephadm' is already enabled". I really don't know where to look or what to try to resolve this. Any "ceph orch" command results in "Error ENOENT: Module not found" I don't know that my issue is related to https://tracker.ceph.com/issues/68657, but maybe it is. I have tried the following. Manually adding a new mgr daemon on different node, it starts runs the dashboard fine, but things are still not functional. Failed the mgr several times. Disabled/Enabled balancer. Disabled/Enabled mgr modules. Disabled/Enabled dashboard. All physical nodes are running Debian 12. Frank Frampton Senior Network Services Administrator Salt Lake City School District Desk: (801) 578-8223 Follow the district: Facebook<https://www.facebook.com/slcschools> | Instagram<https://instagram.com/slcschools> | Twitter<https://twitter.com/slcschools> | www.slcschools.org<http://www.slcschools.org/> Excellence and Equity: every student, every classroom, every day Scanned By Microsoft EOP

2 2

Enterprise SSD/NVME
by Martin Konold 13 Jan '25

13 Jan '25

3 2

Help in recreating a old ceph cluster
by Jayant Dang 13 Jan '25

13 Jan '25

Hi Can someone help me understand what happens in a scenario if my os disk of all the ceph nodes gets destroyed somehow and I am only left with the OSDs or physical storage devices, How can I recreate the same ceph cluster using those old OSDs without any data loss ? Is there something I should regularly backup to recreate a ceph cluster ?

2 1

Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience
by Anthony D'Atri 13 Jan '25

13 Jan '25

[ ed: snag during moderation (somehow a newline was interpolated in the Subject), so I’m sending this on behalf of kasper_steengaard(a)hotmail.com <mailto:kasper_steengaard@hotmail.com> , to whom replies should be sent] I'm managing a ceph cluster with +1K OSDs distributed accross 56 host. Until now the crush rule used is the default replicated rule, but I want to change that in order to implement failure domain on rack level. Current plan is to - Disable rebalancing by executing - ceph osd set norebalance; - Add Rack to crushmap and distribute the hosts accordingly (8 in each.) by using the built in commands - ceph osd crush add-bucket rack1 rack root=default - ceph osd crush move osd-host1 rack=rack1 - Create the new rack split rule with command - ceph osd crush rule create-replicated rack_split default rack - Set the rule across all my pools - for p in $(ceph osd lspools | cut -d' ' -f 2) ; do echo $p $(ceph osd pool set $p crush_rule rack_split) ; done - Finally enable rebalancing - ceph osd unset norebalance; However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who have been in similar situations. —— [ ed: You only include commands for one CRUSH `rack` — would you create multiple `rack` CRUSH buckets, at least three of them? Are all of your pools replicated? No EC pools for RGW buckets, CephFS data, etc? What OSD media and networking does this cluster have? HDDs will be much slower and much more impacted during the process than SSDs. Is your client workload 24x7? Which Ceph release? These factors inform how impactful the grand shuffle will be. Are your mon DBs on SSDs? A popular strategy is to use upmap-remapped.py to freeze all of the PG mappings before unsetting the norebalance flag, then the balancer will gradually undo the mappings as it moves data to where it now belongs. This process has built-in throttling. ]

3 2

Most OSDs down and all PGs unknown after P2V migration
by leo＠mylonas.co 13 Jan '25

13 Jan '25

I run a small single-node Ceph deployment (had plans to scale up once I got more equipment) for home file storage deployed using cephadm. It was running bare-metal, and I attempted a physical-to-virtual migration to a Proxmox VM. After doing so, all of my PGs seemed to be "unknown". Initial after a boot, the OSDs appear to be up, but after a while, most of them go down. I assume some sort of timeout in the OSD start process. The systemd processes (and podman containers) are still running and appear to be happy. I don't see anything crazy in their logs. I'm relatively new to Ceph, so I don't really know where to go from here. Can anyone provide any guidance? The logs for the monitor and one of the OSDs can be found here - https://gitlab.com/-/snippets/4793143 And here is the output from a few commands that might be useful. ceph -s ``` cluster: id: 768819b0-a83f-11ee-81d6-74563c5bfc7b health: HEALTH_WARN Reduced data availability: 545 pgs inactive 139 pgs not deep-scrubbed in time 17 slow ops, oldest one blocked for 1668 sec, mon.fileserver has slow ops services: mon: 1 daemons, quorum fileserver (age 28m) mgr: fileserver.rgtdvr(active, since 28m), standbys: fileserver.gikddq osd: 17 osds: 5 up (since 116m), 5 in (since 10m) data: pools: 3 pools, 545 pgs objects: 1.97M objects, 7.5 TiB usage: 7.7 TiB used, 1.4 TiB / 9.1 TiB avail pgs: 100.000% pgs unknown 545 unknown ``` ceph osd df ``` ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 1 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 3 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 112 down 4 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 117 down 5 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 6 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 7 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 8 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 106 down 20 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 115 down 21 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 94 down 22 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 98 down 23 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 109 down 24 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 4 KiB 3.0 GiB 186 GiB 90.00 1.06 117 up 25 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 10 KiB 2.8 GiB 220 GiB 88.18 1.04 114 up 26 hdd 1.81940 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 9 KiB 2.8 GiB 297 GiB 84.07 0.99 109 up 27 hdd 1.81940 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 7 KiB 2.5 GiB 474 GiB 74.58 0.88 98 up 28 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 10 KiB 3.0 GiB 206 GiB 88.93 1.04 115 up TOTAL 9.1 TiB 7.7 TiB 7.7 TiB 42 KiB 14 GiB 1.4 TiB 85.15 MIN/MAX VAR: 0.88/1.06 STDDEV: 5.65 ``` ceph pg stat ``` 545 pgs: 545 unknown; 7.5 TiB data, 7.7 TiB used, 1.4 TiB / 9.1 TiB avail ``` systemctl | grep ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b ``` ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)alertmanager.fileserver.serv ice loaded active running Ceph alertmanager.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)ceph-exporter.fileserver.ser vice loaded active running Ceph ceph-exporter.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)crash.fileserver.service loaded active running Ceph crash.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)grafana.fileserver.service loaded active running Ceph grafana.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mgr.fileserver.gikddq.servic e loaded active running Ceph mgr.fileserver.gikddq for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mgr.fileserver.rgtdvr.servic e loaded active running Ceph mgr.fileserver.rgtdvr for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mon.fileserver.service loaded active running Ceph mon.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.0.service loaded active running Ceph osd.0 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.1.service loaded active running Ceph osd.1 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.20.service loaded active running Ceph osd.20 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.21.service loaded active running Ceph osd.21 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.22.service loaded active running Ceph osd.22 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.23.service loaded active running Ceph osd.23 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.24.service loaded active running Ceph osd.24 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.25.service loaded active running Ceph osd.25 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.26.service loaded active running Ceph osd.26 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.27.service loaded active running Ceph osd.27 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.28.service loaded active running Ceph osd.28 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.3.service loaded active running Ceph osd.3 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.4.service loaded active running Ceph osd.4 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.5.service loaded active running Ceph osd.5 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.6.service loaded active running Ceph osd.6 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.7.service loaded active running Ceph osd.7 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.8.service loaded active running Ceph osd.8 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)prometheus.fileserver.servic e loaded active running Ceph prometheus.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b system-ceph\x2d768819b0\x2da83f\x2d11ee\x2d81d6\x2d74563c5bfc7b.slice loaded active active Slice /system/ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b.target loaded active active Ceph cluster 768819b0-a83f-11ee-81d6-74563c5bfc7b ```

1 0

Most OSDs down and all PGs unknown after P2V migration
by leo＠mylonas.co 12 Jan '25

12 Jan '25

I run a small single-node ceph cluster (the plan was to scale out when I could get more equipment) for home file storage deployed by cephadm. It was running bare-metal, and I attempted a physical-to-virtual migration to a Proxmox VM After doing so, all of my PGs seemed to be "unknown". Initial after a boot, the OSDs appear to be up, but after a while, they go down. I assume some sort of timeout in the OSD start process. The systemd processes (and podman containers) are still running and appear to be happy. I don't see anything that jumps out at me in their logs. I'm relatively new to Ceph, so I don't really know where to go from here. Can anyone provide any guidance? **ceph -s** ``` cluster: id: 768819b0-a83f-11ee-81d6-74563c5bfc7b health: HEALTH_WARN Reduced data availability: 545 pgs inactive 139 pgs not deep-scrubbed in time 17 slow ops, oldest one blocked for 1668 sec, mon.fileserver has slow ops services: mon: 1 daemons, quorum fileserver (age 28m) mgr: fileserver.rgtdvr(active, since 28m), standbys: fileserver.gikddq osd: 17 osds: 5 up (since 116m), 5 in (since 10m) data: pools: 3 pools, 545 pgs objects: 1.97M objects, 7.5 TiB usage: 7.7 TiB used, 1.4 TiB / 9.1 TiB avail pgs: 100.000% pgs unknown 545 unknown ``` **ceph osd df** ``` ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 1 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 3 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 112 down 4 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 117 down 5 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 6 hdd 3.63869 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 7 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 8 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 106 down 20 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 115 down 21 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 94 down 22 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 98 down 23 hdd 1.81940 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 109 down 24 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 4 KiB 3.0 GiB 186 GiB 90.00 1.06 117 up 25 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 10 KiB 2.8 GiB 220 GiB 88.18 1.04 114 up 26 hdd 1.81940 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 9 KiB 2.8 GiB 297 GiB 84.07 0.99 109 up 27 hdd 1.81940 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 7 KiB 2.5 GiB 474 GiB 74.58 0.88 98 up 28 hdd 1.81940 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 10 KiB 3.0 GiB 206 GiB 88.93 1.04 115 up TOTAL 9.1 TiB 7.7 TiB 7.7 TiB 42 KiB 14 GiB 1.4 TiB 85.15 MIN/MAX VAR: 0.88/1.06 STDDEV: 5.65 ``` **ceph pg stat** ``` 545 pgs: 545 unknown; 7.5 TiB data, 7.7 TiB used, 1.4 TiB / 9.1 TiB avail ``` **systemctl | grep ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b** ``` ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)alertmanager.fileserver.service loaded active running Ceph alertmanager.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)ceph-exporter.fileserver.service loaded active running Ceph ceph-exporter.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)crash.fileserver.service loaded active running Ceph crash.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)grafana.fileserver.service loaded active running Ceph grafana.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mgr.fileserver.gikddq.service loaded active running Ceph mgr.fileserver.gikddq for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mgr.fileserver.rgtdvr.service loaded active running Ceph mgr.fileserver.rgtdvr for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)mon.fileserver.service loaded active running Ceph mon.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.0.service loaded active running Ceph osd.0 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.1.service loaded active running Ceph osd.1 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.20.service loaded active running Ceph osd.20 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.21.service loaded active running Ceph osd.21 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.22.service loaded active running Ceph osd.22 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.23.service loaded active running Ceph osd.23 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.24.service loaded active running Ceph osd.24 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.25.service loaded active running Ceph osd.25 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.26.service loaded active running Ceph osd.26 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.27.service loaded active running Ceph osd.27 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.28.service loaded active running Ceph osd.28 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.3.service loaded active running Ceph osd.3 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.4.service loaded active running Ceph osd.4 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.5.service loaded active running Ceph osd.5 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.6.service loaded active running Ceph osd.6 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.7.service loaded active running Ceph osd.7 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)osd.8.service loaded active running Ceph osd.8 for 768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b(a)prometheus.fileserver.service loaded active running Ceph prometheus.fileserver for 768819b0-a83f-11ee-81d6-74563c5bfc7b system-ceph\x2d768819b0\x2da83f\x2d11ee\x2d81d6\x2d74563c5bfc7b.slice loaded active active Slice /system/ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b ceph-768819b0-a83f-11ee-81d6-74563c5bfc7b.target loaded active active Ceph cluster 768819b0-a83f-11ee-81d6-74563c5bfc7b ``` The logs for `mon` and `osd.3` can be found here - https://gitlab.com/-/snippets/4793143

1 0

MDSs report oversized cache during forward scrub
by Frank Schilder 10 Jan '25

10 Jan '25

Hi all, we started a forward scrub on our 5.x PB ceph file system and observe a massive ballooning of MDS caches. Our status is: # ceph status cluster: id: xxx health: HEALTH_WARN 1 MDSs report oversized cache (muted: MDS_CLIENT_LATE_RELEASE(12d) MDS_CLIENT_RECALL(12d) PG_NOT_DEEP_SCRUBBED(5d) PG_NOT_SCRUBBED(5d) POOL_NEAR_FULL(4w)) services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 4w) mgr: ceph-03(active, since 3w), standbys: ceph-01, ceph-25, ceph-26, ceph-02 mds: 8/8 daemons up, 4 standby osd: 1317 osds: 1316 up (since 5d), 1316 in (since 5w) task status: scrub status: mds.ceph-08: active paths [/] mds.ceph-11: active paths [/] mds.ceph-12: active paths [/] mds.ceph-14: active paths [/] mds.ceph-15: active paths [/] mds.ceph-17: active paths [/] mds.ceph-24: active paths [/] data: volumes: 1/1 healthy pools: 14 pools, 29161 pgs objects: 4.38G objects, 5.5 PiB usage: 7.2 PiB used, 5.9 PiB / 13 PiB avail pgs: 29129 active+clean 28 active+clean+scrubbing+deep 2 active+clean+snaptrim 2 active+clean+scrubbing io: client: 484 MiB/s rd, 64 MiB/s wr, 4.64k op/s rd, 1.50k op/s wr # ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-12 Reqs: 3 /s 13.8M 13.8M 478k 29.2k 1 active ceph-15 Reqs: 31 /s 10.5M 10.5M 140k 441k 2 active ceph-14 Reqs: 0 /s 11.5M 11.5M 504k 32.8k 3 active ceph-17 Reqs: 5 /s 12.4M 12.4M 487k 30.9k 4 active ceph-08 Reqs: 0 /s 15.3M 15.3M 247k 47.2k 5 active ceph-11 Reqs: 7 /s 4414k 4413k 262k 72.5k 6 active ceph-16 Reqs: 409 /s 1079k 1057k 7766 17.3k 7 active ceph-24 Reqs: 41 /s 4074k 4074k 448k 109k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 6657G con-fs2-meta2 data 0 6657G con-fs2-data data 1225T 2272T con-fs2-data-ec-ssd data 794G 20.8T con-fs2-data2 data 5745T 2066T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-13 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) Ranks 0-4 are massively oversized. Ranks 5 and 7 show usual values. Rank 6 is low, because we restarted it already due to this warning. It seems as if MDSes don't trim cache during scrub, the sizes only increase. Is this ballooning normal or a bug? Is there a workaround apart from restarting MDSes all the time? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

MDSs report oversized cache during forward scrub
by Frank Schilder 10 Jan '25

10 Jan '25

Hi all, we started a forward scrub on our 5.x PB ceph file system and observe a massive ballooning of MDS caches. Our status is: # ceph status cluster: id: xxx health: HEALTH_WARN 1 MDSs report oversized cache (muted: MDS_CLIENT_LATE_RELEASE(12d) MDS_CLIENT_RECALL(12d) PG_NOT_DEEP_SCRUBBED(5d) PG_NOT_SCRUBBED(5d) POOL_NEAR_FULL(4w)) services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 4w) mgr: ceph-03(active, since 3w), standbys: ceph-01, ceph-25, ceph-26, ceph-02 mds: 8/8 daemons up, 4 standby osd: 1317 osds: 1316 up (since 5d), 1316 in (since 5w) task status: scrub status: mds.ceph-08: active paths [/] mds.ceph-11: active paths [/] mds.ceph-12: active paths [/] mds.ceph-14: active paths [/] mds.ceph-15: active paths [/] mds.ceph-17: active paths [/] mds.ceph-24: active paths [/] data: volumes: 1/1 healthy pools: 14 pools, 29161 pgs objects: 4.38G objects, 5.5 PiB usage: 7.2 PiB used, 5.9 PiB / 13 PiB avail pgs: 29129 active+clean 28 active+clean+scrubbing+deep 2 active+clean+snaptrim 2 active+clean+scrubbing io: client: 484 MiB/s rd, 64 MiB/s wr, 4.64k op/s rd, 1.50k op/s wr # ceph fs status con-fs2 - 1554 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-12 Reqs: 3 /s 13.8M 13.8M 478k 29.2k 1 active ceph-15 Reqs: 31 /s 10.5M 10.5M 140k 441k 2 active ceph-14 Reqs: 0 /s 11.5M 11.5M 504k 32.8k 3 active ceph-17 Reqs: 5 /s 12.4M 12.4M 487k 30.9k 4 active ceph-08 Reqs: 0 /s 15.3M 15.3M 247k 47.2k 5 active ceph-11 Reqs: 7 /s 4414k 4413k 262k 72.5k 6 active ceph-16 Reqs: 409 /s 1079k 1057k 7766 17.3k 7 active ceph-24 Reqs: 41 /s 4074k 4074k 448k 109k POOL TYPE USED AVAIL con-fs2-meta1 metadata 4078G 6657G con-fs2-meta2 data 0 6657G con-fs2-data data 1225T 2272T con-fs2-data-ec-ssd data 794G 20.8T con-fs2-data2 data 5745T 2066T STANDBY MDS ceph-09 ceph-10 ceph-23 ceph-13 MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) Ranks 0-4 are massively oversized. Ranks 5 and 7 show usual values. Rank 6 is low, because we restarted it already due to this warning. It seems as if MDSes don't trim cache during scrub, the sizes only increase. Is this ballooning normal or a bug? Is there a workaround apart from restarting MDSes all the time? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Per-Client Quality of Service settings
by Olaf Seibert 10 Jan '25

10 Jan '25

Hi! I am trying to find if Ceph has any QoS settings that apply per-client. I would like to be able, say, to have different QoS settings for RBD clients named "nova" and "cinder" and different again from an RGW client named "enduser". While researching this, I found amongst other things, the following references, which do not appear to be what I want: - https://docs.ceph.com/en/reef/dev/osd_internals/mclock_wpq_cmp_study/ and https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#dmclock-q… . This appears to be instead about the balancing of user-iops against ceph-internal iops, not balancing clients against each other. (aside: somebody should rewrite this paragraph, I find it totally incomprehensible: https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#subtletie… ) - https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#qos-settings and https://tracker.ceph.com/issues/36191 This seems to be about setting limits on the RBD pools as a whole, or on separate images inside them. However separate clients each still get to use up to the limits. Is there anything else that I have not found so far but which is about balancing individual clients across all services? -- Olaf Seibert Site Reliability Engineer SysEleven GmbH Boxhagener Straße 80 10245 Berlin T +49 30 233 2012 0 F +49 30 616 7555 0 https://www.syseleven.de https://www.linkedin.com/company/syseleven-gmbh/ Current system status always at: https://www.syseleven-status.net/ Company headquarters: Berlin Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens Plogsties

2 2

Ceph Orchestrator ignores attribute filters for SSDs
by Janek Bevendorff 10 Jan '25

10 Jan '25

Hi, I'm having a strange problem with the orchestrator. My cluster has the following OSD services configured based on certain attributes of the disks: NAME PORTS RUNNING REFRESHED AGE PLACEMENT ... osd.osd-default-hdd 1351 2m ago 22m label:osd;HOSTPREFIX* osd.osd-default-ssd 0 - 22m label:osd;HOSTPREFIX* osd.osd-small-hdd 41 2m ago 22m label:osd;HOSTPREFIX* These apply to three device types: large HDDs (8TB+), small HDDs (250G-7TB), and SSDs (1TB+). I did that with the following YAML definition: service_type: osd service_id: osd-default-hdd service_name: osd.osd-default-hdd placement: host_pattern: HOSTPREFIX* label: osd spec: crush_device_class: hdd data_devices: rotational: 1 size: '8T:' filter_logic: AND objectstore: bluestore osds_per_device: 1 --- service_type: osd service_id: osd-default-ssd service_name: osd.osd-default-ssd placement: host_pattern: HOSTPREFIX* label: osd spec: crush_device_class: ssd data_devices: rotational: 0 size: '1T:' filter_logic: AND objectstore: bluestore osds_per_device: 1 --- service_type: osd service_id: osd-small-hdd service_name: osd.osd-small-hdd placement: host_pattern: HOSTPREFIX* label: osd spec: crush_device_class: hdd-small data_devices: rotational: 1 size: 250G:7T filter_logic: AND objectstore: bluestore osds_per_device: 1 Previously, this worked perfectly, but as you can see in the summary above, now the orchestrator suddenly started to ignore the device class and data_devices filters for SSDs and incorrectly added all SSDs to the osd.osd-default-hdd service (weirdly enough, hdd-small still works). The affected devices still have the correct device class in the CRUSH tree and it also looks like the data placement is fine. The orchestrator service listing, however, is incorrect. I tried cleaning out and freshly redeploying one of the SSD OSDs, but the redeployed service still has the following in the unit.meta file: { "service_name": "osd.osd-default-hdd", "ports": [], "ip": null, "deployed_by": [ "quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77", "quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906" ], "rank": null, "rank_generation": null, "extra_container_args": null, "extra_entrypoint_args": null, "memory_request": null, "memory_limit": null } Any idea what might be causing this? I'm on Ceph 18.2.4 (upgrade planned, but I need to wait out some remapped PGs first). Janek

2 1

ceph tell throws WARN: the service id you provided does not exist.
by Frank Schilder 10 Jan '25

10 Jan '25

Hi all, following the documentation for initiating a file system scrub (pacific: https://docs.ceph.com/en/pacific/cephfs/scrub/) I can't get the syntax with "mds.<fsname>" to work: # ceph tell mds.con-fs2:0 help WARN: the service id you provided does not exist. service id should be one of ceph-09/ceph-10/ceph-23/ceph-16/ceph-24/ceph-08/ceph-17/ceph-12/ceph-13/ceph-15/ceph-11/ceph-14. target mds.con-fs2:0 doesn't exists, please pass correct target to tell command, such as mon.a/osd.1/mds.a/mgr The file system name is "con-fs2". I can reach the current rank-0 mds though. We are on the latest pacific version. Is this expected or do we need to configure something? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

Find out num of PGs that would go offline on OSD shutdown
by Andre Tann 10 Jan '25

10 Jan '25

Hi all, Proxmox VE warns me if I want to shut down an OSD, and some PGs would go offline if I continue. In the warning it tells me the exact number of affected PGs. I have now searched a bit, but wasn't able to find out how Proxmox knows the exact number. How can that be done on the CLI? Thanks for some hints. -- Andre Tann

3 3

Can I delete cluster_network?
by 苏察哈尔灿 10 Jan '25

10 Jan '25

   Hello, I have installed a ceph cluster, version v17.2.8, the cluster has public_network and cluster_network. For some other reason, I want to temporarily remove cluster_network and let the service data, heartbeat detection, and data recovery be done on public_network. Is this achievable? After removing it, if you want to configure it back, can you also achieve it? Thank you!

2 1