October 2024 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

CRC Bad Signature when using KRBD
by jsterr＠deckplosion.de 19 Dec '24

19 Dec '24

Hello Ceph Users, * Problem: we get the following errors when using krbd, we are using rbd for vms. * Workaround: by switching to librbd the errors disappear. * Software: ** Kernel: 6.8.8-2 (parameters: intel_iommu=on iommu=pt pcie_aspm.policy=performance) ** Ceph: 18.2.2 Description/Details: Errors from using krbd with ceph. Side-effects: [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000015af2284 data crc 1221767919 != exp. 282251377 [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000066b200ab data crc 3817026135 != exp. 3925662391 [Wed Aug 21 03:04:17 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:17 2024] libceph: osd13 (1)10.1.4.13:6809 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000008a131738 data crc 2612835980 != exp. 917302924 [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000005160776b data crc 2965872045 != exp. 563323792 [Wed Aug 21 03:04:21 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: osd6 (1)10.1.4.12:6843 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000007e548354 data crc 1265032637 != exp. 2426281931 [Wed Aug 21 03:06:44 2024] libceph: osd0 (1)10.1.4.11:6835 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000009214d802 data crc 2596010853 != exp. 1221875667 [Wed Aug 21 03:06:44 2024] libceph: osd10 (1)10.1.4.12:6809 bad crc/signature [Wed Aug 21 03:06:47 2024] libceph: read_partial_message 000000000f9edc73 data crc 1326019705 != exp. 3079604517 [Wed Aug 21 03:06:47 2024] libceph: osd3 (1)10.1.4.11:6803 bad crc/signature [Wed Aug 21 03:06:50 2024] libceph: read_partial_message 000000004769da61 data crc 3421275194 != exp. 4183754554 [Wed Aug 21 03:06:50 2024] libceph: osd8 (1)10.1.4.12:6835 bad crc/signature [Wed Aug 21 03:06:51 2024] libceph: read_partial_message 0000000044db9a59 data crc 2603270708 != exp. 4150529351 [Wed Aug 21 03:06:51 2024] libceph: osd14 (1)10.1.4.13:6806 bad crc/signature Description/Details 2: vms get problems with buffer i/o errors on rbd-backed virtual disks: Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=885s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=847s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Add. Sense: I/O process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 CDB: Write(10) 2a 00 34 87 48 08 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 CDB: Write (10) 2a 00 34 81 52 70 00 00 58 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281032 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 880890480 op 0x1: (WRITE) flags 0x103000 phys_seg 11 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: EXT4-fs warning (device sdb1): ext4_end_bio:343: I/0 error 10 writing to inode 27525908 starting Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111054, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: buffer_io_error: 21 callbacks suppressed Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on device sdbl, logical block 110159873 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on dev sdbl, logical block 110111055, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=875s Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111056, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111057, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111058, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 CDB: Write(10) 2a 00 34 87 48 10 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111059, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281040 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111060, lost async page write Thanks for helping out. Greetings Jonas

3 4

Re: Squid Manager Daemon: balancer crashing orchestrator and dashboard
by Alexander Closs 18 Dec '24

18 Dec '24

Just chiming in to say this also affected our cluster, same symptoms and a temporary fix of disabling the balancer. Happy to add my cluster's logs to the issue, though I suspect they'll look the same as Laimis' cluster. -Alex MIT CSAIL

7 7

failed to load OSD map for epoch 2898146, got 0 bytes
by Frank Schilder 15 Dec '24

15 Dec '24

Hi all, I have a strange problem on an octopus latest cluster. We had a couple of SSD OSDs down for a while and brought them up today again. For some reason, these OSDs don't come up and flood the log with messages like osd.1004 2971464 failed to load OSD map for epoch 2898146, got 0 bytes These messages cycle through the same epochs over and over again. I did not really fine too much help, there is an old thread about a similar/the same problem on a home lab cluster, with new OSDs though, I believe. I couldn't really find useful information. The OSDs seem to boot fine and then end up in something like a death loop. Below some snippets from the OSD log. Any hints appreciated. Thanks and best regards, Frank After OSD start, everything looks normal up to here: 2024-10-21T17:41:39.136+0200 7fad73cf6f00 0 osd.1004 2971464 load_pgs opened 205 pgs 2024-10-21T17:41:39.140+0200 7fad73cf6f00 -1 osd.1004 2971464 log_to_monitors {default=true} 2024-10-21T17:41:39.150+0200 7fad73cf6f00 -1 osd.1004 2971464 mon_cmd_maybe_osd_create fail: 'osd.1004 has already bound to class 'fs_meta', can not reset class to 'ssd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy 2024-10-21T17:41:39.155+0200 7fad519a3700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898134, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898135, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898136, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4f99f700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4f99f700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898134, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898135, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898136, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898137, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad73cf6f00 0 osd.1004 2971464 done with init, starting boot process 2024-10-21T17:41:39.155+0200 7fad73cf6f00 1 osd.1004 2971464 start_boot 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898138, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898139, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898140, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898141, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898142, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898143, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898144, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898145, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898146, got 0 bytes These messages repeat over and over again with some others of this form showing up every now and then: 2024-10-21T17:41:39.476+0200 7fad651ca700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 12] Generated table #82879: 76571 keys, 67866714 bytes 2024-10-21T17:41:39.688+0200 7fad651ca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1729525299690000, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 82879, "file_size": 67866714, "table_properties": {"data_size": 67111697, "index_size": 562601, "filter_size": 191557, "raw_key_size": 4823973, "raw_average_key_size": 63, "raw_value_size": 62631087, "raw_average_value_size": 817, "num_data_blocks": 15644, "num_entries": 76571, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} And another occasion: 2024-10-21T17:41:40.520+0200 7fad651ca700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 12] Generated table #82880: 76774 keys, 67868330 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899234, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899235, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899236, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad651ca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1729525300521403, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 82880, "file_size": 67868330, "table_properties": {"data_size": 67113021, "index_size": 562509, "filter_size": 191941, "raw_key_size": 4836742, "raw_average_key_size": 62, "raw_value_size": 62623274, "raw_average_value_size": 815, "num_data_blocks": 15630, "num_entries": 76774, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

6 12

Cephalocon Update - New Users Workshop and Power Users Session
by Dan van der Ster 26 Nov '24

26 Nov '24

Dear Fellow Cephalopods, Cephalocon 2024 hosted by CERN is just around the corner! In case you missed it, the full schedule is now live: https://events.linuxfoundation.org/cephalocon/ The Program Committee received a record-breaking 176 submissions, carefully selecting 60 speakers to present an exciting lineup. Attendees will have the chance to dive into the latest updates from Ceph Developers and engage with fellow users. We're thrilled to report that registration is approaching our target of 300 attendees, and interest is higher than anticipated! Our hosts are working hard to accommodate even more participants. If you haven't registered yet and plan to join us, don't delay---secure your spot today! Additionally, two pre-Cephalocon events are open for attendees: - New to Ceph? Join the Ceph New Users Workshop - https://indico.cern.ch/e/ceph-new-users-workshop It’s a great opportunity for beginners to get up to speed. - Experienced Ceph User? Participate in the Power Users afternoon session at the Developers Summit - https://indico.cern.ch/e/ceph-developer-summit These pre-events are free but require a valid Cephalocon registration. We look forward to seeing you there! Warm regards, Dan, on behalf of the Cephalocon Organizers -- Dan van der Ster CTO @ CLYSO

4 5

Ceph OSD perf metrics missing
by Kilian Ries 25 Nov '24

25 Nov '24

Hi, i'm running a Ceph v18.2.4 cluster. I'm trying to build some latency monitoring with the ceph daemon osd.4 perf dump cli command. On most of the OSDs i get all the metrics i need. On some OSDs i only get zero values: osd.op_latency.avgcount: 0 I already tried restarting the OSD process which didn't help. I also tried to reset the metrics via ceph daemon osd.4 perf reset all but that didn't help either. How can that be that some OSDs don't show values here? How can i fix that? Thanks Regards, Kilian

3 4

Question about speeding hdd based cluster
by Kyriazis, George 16 Nov '24

16 Nov '24

Hello ceph-users, I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools. What would increase the speed of the hdd-based cephfs more, and in what usage scenarios: 1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links. It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck? Thank you! George

6 15

Strange container restarts?
by Jan Marek 13 Nov '24

13 Nov '24

Hello, we have ceph cluster which consists of 12 host, on every host we have 12 NVMe "disks". On most of these host (9 of 12) we have in logs errors, see attached file. We tried to check this problem, and we have these points: 1) On every host there is only one OSD. Thus it's not problem in version 18.2.2 generally, because there will be on another OSD, not only one of host? 2) Sometimes one of this OSD crashed :-( It seems, that crashed OSD are from set of OSDs, which have this problem. 3) ceph cluster goes OK and it "doesn't know" about any problem with these OSD. It seem's, that this new instance of ceph-osd daemon tried to start either podman or conmon itself. We've tried to control PID files for conman, but they're seems to be OK? 4) We tried to check 'ceph orch' command, but it does not try to start these containers, because it know, that they exists and run ('ceph orch ps' list these containers as running). 5) I've tried to pause ochestrator, but I've still found in syslog these entries... :-( Please, is there any possibility to find out, where is problem and stop this? We have all of the ceph host prepared by ansible, thus there is the same environment. On every machine we have podman version 4.3.1+ds1-8+deb12u1 and conmon version 2.1.6+ds1-1. OS is Debian bookworm. Attached logs was prepared by: grep exec_died /var/log/syslog Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

3 3

9 out of 11 missing shards of shadow object in ERC 8:3 pool.
by Robert Kihlberg 12 Nov '24

12 Nov '24

After an upgrade from Nautilus to Pacific the scrub has found an inconsistent object and reports that 9 out of 11 shards are missing. (However, we're not sure this has to do with the upgrade). We have been able to trace it to a S3 bucket, but not to a specific S3 object. # radosgw-admin object stat --bucket=$BUCKET --object=$OBJECT ERROR: failed to stat object, returned error: (2) No such file or directory By design, we have a complete mirror of the bucket in another Ceph cluster and the amount of objects in the buckets match between the clusters. We are therefore somewhat confident that we are not missing any objects. Could this be a failed garbage collection where perhaps the primary OSD failed during gc? The garbage collector does not show anything that seems relevant though... radosgw-admin gc list --include-all | grep "eaa6801e-3967-4541-9b8ca98aa5c2.791015596" Any suggestions on how we can trace and/or fix this inconsistent object? # rados list-inconsistent-obj 11.3ff | jq { "epoch": 177981, "inconsistents": [ { "object": { "name": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "nspace": "", "locator": "", "snap": "head", "version": 109853 }, "errors": [], "union_shard_errors": [ "missing" ], "selected_object_info": { "oid": { "oid": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "key": "", "snapid": -2, "hash": 4294967295, "max": 0, "pool": 11, "namespace": "" }, "version": "17636'109853", "prior_version": "0'0", "last_reqid": "client.791015590.0:449317175", "user_version": 109853, "size": 8388608, "mtime": "2022-01-24T03:33:42.457722+0000", "local_mtime": "2022-01-24T03:33:42.471042+0000", "lost": 0, "flags": [ "dirty", "data_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xe588978d", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 14, "primary": true, "shard": 0, "errors": [], "size": 1048576 }, { "osd": 67, "primary": false, "shard": 1, "errors": [ "missing" ] }, { "osd": 77, "primary": false, "shard": 4, "errors": [], "size": 1048576 }, { "osd": 225, "primary": false, "shard": 9, "errors": [ "missing" ] }, { "osd": 253, "primary": false, "shard": 8, "errors": [ "missing" ] }, { "osd": 327, "primary": false, "shard": 6, "errors": [ "missing" ] }, { "osd": 568, "primary": false, "shard": 2, "errors": [ "missing" ] }, { "osd": 610, "primary": false, "shard": 7, "errors": [ "missing" ] }, { "osd": 700, "primary": false, "shard": 3, "errors": [ "missing" ] }, { "osd": 736, "primary": false, "shard": 10, "errors": [ "missing" ] }, { "osd": 764, "primary": false, "shard": 5, "errors": [ "missing" ] } ] } ] }

3 4

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Squid 19.2.0 balancer causes restful requests to be lost
by Chris Palmer 07 Nov '24

07 Nov '24

I've just upgraded a test cluster from 18.2.4 to 19.2.0. Package install on centos 9 stream. Very smooth upgrade. Only one problem so far... The MGR restful api calls work fine. EXCEPT whenever the balancer kicks in to find any new plans. During the few seconds that the balancer takes to run, all REST calls seem to be completely dropped. The MGR log file normally logs the POST requests, but the ones during these few seconds don't appear at all. This causes our monitoring to keep raising alarms. The cluster is in a completely stable state, HEALTH_OK, very little activity, just the occasional scrubs. We use the restful API for monitoring (using the Ceph for Zabbix Agent 2 plugin, as Zabbix is the over-arching monitoring platform in the data centre). I haven't yet checked the memory leak problems that we (like many) were having, because I have been chasing this new problem. The problem is quite repeatable. To diagnose I use the zabbix_get utility to query every second. Whenever the MGR log file shows the balancer kick in the REST requests time out (after 3 seconds - not sure whether the utility or the MGR is timing them out - I suspect the utility). They normally complete after a small fraction of a second. With the balancer disabled the REST interface works reliably again. The problem does not occur pre-squid. Anyone any ideas, or shall I raise a bug? Thanks, Chris

6 9

Recreate Destroyed OSD
by Dave Hall 07 Nov '24

07 Nov '24

Hello. Sorry if it appears that I am reposting the same issue under a different topic. However, I feel that the problem has moved and I now have different questions. At this point I have, I believe, removed all traces of OSD.12 from my cluster - based on steps in the Reef docs at https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#. I have further located and removed the WAL/DB LV on an associated NVMe drive (shared with 3 other OSDs). I don't believe the instructions for replacing an OSD (ceph-volume lvm prepare) still apply, so I have been trying to work with the instructions under ADDING AN OSD (MANUAL). However, since my installation is containerized (Podman), it is unclear which steps should be issued on the host and which within 'cephadm shell'. There is also another ambiguity: In step 3 the instruction is to 'mkfs -t {fstype}' and then to 'mount -o user_xattr'. However, which fs type? After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets throws errors about the keyring file. So, are these the right instructions to be using in a containerized installation? Are there, in general, alternate documents for containerized installations? Lastly, the above cited instructions don't say anything about the separate WAL/DB LV. Please advise. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

5 17

Assistance Required: Ceph OSD Out of Memory (OOM) Issue
by Md Mosharaf Hossain 03 Nov '24

03 Nov '24

Dear Ceph Community, I hope this message finds you well. I am encountering an out-of-memory (OOM) issue with one of my Ceph OSDs, which is repeatedly getting killed by the OOM killer on my system. Below are the relevant details from the log: *OOM Log*: [Wed Oct 30 13:14:48 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/system-ceph\x2dosd.slice,task=ceph-osd,pid=6213,uid=64045 [Wed Oct 30 13:14:48 2024] Out of memory: Killed process 6213 (ceph-osd) total-vm:216486528kB, anon-rss:211821164kB, file-rss:0kB, shmem-rss:0kB, UID:64045 pgtables:418836kB oom_score_adj:0 [Wed Oct 30 13:14:58 2024] oom_reaper: reaped process 6213 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB *Ceph OSD Log*: 2024-10-30T13:15:30.207+0600 7f906c74dd80 0 _get_class not permitted to load lua 2024-10-30T13:15:30.211+0600 7f906c74dd80 0 <cls> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello 2024-10-30T13:15:30.215+0600 7f906c74dd80 0 _get_class not permitted to load kvs 2024-10-30T13:15:30.219+0600 7f906c74dd80 0 _get_class not permitted to load queue 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 <cls> /build/ceph-15.2.17/src/cls/cephfs/cls_cephfs.cc:198: loading cephfs 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 432629239337189376, adjusting msgr requires for clients 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 3314933000854323200, adjusting msgr requires for osds 2024-10-30T13:15:30.223+0600 7f906c74dd80 1 osd.13 299547 check_osdmap_features require_osd_release unknown -> octopus 2024-10-30T13:15:31.023+0600 7f906c74dd80 0 osd.13 299547 load_pgs *Environment Details*: - Ceph Version: 15.2.17 (Octopus) - OSD: osd.13 - Kernel: Linux kernel version It seems that the OSD process is consuming a substantial amount of memory (total-vm: 216486528kB, anon-rss: 211821164kB), leading to OOM kills on the node. The OSD service restarts but continues to showing consumption excessive memory and OSD get down. Could you please provide guidance or suggestions on how to mitigate this issue? Are there any known memory management settings, configuration adjustments, or OSD-specific tuning parameters that could help prevent this from recurring? Any help would be greatly appreciated. Thank you for your time and assistance! Regards Mosharaf Hossain Manager, Product Development Bangladesh Online (BOL) Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 95757 Cell: +880 1787 680828, Web: www.bol-online.com

2 2

MDS and stretched clusters
by Sake Ceph 01 Nov '24

01 Nov '24

Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other? We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode.

5 8

KRBD: downside of setting alloc_size=4M for discard alignment?
by Friedrich Weber 31 Oct '24

31 Oct '24

Hi, Some of our Proxmox VE users have noticed that a large fstrim inside a QEMU/KVM guest does not free up as much space as expected on the backing RBD image -- if the image is mapped on the host via KRBD and passed to QEMU as a block device (checked via `rbd du --exact`). If the image is attached via QEMU's librbd integration, fstrim seems to work much better. I've found an earlier discussion [0] according to which, for fstrim to work properly, the filesystem should be aligned a object size (4M) boundaries. Indeed, in the test setups I've looked at, the filesystem is not aligned to 4M boundaries. Still, I'm wondering if there might be a solution that doesn't require a specific partitioning/filesystem layout. To have a simpler test setup, I'm not looking at VMs and instead into unaligned blkdiscard on a KRBD-backed block device (on the host). On my test cluster (for versions see [5]), I create an 1G test volume, map it with default settings, write random data to it, and then issue blkdiscard with an 1M offset (see [1] for complete commands): > # blkdiscard --offset 1M /dev/rbd/vmpool/test An `rbd du --exact` reports a size of 256M: > # rbd du --exact -p vmpool test > NAME PROVISIONED USED > test 1 GiB 256 MiB Naively I would expect a result between 1 and 4M, my reasoning being that the 1023M discard could be split into 3M (to get to 4M alignment) plus 1020M. But I've checked the kernel's discard splitting logic [2], and as far as I understand it, it aims to align the discard requests to `discard_granularity`, which is 64k here: > /sys/class/block/rbd0/queue/discard_granularity:65536 I've found I can set the `alloc_size` option [3] to 4M which sets `discard_granularity` to 4M. The result of the blkdiscard is much closer to my expectations (see [4] for complete commands). > # blkdiscard --offset 1M /dev/rbd/vmpool/test > # rbd du --exact -p vmpool test > NAME PROVISIONED USED > test 1 GiB 1 MiB However, apparently with `alloc_size` set to 4M, `minimum_io_size` is also set to 4M (it was 64k before, see [1]): > /sys/class/block/rbd0/queue/minimum_io_size:4194304 My expectation is that this could negatively impact non-discard IO performance (write amplification?). But I am unsure, as I ran a few small benchmarks and couldn't really see any difference between the two settings. Thus, my questions: - Should I expect any downside for non-discard IO after setting `alloc_size` to 4M? - If yes: would it be feasible for KRBD to decouple `discard_granularity` and `minimum_io_size`, i.e., expose an option to set only `discard_granularity` to 4M? Happy about any pointers, and let me know if I can provide any further information. Thanks and best wishes, Friedrich [0] https://www.spinics.net/lists/ceph-users/msg67740.html [1] > # rbd create -p vmpool test --size 1G > # rbd map -p vmpool test > /dev/rbd0 > # grep '' /sys/class/block/rbd0/queue/{discard_*,minimum_io_size,optimal_*} > /sys/class/block/rbd0/queue/discard_granularity:65536 > /sys/class/block/rbd0/queue/discard_max_bytes:4194304 > /sys/class/block/rbd0/queue/discard_max_hw_bytes:4194304 > /sys/class/block/rbd0/queue/discard_zeroes_data:0 > /sys/class/block/rbd0/queue/minimum_io_size:65536 > /sys/class/block/rbd0/queue/optimal_io_size:4194304 > # dd if=/dev/urandom of=/dev/rbd/vmpool/test bs=4M > dd: error writing '/dev/rbd/vmpool/test': No space left on device > 257+0 records in > 256+0 records out > 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.73227 s, 227 MB/s > # rbd du --exact -p vmpool test > NAME PROVISIONED USED > test 1 GiB 1 GiB [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/blo… [3] https://docs.ceph.com/en/reef/man/8/rbd/ [4] > # rbd map -p vmpool test -o alloc_size=4194304 > /dev/rbd0 > # grep '' /sys/class/block/rbd*/device/config_info > 10.1.1.201:6789,10.1.1.202:6789,10.1.1.203:6789 name=admin,key=client.admin,alloc_size=4194304 vmpool test - > # grep '' /sys/class/block/rbd0/queue/{discard_*,minimum_io_size,optimal_*} > /sys/class/block/rbd0/queue/discard_granularity:4194304 > /sys/class/block/rbd0/queue/discard_max_bytes:4194304 > /sys/class/block/rbd0/queue/discard_max_hw_bytes:4194304 > /sys/class/block/rbd0/queue/discard_zeroes_data:0 > /sys/class/block/rbd0/queue/minimum_io_size:4194304 > /sys/class/block/rbd0/queue/optimal_io_size:4194304 > # dd if=/dev/urandom of=/dev/rbd/vmpool/test bs=4M > dd: error writing '/dev/rbd/vmpool/test': No space left on device > 257+0 records in > 256+0 records out > 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.39016 s, 245 MB/s > # rbd du --exact -p vmpool test > NAME PROVISIONED USED > test 1 GiB 1 GiB [5] Host: Proxmox VE 8.2 but with Ubuntu mainline kernel 6.11 build (6.11.0-061100-generic from https://kernel.ubuntu.com/mainline/v6.11/) Ceph: Proxmox build of 18.2.4, but happy try a different build if needed.

3 3

"ceph orch" not working anymore
by Malte Stroem 31 Oct '24

31 Oct '24

Hi there, I have a three node cluster with latest Ceph 18 and cephadm as orchestrator. One node is dead. So the cluster is not healthy but working. Now: Working: cephadm ls or other commands with cephadm... However: ceph orch... does not work at all: Error ENOENT: No orchestrator configured (try `ceph orch set backend`) But it is set! And disabling and enabling it does not work! The cluster is running and working. All ceph commands are working fine. ceph mgr fail or restarting the MGRs did not help. Checked the configuration: ceph config get... everything is fine. Any ideas? Best, Malte

7 24

Deploy custom mgr module
by Darrell Enns 31 Oct '24

31 Oct '24

Is there a simple way to deploy a custom (in-house) mgr module to an orchestrator managed cluster? I assume the module code would need to be included in the mgr container image. However, there doesn't seem to be a straightforward way to do this without having the module merged to upstream ceph (not possible for a custom/in-house solution) or maintaining an in-house container repository and custom container images (a lot of extra maintenance overhead). Also, what's the best way to handle testing during development? Custom scripts to push the code into to a mgr container in a dev cluster?

4 4

Destroyed OSD clinging to wrong disk
by Dave Hall 30 Oct '24

30 Oct '24

Hello. The following is on a Reef Podman installation: In attempting to deal over the weekend with a failed OSD disk, I have somehow managed to have two OSDs pointing to the same HDD, as shown below. [image: image.png] To be sure, the failure occurred on OSD.12, which was pointing to /dev/sdi. I disabled the systemd unit for OSD.12 because it kept restarting. I then destroyed it. When I physically removed the failed disk and rebooted the system, the disk enumeration changed. So, before the reboot, OSD.12 was using /dev/sdi. After the reboot, OSD.9 moved to /dev/sdi. I didn't know that I had an issue until 'ceph-volume lvm prepare' failed. It was in the process of investigating this that I found the above. Right now I have reinserted the failed disk and rebooted, hoping that OSD.12 would find its old disk by some other means, but no joy. My concern is that if I run 'ceph osd rm' I could take out OSD.9. I could take the precaution of marking OSD.9 out and let it drain, but I'd rather not. I am, perhaps, more inclined to manually clear the lingering configuration associated with OSD.12 if someone could send me the list of commands. Otherwise, I'm open to suggestions. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 4

no recovery running
by Joffrey 30 Oct '24

30 Oct '24

HI, This is my cluster: cluster: id: c300532c-51fa-11ec-9a41-0050569c3b55 health: HEALTH_WARN Degraded data redundancy: 2062374/1331064781 objects degraded (0.155%), 278 pgs degraded, 40 pgs undersized 2497 pgs not deep-scrubbed in time 2497 pgs not scrubbed in time services: mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 (age 9d) mgr: hbgt-ceph1-mon3.gmfzqm(active, since 10d), standbys: hbgt-ceph1-mon2.nteihj, hbgt-ceph1-mon1.thrnnu osd: 96 osds: 96 up (since 9d), 96 in (since 45h); 1588 remapped pgs rgw: 3 daemons active (3 hosts, 2 zones) data: pools: 16 pools, 2497 pgs objects: 266.22M objects, 518 TiB usage: 976 TiB used, 808 TiB / 1.7 PiB avail pgs: 2062374/1331064781 objects degraded (0.155%) 349917519/1331064781 objects misplaced (26.289%) 1312 active+remapped+backfill_wait 864 active+clean 199 active+recovery_wait+degraded+remapped 38 active+recovery_wait+degraded 33 active+undersized+degraded+remapped+backfill_wait 33 active+recovery_wait+remapped 7 active+recovery_wait 6 active+undersized+degraded+remapped+backfilling 2 active+recovering+remapped 1 active+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+recovery_wait+undersized+degraded+remapped io: client: 683 KiB/s rd, 2.2 KiB/s wr, 51 op/s rd, 2 op/s wr No recovery is running and I don't understand why. I have free space: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1784.12231 - 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1 TiB 808 TiB 54.72 1.00 - root default -5 208.09680 - 208 TiB 142 TiB 130 TiB 51 GiB 605 GiB 66 TiB 68.14 1.25 - host hbgt-ceph1-osd01 1 hdd 17.34140 1.00000 17 TiB 11 TiB 11 TiB 33 KiB 49 GiB 5.9 TiB 66.16 1.21 136 up osd.1 3 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 23 GiB 49 GiB 6.3 TiB 63.80 1.17 139 up osd.3 5 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 139 MiB 53 GiB 4.8 TiB 72.31 1.32 142 up osd.5 7 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 51 GiB 5.6 TiB 67.97 1.24 145 up osd.7 9 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 2.2 GiB 49 GiB 6.0 TiB 65.67 1.20 140 up osd.9 11 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 329 MiB 50 GiB 5.5 TiB 68.42 1.25 145 up osd.11 13 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52 GiB 5.1 TiB 70.45 1.29 153 up osd.13 15 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 61 KiB 48 GiB 5.7 TiB 66.85 1.22 144 up osd.15 17 hdd 17.34140 1.00000 17 TiB 11 TiB 9.5 TiB 272 MiB 45 GiB 6.8 TiB 60.63 1.11 120 up osd.17 19 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 12 GiB 50 GiB 5.9 TiB 65.90 1.20 134 up osd.19 21 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.6 GiB 57 GiB 4.1 TiB 76.49 1.40 152 up osd.21 23 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 31 KiB 54 GiB 4.7 TiB 73.10 1.34 124 up osd.23 -3 208.09680 - 208 TiB 146 TiB 134 TiB 64 GiB 629 GiB 62 TiB 70.05 1.28 - host hbgt-ceph1-osd02 0 hdd 17.34140 1.00000 17 TiB 11 TiB 9.8 TiB 22 GiB 49 GiB 6.6 TiB 62.07 1.13 124 up osd.0 2 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.7 GiB 52 GiB 5.2 TiB 70.14 1.28 150 up osd.2 4 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 48 GiB 5.8 TiB 66.83 1.22 152 up osd.4 6 hdd 17.34140 0.85004 17 TiB 13 TiB 12 TiB 11 GiB 58 GiB 4.0 TiB 76.85 1.40 153 up osd.6 8 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 54 GiB 4.9 TiB 71.58 1.31 152 up osd.8 10 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 6.3 MiB 47 GiB 6.1 TiB 64.91 1.19 133 up osd.10 12 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 109 MiB 51 GiB 5.6 TiB 67.72 1.24 137 up osd.12 14 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 GiB 53 GiB 5.1 TiB 70.37 1.29 148 up osd.14 16 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 2.9 GiB 54 GiB 4.9 TiB 71.65 1.31 145 up osd.16 18 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 88 MiB 55 GiB 4.8 TiB 72.45 1.32 154 up osd.18 20 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 108 MiB 55 GiB 4.6 TiB 73.39 1.34 166 up osd.20 22 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 209 MiB 54 GiB 4.7 TiB 72.68 1.33 138 up osd.22 -7 208.09680 - 208 TiB 148 TiB 136 TiB 29 GiB 635 GiB 60 TiB 71.03 1.30 - host hbgt-ceph1-osd03 24 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.4 GiB 56 GiB 4.5 TiB 74.27 1.36 150 up osd.24 25 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 32 KiB 57 GiB 4.1 TiB 76.27 1.39 162 up osd.25 26 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 1.4 GiB 49 GiB 6.0 TiB 65.30 1.19 134 up osd.26 27 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.7 GiB 53 GiB 4.7 TiB 73.16 1.34 152 up osd.27 28 hdd 17.34140 1.00000 17 TiB 12 TiB 12 TiB 2.3 MiB 52 GiB 4.9 TiB 71.97 1.32 158 up osd.28 29 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 10 GiB 52 GiB 5.3 TiB 69.72 1.27 141 up osd.29 30 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 266 MiB 54 GiB 4.6 TiB 73.38 1.34 142 up osd.30 31 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 93 MiB 50 GiB 5.7 TiB 67.36 1.23 145 up osd.31 32 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 13 GiB 50 GiB 6.1 TiB 64.77 1.18 131 up osd.32 33 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 MiB 53 GiB 5.2 TiB 69.77 1.28 132 up osd.33 34 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 242 MiB 50 GiB 5.5 TiB 68.17 1.25 139 up osd.34 35 hdd 17.34140 1.00000 17 TiB 14 TiB 13 TiB 135 MiB 58 GiB 3.8 TiB 78.26 1.43 157 up osd.35 -9 208.09680 - 208 TiB 146 TiB 134 TiB 38 GiB 627 GiB 62 TiB 70.19 1.28 - host hbgt-ceph1-osd04 36 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 210 MiB 47 GiB 6.4 TiB 63.16 1.15 139 up osd.36 37 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 222 MiB 53 GiB 5.0 TiB 71.34 1.30 147 up osd.37 38 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 13 MiB 55 GiB 4.5 TiB 74.33 1.36 155 up osd.38 39 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 135 MiB 52 GiB 5.2 TiB 70.22 1.28 140 up osd.39 40 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 25 MiB 53 GiB 4.9 TiB 71.46 1.31 150 up osd.40 41 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 54 GiB 5.0 TiB 71.36 1.30 150 up osd.41 42 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 823 MiB 50 GiB 5.8 TiB 66.49 1.22 156 up osd.42 43 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52 GiB 5.2 TiB 69.97 1.28 160 up osd.43 44 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52 GiB 5.3 TiB 69.35 1.27 133 up osd.44 45 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 12 GiB 55 GiB 4.8 TiB 72.12 1.32 148 up osd.45 46 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52 GiB 5.1 TiB 70.74 1.29 148 up osd.46 47 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 188 MiB 54 GiB 4.9 TiB 71.74 1.31 151 up osd.47 -11 208.09680 - 208 TiB 147 TiB 135 TiB 27 GiB 627 GiB 62 TiB 70.45 1.29 - host hbgt-ceph1-osd05 48 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 49 GiB 5.7 TiB 67.11 1.23 152 up osd.48 49 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 52 GiB 5.1 TiB 70.87 1.30 155 up osd.49 50 hdd 17.34140 0.90002 17 TiB 12 TiB 11 TiB 11 GiB 54 GiB 5.2 TiB 70.01 1.28 147 up osd.50 51 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 84 MiB 54 GiB 4.6 TiB 73.24 1.34 147 up osd.51 52 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 337 MiB 53 GiB 4.8 TiB 72.26 1.32 146 up osd.52 53 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.6 GiB 54 GiB 5.1 TiB 70.56 1.29 132 up osd.53 54 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.3 GiB 56 GiB 4.5 TiB 73.77 1.35 138 up osd.54 55 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 10 GiB 53 GiB 4.8 TiB 72.59 1.33 148 up osd.55 56 hdd 17.34140 1.00000 17 TiB 11 TiB 9.6 TiB 1.8 GiB 46 GiB 6.7 TiB 61.16 1.12 127 up osd.56 57 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 3.1 MiB 50 GiB 5.6 TiB 67.96 1.24 148 up osd.57 58 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 687 KiB 52 GiB 5.0 TiB 70.93 1.30 152 up osd.58 59 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 22 KiB 55 GiB 4.4 TiB 74.88 1.37 147 up osd.59 -13 245.93866 - 246 TiB 112 TiB 106 TiB 28 GiB 498 GiB 134 TiB 45.56 0.83 - host hbgt-ceph1-osd06 60 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 10 GiB 51 GiB 9.3 TiB 54.43 0.99 134 up osd.60 61 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 216 MiB 43 GiB 11 TiB 47.06 0.86 120 up osd.61 62 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 330 MiB 41 GiB 11 TiB 44.53 0.81 116 up osd.62 63 hdd 20.49489 1.00000 20 TiB 8.9 TiB 8.4 TiB 354 MiB 39 GiB 12 TiB 43.29 0.79 113 up osd.63 64 hdd 20.49489 1.00000 20 TiB 8.1 TiB 7.7 TiB 148 MiB 36 GiB 12 TiB 39.72 0.73 117 up osd.64 65 hdd 20.49489 1.00000 20 TiB 12 TiB 12 TiB 14 KiB 53 GiB 8.4 TiB 58.90 1.08 159 up osd.65 66 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 18 MiB 49 GiB 9.8 TiB 52.33 0.96 129 up osd.66 67 hdd 20.49489 1.00000 20 TiB 498 GiB 1.5 GiB 1 KiB 745 MiB 20 TiB 2.37 0.04 6 up osd.67 68 hdd 20.49489 1.00000 20 TiB 10 TiB 9.8 TiB 13 GiB 47 GiB 10 TiB 50.12 0.92 138 up osd.68 69 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 2.0 GiB 48 GiB 9.7 TiB 52.77 0.96 135 up osd.69 70 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 1.6 GiB 50 GiB 9.2 TiB 55.24 1.01 150 up osd.70 71 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 86 MiB 42 GiB 11 TiB 45.96 0.84 130 up osd.71 -15 245.93866 - 246 TiB 114 TiB 108 TiB 62 GiB 505 GiB 132 TiB 46.18 0.84 - host hbgt-ceph1-osd07 72 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 12 GiB 49 GiB 9.6 TiB 53.39 0.98 146 up osd.72 73 hdd 20.49489 1.00000 20 TiB 8.5 TiB 8.0 TiB 194 MiB 38 GiB 12 TiB 41.31 0.75 112 up osd.73 74 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 10 GiB 42 GiB 11 TiB 46.12 0.84 124 up osd.74 75 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 12 GiB 41 GiB 11 TiB 44.16 0.81 125 up osd.75 76 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 114 MiB 41 GiB 11 TiB 45.91 0.84 111 up osd.76 77 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 1.5 GiB 42 GiB 11 TiB 47.03 0.86 127 up osd.77 78 hdd 20.49489 1.00000 20 TiB 7.7 TiB 7.3 TiB 12 GiB 36 GiB 13 TiB 37.77 0.69 105 up osd.78 79 hdd 20.49489 1.00000 20 TiB 8.2 TiB 7.7 TiB 1 KiB 36 GiB 12 TiB 39.92 0.73 91 up osd.79 80 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 371 MiB 41 GiB 11 TiB 46.13 0.84 132 up osd.80 81 hdd 20.49489 1.00000 20 TiB 9.8 TiB 9.3 TiB 353 MiB 45 GiB 11 TiB 47.95 0.88 121 up osd.81 82 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 12 GiB 51 GiB 9.2 TiB 55.23 1.01 142 up osd.82 83 hdd 20.49489 1.00000 20 TiB 10 TiB 9.6 TiB 1.9 GiB 44 GiB 10 TiB 49.28 0.90 125 up osd.83 -17 251.76105 - 252 TiB 23 TiB 11 TiB 43 MiB 64 GiB 229 TiB 8.99 0.16 - host hbgt-ceph1-osd08 84 hdd 20.98009 1.00000 21 TiB 1.0 TiB 65 GiB 1 KiB 1.4 GiB 20 TiB 4.93 0.09 5 up osd.84 85 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 813 MiB 20 TiB 4.63 0.08 7 up osd.85 86 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.0 GiB 20 TiB 4.63 0.08 2 up osd.86 87 hdd 20.98009 1.00000 21 TiB 1008 GiB 14 GiB 1 KiB 1.0 GiB 20 TiB 4.69 0.09 7 up osd.87 88 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.6 GiB 20 TiB 4.63 0.08 6 up osd.88 89 hdd 20.98009 1.00000 21 TiB 4.9 TiB 4.0 TiB 17 MiB 20 GiB 16 TiB 23.56 0.43 51 up osd.89 90 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.5 GiB 20 TiB 4.63 0.08 5 up osd.90 91 hdd 20.98009 1.00000 21 TiB 3.3 TiB 2.3 TiB 1 KiB 11 GiB 18 TiB 15.70 0.29 40 up osd.91 92 hdd 20.98009 1.00000 21 TiB 5.1 TiB 4.1 TiB 26 MiB 20 GiB 16 TiB 24.34 0.44 60 up osd.92 93 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.3 GiB 20 TiB 4.63 0.08 9 up osd.93 94 hdd 20.98009 1.00000 21 TiB 1.4 TiB 489 GiB 1 KiB 3.4 GiB 20 TiB 6.90 0.13 14 up osd.94 95 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1016 MiB 20 TiB 4.63 0.08 4 up osd.95 TOTAL 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1 TiB 808 TiB 54.72 I tried: - update backfill and recovery settings - change osd mclock profile - change pg/pgp_num The cluster does not recovery. Clients are working well. What can I do ? Thanks for your help

3 2

why performance difference between 'rados bench seq' and 'rados bench rand' quite significant
by Louisa 30 Oct '24

30 Oct '24

Hi all, We used 'rados bench' to test 4k object read and write operations. Our cluster is pacific, one node, 11 bluestore osd ,db and wal share the block device. Block device is HDD. 1. testing 4k write with command 'rados bench 120 write -t 16 -b 4K -p rep3datapool --run-name 4kreadwrite --no-cleanup' 2. Before tesing 4k reads, we restarted all OSD daemons. The perfomance of 'rados bench 120 seq -t 16 -p rep3datapool --run-name 4kreadwrite' was very good, which Average IOPS: 17735; using 'ceph daemon osd.1 perf dump rocksdb' , we found the rocksdb:get_latency avgcount: 15189, avgtime: 0.000012947 (12.9us) 3. Before tesing 4k rand reads, we restarted all OSD daemons. 'rados bench 60 rand -t 16 -p rep3datapool --run-name 4kreadwrite' average IOPS: 2071 rocksdb:get_latency avgcount: 8756, avgtime: 0.001761293 (1.7ms) Q1: Why performance difference between 'rados bench seq' and 'rados bench rand' quite significant? How to explain the rocksdb get_latency perfomance between this two scenario? 4. We write 40w 4k object to the pool, restarted all OSD daemons. running 'rados bench 120 seq -t 16 -p rep3datapool --run-name 4kreadwrite' again. Average IOPS~= 2000. rocsdb:get_latency avgtime also reached milliseconds level Q2: Why 'rados bench seq' performance decresing extremly after writing some more 4k object to the pool? Q3: Is there any methods and suggestions to optimized the read performance of this scenario under this hardware configuration.

2 2

Ceph Crash Module "RADOS permission denied"
by mailing-lists 29 Oct '24

29 Oct '24

Hey Cephers, i was investigating some other issue, when I stumbled across this. I am not sure, if this is "as intended" or faulty. This is a cephadm cluster on reef 18.2.4, containerized with docker. The ceph-crash module states that it cant find its key and that it cant access RADOS. Pre-face, this is how its key looks like and what its caps are (dont worry about leaking the key, its a Test-Cluster): client.crash.bi-ubu-srv-ceph2-01 key: AQBi5CBnedEKORAAgaTwkiqO7KJ+wzu8+EGXEQ== caps: [mgr] profile crash caps: [mon] profile crash Looks like it should. As per documentation: https://docs.ceph.com/en/reef/mgr/crash/#automated-collection The key on the node/directory of this ceph-crash container is the same. This is the docker logs of the container running ceph-crash. INFO:ceph-crash:pinging cluster to exercise our key 2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 AuthRegistry(0x7f7ab4067810) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 AuthRegistry(0x7f7abb1ae0c0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-10-29T13:34:28.265+0000 7f7ab99ac640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7aba1ad640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7aba9ae640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 13] RADOS permission denied (error connecting to the cluster) INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s As we can see here, it says that it couldn't find its key. After some online research, I found that some names are not good/interpreted correctly. Well, just to test, i've changed the unit.run file of ceph-crash-container and restarted it. This is the part I've changed: From /var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/ceph.client.crash.bi-ubu-srv-ceph2-01.keyring To /var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/keyring Then the logs look different, but the permissions denied error is still there. INFO:ceph-crash:pinging cluster to exercise our key 2024-10-29T13:36:42.273+0000 7f1d04b49640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2024-10-29T13:36:42.277+0000 7f1cff7fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2024-10-29T13:36:42.281+0000 7f1cfffff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s Can anybody tell me if this is "normal"? I think this is suspicious. Because for example, if i do ceph crash ls, it does not yield anything, while there is definitely something on a node inside the folder /var/lib/ceph/xyz/crash/ Thanks in advance and best wishes!

2 1

Ceph Steering Committee Meeting 2024-10-28
by Laura Flores 28 Oct '24

28 Oct '24

Hi all, Topics we discussed today included an explanation of the CVE fix process, an evaluation of the OS recommendation documentation, a discussion on some options about releasing el8 packages for the upcoming quincy point release, and CEC election results. Here are the highlights from today's meeting: - Existing CVEs and fix/release process [Ernesto + Sage + Yaarit]: - Sage: Better to build things into the process instead of a hotfix (exception for serious situation) - Revising CVE reporting process to use github - https://docs.ceph.com/en/latest/security/process/ - [zdover] os-recommendations.rst possibly out of date -- who is the authority on which distros are supported? - add squid: centos 9 (AH), ubuntu 22 (AH), debian 12 (C) - Is Debian 12 supported in Reef? - i think so, based on https://download.ceph.com/debian-reef/dists/bookworm/ --[zdover again]: that was exactly the source upon which I based the question. So I think I have all the information I need. - [cbodley] should quincy release block on centos 8 or no? - Blocked on adding back a centos 8 builder to our matrix - Community members have managed to build packages- could we point to these? - CERN archives is another option - If we drop in packages/containers built outside of the usual upstream build process, how confident are we that these artifacts will be functionally identical to what we would have built ourselves? cf. the issues seen in the past with Debian/Ubuntu-channel builds. A build done by a community member might have different compilation options, directory paths, non-stock libraries etc. - Will meet later on this with Dan Mick - Election: - Results: https://vote.heliosvoting.org/helios/elections/e03494ce-e04c-41d0-bb05-ec5c… - Governance Update: https://github.com/ceph/ceph/pull/60518 - https://pad.ceph.com/p/css-vote-2024q4 Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Influencing the osd.id when creating or replacing an osd
by Shain Miley 28 Oct '24

28 Oct '24

Hello, I am still using ceph-deploy to add osd’s to my cluster. From what I have read ceph-deploy does not allow you to specify the osd.id when creating new osds, however I am wondering if there is a way to influence the number that ceph will assign for the next osd that is created. I know that it really shouldn’t matter what osd number gets assigned to the disk but as the number of osd increases it is much easier to keep track of where things are if you can control the id when replacing failed disks or adding new nodes. Thank you, Shain

7 11

RGW lifecycle wrongly removes NOT expired delete-markers which have a objects
by Александр Руденко 28 Oct '24

28 Oct '24

Hi, We have very strange behavior, when LC "restores" previous versions of objects. It looks like: 1. We have latest object: *object1* 2. We remove object and have delete-marker on top of it. We can't see object1 in bucket listing: *marker (latest)* *object1 (not latests, version1)* 3. LC decides to remove marker as Expired: *RGW log: DELETED::REDACTED-bucket-REDACTED[default.1471649938.1]):REDACTED-object-REDACTED[N3gFGHIZLylFcO9RdnlChEfIAL076VC] (delete marker expiration) wp_thrd: 0, 0* 4. and we can see object1 in bucket listing: *object1 (latest, version1)* But, it is wrong behavior because ExpiredObjectDeleteMarker - it's marker that doesn't have ANY version under it. Unfortunately we can't reproduce this behavior in our test environment (proof that it's not a normal behavior). But it occurs regularly in our production in one customer's bucket. It crashes the customer's application because for it, it looks like an "unknown file". We are 100% sure that it's LC. Because we have logs when an object was uploaded, removed (no version specified), and we have LC log which removes DM but object1 exists. In this case we have unique object names like sha256. It means that we cant have name duplication in this bucket, it can't be a new object with the same name. And we have mtime of "restored" object1 which in the past, for example 20 days past. All our findings are matched together. LC policy: { "Rules": [ { "Status": "Enabled", "Prefix": "", "NoncurrentVersionExpiration": { "NoncurrentDays": 30 }, "Expiration": { "ExpiredObjectDeleteMarker": true }, "ID": "Remove all NOT latest version" } ] } Ceph 16.2.13 We have about 20 RGW instances and only two of them have LC thread enabled. The bucket has about 1.5M objects and 701 shards. I know that it's too many shards for this bucket, but it's another story) Where can be a problem?

1 0

Re: Procedure for temporary evacuation and replacement
by Frank Schilder 28 Oct '24

28 Oct '24

Hi everybody, apparently, I forgot to report back. The evacuation completed without problems and we are replacing disks at the moment. This procedure worked like a charm (please read the thread to see why we didn't just shut down OSDs and used recovery for rebuild): 1.) For all OSDs: ceph osd out ID # just set them out, this is sticky and does what you want 2.) Wait for rebalance to finish 3.) Replace disks. 4.) Deploy OSDs with the same IDs as before per host. 5.) Start OSDs and let rebalance back. During the evacuation you might want to consider setting "osd_delete_sleep" to a high value to avoid issues due to PG removal reported in this thread; see messages by Joshua Baergen in this thread. The only wish I have is that after setting the OSDs "out" it would be great to have an option to have recovery kick in in addition to speed up movement of data. Instead of just reading shard by shard from the out-OSDs, shards should also be reconstructed by recovery from all other OSDs. Our evacuation lasted for about 2 weeks. If recovery would kick in, this time would go down to 2-3 days. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Sent: Monday, October 28, 2024 4:41 AM To: Frank Schilder Subject: Re: [ceph-users] Re: Procedure for temporary evacuation and replacement Hi Frank, Finally what was the best way to do this evacuation replacement? I want to destroy all my osds node by node in my cluster due to high fragmentation so might follow your method. Thank you

1 0

Re: Install on Ubuntu Noble on Arm64?
by Alex Closs 28 Oct '24

28 Oct '24

Ubuntu noble *is* an LTS release, 24.04 > On Oct 28, 2024, at 06:40, Robert Sander <r.sander(a)heinlein-support.de> wrote: > > Hi > >> On 10/25/24 19:57, Daniel Brown wrote: >> Think I’ve asked this before but — has anyone attempted to use a cephadm type install with Debian Nobel running on Arm64? Have tried both Reef and Squid, neither gets very far. Do I need to file a request for it? > > You mean Ubuntu Noble, right? > > For Ceph Squid there are packages available for Ubuntu Jammy and Debian Bookworm: https://download.ceph.com/debian-squid/dists/ > > For Ceph Reef there are packages available for Ubuntu Focal, Ubuntu Jammy and Debian Bookworm: https://download.ceph.com/debian-reef/dists/ > > I do not think that the Ceph project builds packages for non-LTS distribution versions. > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Install on Debian Nobel on Arm64?
by Daniel Brown 28 Oct '24

28 Oct '24

Think I’ve asked this before but — has anyone attempted to use a cephadm type install with Debian Nobel running on Arm64? Have tried both Reef and Squid, neither gets very far. Do I need to file a request for it? myhost-01:~/ceph$ uname -a Linux cube-man-01 6.8.0-1010-raspi #11-Ubuntu SMP PREEMPT_DYNAMIC Thu Aug 8 23:22:41 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux myhost-01:~/ceph$ sudo ./cephadm add-repo --release reef Installing repo GPG key from https://download.ceph.com/keys/release.gpg... Installing repo file at /etc/apt/sources.list.d/ceph.list... Updating package list... Non-zero exit code 100 from apt-get update apt-get: stdout Hit:1 http://ports.ubuntu.com/ubuntu-ports noble InRelease apt-get: stdout Hit:2 http://ppa.launchpad.net/ansible/ansible/ubuntu noble InRelease apt-get: stdout Hit:3 http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease apt-get: stdout Hit:4 http://ports.ubuntu.com/ubuntu-ports noble-backports InRelease apt-get: stdout Hit:5 http://ports.ubuntu.com/ubuntu-ports noble-security InRelease apt-get: stdout Hit:6 https://ppa.launchpadcontent.net/ansible/ansible/ubuntu noble InRelease apt-get: stdout Hit:7 https://download.ceph.com/debian-reef jammy InRelease apt-get: stdout Ign:8 https://download.ceph.com/debian-reef noble InRelease apt-get: stdout Err:9 https://download.ceph.com/debian-reef noble Release apt-get: stdout 404 Not Found [IP: 10.20.1.1 3142] apt-get: stdout Reading package lists... apt-get: stderr E: The repository 'https://download.ceph.com/debian-reef noble Release' does not have a Release file. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/dhbrown/ceph/./cephadm/__main__.py", line 5579, in <module> File "/home/dhbrown/ceph/./cephadm/__main__.py", line 5567, in main File "/home/dhbrown/ceph/./cephadm/__main__.py", line 4575, in command_add_repo File "/home/dhbrown/ceph/./cephadm/cephadmlib/packagers.py", line 192, in add_repo File "/home/dhbrown/ceph/./cephadm/cephadmlib/packagers.py", line 213, in update File "/home/dhbrown/ceph/./cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: apt-get update: E: The repository 'https://download.ceph.com/debian-reef noble Release' does not have a Release file.

2 2

Ceph native clients
by Tim Holloway 28 Oct '24

28 Oct '24

On the whole, I prefer to use NFS for my clients to use Ceph filesystem. It has the advantage that NFS client/mount is practically guaranteed to be pre-installed on all my client systems. On the other hand, there are downsides. NFS (Ceph/NFS-Ganesha) has been known to be cranky on my network when it comes to serving resources on occasion, and should the NFS host go down, the NFS client cannot fallback to the alternate NFS host without assistance. Which I should probably add, since I run keepalive anyway, but that's another matter. My issue, then is that if I do a native/FUSE mount of Ceph FS on my desktop it hurts Ceph. The desktop hibernates when not in use and that causes the network communications between it and the Ceph servers to break until it wakes up again. In the mean time, I get "laggy OSD" errors. In extreme cases, where the desktop doesn't wake up clean and requires a hardware reset, I think it even ends up with orphan connections, since at one point I recall having to forcible terminate close to 20 connections. Ideally, then, since systemd is handling hibernation, there should be a hook in the Ceph client to ensure it and the Ceph servers are at peace with each other while it's down. Probably better reporting on the Ceph admin side as to the hostnames of clients with connection issues, but that's just icing. Tim

2 2

How Ceph cleans stale object on primary OSD failure?
by Vigneshwar S 28 Oct '24

28 Oct '24

Hi everyone, I'm studying Ceph replication and peering and have a doubt in how Ceph handles cleaning of stale objects on primary OSD failure. Scenario: Consider a placement group: PG1, whose acting set is [A, B, C] and A is the primary. t1: Client writes an object to A t2: A initiates local write t3: A sends replication request to B t4: A sends replication request to C t5: A completes local write t6: A gets replication response from B t7: A gets replication response from C t8: A is crashed and haven't sent acknowledgement to the client In case if A never comes back, the object written in B and C would be stale as it is not acknowledged to the client. How does Ceph cleans this stale object? Also, it would be helpful if you can share any documentation or articles explaining Ceph replication and peering in low-level. Regards, Vigneshwar.

4 8

Cephfs snapshot
by s.dhivagar.cse＠gmail.com 28 Oct '24

28 Oct '24

Hi.. If we enable cephfs snapshot will we face performance issue? And does snapshots take up any specific storage?

1 0

RGW Graphs in cephadm setup
by bkennedy＠cfl.rr.com 26 Oct '24

26 Oct '24

We recently upgraded all our clusters to rocky 9.4 and reef 18.2.4. Two of the clusters show the rgw metrics in the ceph dashboard and the other two don't. I made sure the firewalls were open for ceph-exporter and that Prometheus was gathering the stats on all 4 clusters. For the clusters that are not working, when I do a search on the Prometheus web console at port 9095 on the performance monitoring node using "{instance="rgw1"}, I noticed that the instance_id shows a long number. I did the same search on the clusters that show rgw stats in grafana and they have an instance_id that looks like the drop down in the Grafana dashboard. E.g. "rgw.rgw.rgw1.biewae". So it seems that for some reason, on the clusters that are not working, the ceph-exporter running on the rgw nodes is not pulling the proper instance_id rgw instance. Is there a way to fix the instance_id that ceph-exporter is pulling? I can see this instance id when browsing to the servers http://rgw1:9926/metrics portal. If I run a podman ps, I can see the container running and it has the unique id .biewae showing. I used cephadm to enter the ceph-exporter container on the rgw to see what the logs were saying and I see the same thing repeated: 1 pid path is empty; process metrics won't be fetched for: ceph-client.rgw.rgw.rgw1.biewae.2.94759457265864 Its only the rgw's that show this error in the ceph-exporter log file, the mons and osd nodes show no errors in the log and show up in grafana properly with data. I redeployed ceph-exporter and rgw to the same node in that order, but no change. I did set the default image container to 18.2.4 prior to redeploy cause I ran into that issue as well. Regards, -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw

3 3

Availability of Ceph Reef in Debian 12(bookworm) for arm64
by Karun Josy 26 Oct '24

26 Oct '24

Hello, By default, Debian 12.7 bookwork has only the Pacific (16.x) package for cephadm installation. Is there any plan to have reef or squid for arm64 in Debian 12.x? As a workaround to install ceph, I used curl method to download the cephadm binary from "rpm-reef" $ curl --silent --remote-name --location https://download.ceph.com/rpm-reef/el9/noarch/cephadm $ chmod +x cephadm $ sudo ./cephadm add-repo --version 18.2.4 This helps to create a ceph cluster in Reef version.

2 1

centos9 or el9/rocky9
by Marc 25 Oct '24

25 Oct '24

Sorry for posting off topic, a bit to lazy to create yet another account somewhere. I still need to make this upgrade to different os. I have now some vms on centos9 stream. What annoys me a lot is that tcp wrapper support is not default added to ssh. (I am using auto fed dns blacklists to refuse access) Can anyone tell me if this is the same in el9/rocky9?

5 5

pgs not deep-scrubbed in time and pgs not scrubbed in time
by Götz Reinicke 25 Oct '24

25 Oct '24

Hello Ceph Community, My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well. But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons. Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now. Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again? Thanks for hints and suggestion . Best regards Götz

6 6

IO500 SC24 List Call for Submissions
by IO500 Committee 24 Oct '24

24 Oct '24

Call for Submission Submission Deadline: Nov 10th, 2024 AoE The IO500 is now accepting and encouraging submissions for the upcoming 15th semi-annual IO500 Production and Research lists, in conjunction with SC24. We are also accepting submissions to both the Production and Research 10 Client Node Challenges to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Webpage. The new ranked lists will be announced at the BoF [1]. We look forward to seeing many new results! Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance and the documentation of how the numbers are achieved. Support small to extreme-scale Research and Production HPC systems using flexible storage APIs Maximizing simplicity in running the benchmark suite while offering tunable parameters Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes runs with highly prescribed parameters in an attempt to determine a lower performance bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported Storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. The goals of the community are also multi-fold: Gather historical data for the sake of analysis and to aid predictions of storage futures Collect tuning information to share valuable performance optimizations across the community Encourage vendors and designers to optimize for workloads beyond "hero runs" Establish bounded expectations for users, procurers, and administrators Understand and be able to reproduce performance on storage systems Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. This year, since there are no changes to the benchmark, we are skipping the stabilization period and just releasing the final benchmark. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather We encourage you to submit [2] to join our community, and to attend the SC’24 BoF [1] on Tuesday, Nov 19, 2024 at 12:15pm - 1:15pm EST, where we will announce the new IO500 Production and Research lists and their 10 Client Node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. We will also highlight new and interesting results with invited talk(s) at the BoF. [1] https://io500.org/pages/bof-sc24 [2] https://io500.org/submission -- The IO500 Committee

1 0

Issue with Filter in Lifecycle policy
by mahnoosh shahidi 24 Oct '24

24 Oct '24

Hi everyone, I'm trying to set a lifecycle policy to change the storage class of objects based on their size. Here's the rule I'm using: > { > "Rules": [ > { > "ID": "migrating storage class", > "Filter": > { > "ObjectSizeGreaterThan": 10000000 > }, > "Status": "Enabled", > "Transitions": [ > { > "Days": 0, > "StorageClass": "REPLICATED" > } > ] > } > ] > } The issue is that the rule seems to be applying to all objects, regardless of their size. Is the ObjectSizeGreaterThan filter supported in Ceph, or am I missing something in the configuration? Thanks for your help! Best regards, Mahnoosh

2 1

PG autoscaler taking too long
by AJ_ sunny 24 Oct '24

24 Oct '24

Hello team, I have one small size ceph cluster in production having 53 SSD of 7TB each with 6 node Version:- octopus In last two days we just cleared out ~100 tb of data out of 370tb total size So there is bunch of pg with active+clean+snaptrim & snaptrim_wait state comes into the action after that one of the pool placement group number decrease by autoscaler from 2048 to 512 And autoscaler showing remaining time of 4y which is too long can you guys help me out how to fix this issue root@nvme1:~# ceph -s cluster: id: c30f5720-ca5c-11ec-b19c-f9781f61e8ec health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 34 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum nvme3,nvme2,nvme1 (age 14M) mgr: nvme2.ttfore(active, since 13M), standbys: nvme3.zhnsuf osd: 53 osds: 53 up (since 2h), 53 in (since 11d); 61 remapped pgs flags noscrub,nodeep-scrub task status: data: pools: 7 pools, 2784 pgs objects: 26.37M objects, 69 TiB usage: 177 TiB used, 194 TiB / 370 TiB avail pgs: 1011732/79097316 objects misplaced (1.279%) 2723 active+clean 58 active+remapped+backfill_wait 3 active+remapped+backfilling io: client: 78 KiB/s rd, 3.8 MiB/s wr, 238 op/s rd, 314 op/s wr recovery: 17 MiB/s, 6 objects/s progress: PG autoscaler decreasing pool 8 PGs from 2048 to 512 (1h) [............................] (remaining: 4y) root@nvme1:~# Thanks Arihant Jain

2 2

High number of Cephfs Subvolumes compared to Cephfs persistent volumes in K8S environnement
by Edouard FAZENDA 23 Oct '24

23 Oct '24

Dear Ceph Community, Maybe you can help me with this I have inconsistencies with the number of subvolumes in Cephfs in my Ceph Cluster and the number of Persistent Volumes of StorageClass Cephfs in my K8S cluster I have K8S 1.25 version on my kubernetes cluster and use the Ceph CSI CephFs 3.8.1, my ceph cluster is 16.2.15 pacific release. I have 35 Persistent Volumes of type CephFS $ kubectl get pv -o custom-columns=name:.metadata.name,subvolume:.spec.csi.volumeAttributes.subv olumeName,storageclass:.spec.storageClassName | grep cephfs pvc-0062e845-4a87-4a32-8f19-cbfff3b2789d csi-vol-919a2302-f964-11ed-b5e0-b615b8ae8847 csi-cephfs-sc pvc-0298fc13-bc5f-489a-b88e-64d7cdc61f1e csi-vol-7ce2f152-d96c-4f54-b335-9ede672ce320 csi-cephfs-sc pvc-09991b1e-b9f6-4a7c-b710-25131433c6e9 csi-vol-ef2e7be0-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc pvc-19b8f683-b3e8-4476-abc3-9d7d7e4d8941 csi-vol-5a9ce4b8-1edb-11ed-8f9a-e6ec00c2f5e5 csi-cephfs-sc pvc-2fc90e06-22b4-4e80-b33a-c4c5dd6a0baa csi-vol-760f34d1-4e26-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-316aa495-3d52-45a5-aac3-4f64bf598aa3 csi-vol-9d135b18-8b43-402b-a606-a2c426527b20 csi-cephfs-sc pvc-342b66f9-f6d5-413c-8634-e51a25d638a8 csi-vol-7e3aca09-8d76-11eb-95b9-ee71e85dde57 csi-cephfs-sc pvc-48c664a6-073e-4498-9370-d4a718f11729 csi-vol-72d71655-9518-11ee-a81d-b2c17fca41c9 csi-cephfs-sc pvc-5349d4ec-7e62-48b7-bd8c-8f718a13906b csi-vol-9748018f-4df9-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-5db458a4-2dfb-46d7-8b23-2bde45a9bf46 csi-vol-bd87eade-e726-4a5d-993c-56dbece42420 csi-cephfs-sc pvc-5fd1f3a5-d120-4283-8e2c-b6e1e3e4cbd1 csi-vol-e7ec3159-9d1c-11eb-82b9-aaa0cbaeebfa csi-cephfs-sc pvc-669c35f9-9927-4e44-94d2-a9ddd0eda914 csi-vol-ef9539f6-9c5a-40d8-a89b-087ad7c6e9b3 csi-cephfs-sc pvc-6909652a-2da6-4a13-a111-59a21ed685b2 csi-vol-a9a1fff7-550a-43a1-aa52-4f6710e1ca9d csi-cephfs-sc pvc-72c2f70e-2976-4188-8d1a-f2928745097b csi-vol-51045501-f17b-4f4d-af08-100ea3db3db3 csi-cephfs-sc pvc-73b1d3f3-f861-44d7-bbd0-43fc16f9e7a9 csi-vol-439769d7-9b94-4350-bfa2-fe5e9a182650 csi-cephfs-sc pvc-76b04010-1e00-4fea-82b7-855ff83ab820 csi-vol-28852a9b-79c0-4eb7-afcf-b443ff9f867d csi-cephfs-sc pvc-7e0b08ea-9d31-45a7-a098-588563c18f19 csi-vol-4a5086d9-62ad-11ee-80a7-323a25995550 csi-cephfs-sc pvc-83853b0b-ab55-46be-ae3a-78f8d431e780 csi-vol-ef2e919c-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc pvc-89c85899-c26a-4efd-91dd-cc891d303ad6 csi-vol-e8b89f50-fdf9-11ec-8f9a-e6ec00c2f5e5 csi-cephfs-sc pvc-8a4c12b1-b7d8-4337-b218-7de2bd6eadde csi-vol-718d31a6-9b4a-11ee-a81d-b2c17fca41c9 csi-cephfs-sc pvc-8b56ae01-1809-4552-9192-930bd1db95f8 csi-vol-a72f2d4c-1f8b-4a71-a846-2b7a6febbf82 csi-cephfs-sc pvc-97b80840-fd60-436a-8223-5cf41bfe56ef csi-vol-0289ed46-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-984eadce-c908-4e21-a558-70d12bcff3c2 csi-vol-ef2e8a6b-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc pvc-9de21d9d-80ae-49f8-bea0-2a265945cfa6 csi-vol-c7b4f302-4d71-4777-b6c0-82720ebe8950 csi-cephfs-sc pvc-9f5651b5-0cb2-47d2-889e-69b09d96fe36 csi-vol-9ba650df-3fa5-43f0-abea-5231643fb1bf csi-cephfs-sc pvc-a2607a68-62a9-47a0-8f15-871359460556 csi-vol-028a40c4-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-a84158f8-1d21-475a-9987-a0e8a3347c42 csi-vol-22835b9b-c31c-46c6-aec2-67b554c683d3 csi-cephfs-sc pvc-b3d81d90-eaec-4d8a-8065-790d9c0e6ea7 csi-vol-c908a216-cf78-11eb-a202-52b7fddc54da csi-cephfs-sc pvc-bcc5979b-f9cb-4c10-ab64-24424749856b csi-vol-e8dd361e-b6fb-4a19-94af-7cae07d4702c csi-cephfs-sc pvc-bd09916e-a47d-429f-9371-ac779d3959b3 csi-vol-cc9573bc-6733-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-dac9a949-cd28-4448-9c66-7bab365c3d25 csi-vol-21b0d822-6fc3-11ec-96cb-dec0a2aa44f8 csi-cephfs-sc pvc-e0783e40-f49c-4d7c-a404-4554ce35341c csi-vol-0289e975-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-f660eb28-2af4-45de-b00b-72f73e08745d csi-vol-4b6e4fd4-9e4e-4b0d-a022-305b8e1d8e84 csi-cephfs-sc pvc-fd994df6-9ca5-47d5-af64-afec037fd5d1 csi-vol-ef2e42e3-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc pvc-fe72f0a2-c260-4a40-acca-50f47d31c11a csi-vol-6c0efe87-d02c-11eb-a202-52b7fddc54da csi-cephfs-sc I have 985 subvolumes in Cephfs csi volume , i have no idea on why i have this difference. $ ceph fs subvolume ls cephfs csi | awk '{if ($2 != "") print $2}' | sed 's/"//g' | wc -l 1020 If i check the number of Volume Snapshots of StorageClass Cephfs via the command "kubectl get volumesnapshot -A | grep cephfs | wc -l" i have 177 If i check from the subvolumes of Cephfs that are mapped to the K8S PV i have 178 via the command "for subvolume in $(cat list.txt | xargs);do ceph fs subvolume snapshot ls cephfs $subvolume csi | awk '{if ($2 != "") print $2}' | sed 's/"//g' ;done | wc -l" I have 178 Which seems to be correct and consistent. Are my checks valid to identifies subvolumes that are not anymore used by my K8S workload ? Can i safely delete the subvolumes as they seems not associated with a persistent volume in Kubernetes ? Thanks in advance for the help Have a nice day Best Regards, Edouard Fazenda. Edouard FAZENDA Technical Support <https://www.csti.ch/> www.csti.ch

2 1

Issue with Recovery Throughput Not Visible in Ceph Dashboard After Upgrade to 19.2.0 (Squid)
by Sanjay Mohan 23 Oct '24

23 Oct '24

Dear Ceph Users, I hope this message finds you well. I recently performed an upgrade of my Ceph cluster, moving through the following versions: * 17.2.7 >> 18.2.0 >> 18.2.2 >> 18.2.4 >> 19.2.0 (Squid) After successfully upgrading to Ceph 19.2.0, I noticed an issue where the recovery throughput is no longer visible in the new Ceph dashboard. However, Old dashboard metrics and features seem to be working as expected. It is important to note that the recovery throughput was displayed properly in the previous version of the Ceph dashboard. I am using Cephadm for the installation, not Rook. Current behavior: * Recovery throughput metrics are not displayed in the new dashboard after upgrading to 19.2.0. Expected behavior: * The recovery throughput should be visible, as it was in previous versions of the Ceph dashboard. I am reaching out to inquire if there are any known issues, workarounds, or upcoming fixes for this. Your assistance in this matter would be greatly appreciated. Thank you for your time and support. I look forward to hearing from you soon. Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan(a)am.amrita.edu [Amrita University] Disclaimer: The information transmitted in this email, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham. If you received this in error, please contact the sender and destroy any copies of this information.

5 9

Reef osd_memory_target and swapping
by Dave Hall 23 Oct '24

23 Oct '24

Hello. I'm seeing the following in the Dashboard -> Configuration panel for osd_memory_target: Default: 4294967296 Current Values: osd: 9797659437, osd: 10408081664, osd: 11381160192, osd: 22260320563 I have 4 hoists in the cluster right now - all OSD+MGR+MON. 3 have 128GB RAM, the 4th has 256GB. On the host with 256GB, top shows some OSD processes with very high VIRT and RES values - the highest VIRT OSD has 13.0g. The highest RES is 8.5g. All 4 systems are currently swapping, but the 256GB system has much higher swap usage. I am confused why I have 4 current values for osd_memory_target, and especially about the 4th one at 22GB. Also, I'm recalling that there might be a recommendation to disable swap. and I could easily do 'swapoff -a' when the swap usage is lower than the free RAM. Can anybody shed any light on this? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

7 8

Squid Manager Daemon: balancer crashing orchestrator and dashboard
by laimis.juzeliunas＠oxylabs.io 22 Oct '24

22 Oct '24

Hello community, We are facing one issue after migrating from Reef 18.4.2 to Squid 19.2.0 with the Ceph manager daemon and were wondering if anyone has already faced this or could guide us where to look further. When turning on balancer (upmap mode) it hangs our mgr completely most of the time and leaves orchestrator as well as the dashboard/UI unresponsive. We noticed this initially during the upgrade (as mgr is the first in line) and had to continue with the balancer turned off. Post upgrade we still need to keep it turned off. We run docker daemons with cephadm as the orchestrator, health status is always HEALTH_OK. A few observations: ceph balancer on -> mgr logs show some pg_upmap's being performed debug logs show that balancer is done -> mgr stops working only pgmap debug logs remain on container after some (10-20 minutes) time mgr fails over to standby node mgr starts and begins full cluster inventory (services, disks, daemons, networks, etc) dashboard starts up, orch commands begin working balancer kicks in, performs some upmaps and the cycle continues failing over to standby node afrer quite some time while mgr is not working TCP port is showing as being listened to (netstat), but does not even respond to telnet. but when mgr is working - we can query it with curl Cluster overview: cluster: id: 96df99f6-fc1a-11ea-90a4-6cb3113cb732 health: HEALTH_OK services: mon: 5 daemons, quorum ceph-node004,ceph-node003,ceph-node001,ceph-node005,ceph-node002 (age 3d) mgr: ceph-node001.hgythj(active, since 24h), standbys: ceph-node002.jphtvg mds: 21/21 daemons up, 12 standby osd: 384 osds: 384 up (since 2d), 384 in (since 7d) rbd-mirror: 2 daemons active (2 hosts) rgw: 64 daemons active (32 hosts, 1 zones) data: volumes: 2/2 healthy pools: 16 pools, 12793 pgs objects: 751.20M objects, 1.4 PiB usage: 4.5 PiB used, 1.1 PiB / 5.6 PiB avail pgs: 12410 active+clean 285 active+clean+scrubbing 98 active+clean+scrubbing+deep io: client: 5.8 GiB/s rd, 169 MiB/s wr, 43.77k op/s rd, 9.92k op/s wr We will be able to provide a more detailed log sequence but for now we see these entries: debug 2024-10-18T13:23:34.478+0000 7fad971ab640 0 [balancer DEBUG root] Waking up [active, now 2024-10-18_13:23:34] debug 2024-10-18T13:23:34.478+0000 7fad971ab640 0 [balancer DEBUG root] Running debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer INFO root] Optimize plan auto_2024-10-18_13:23:34 debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer INFO root] Mode upmap, max misplaced 0.050000 debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer DEBUG root] unknown 0.000000 degraded 0.000000 inactive 0.000000 misplaced 0 debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer INFO root] do_upmap debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer INFO root] pools ['cephfs.our-awesome-pool1.meta', 'our-awesome-k8s.ceph-csi.rbd', 'cephfs.our-awesome-poo2l.data', 'cephfs.our-awesome-pool2.meta', 'europe-1.rgw.buckets.non-ec', 'europe-1.rgw.buckets.data', 'our-awesome-k8s2.ceph-csi.rbd', 'europe-1.rgw.log', '.mgr', 'our-awesome-k8s4.ceph-csi.rbd', 'europe-1.rgw.control', 'our-awesome-k8s3.ceph-csi.rbd', 'europe-1.rgw.meta', 'europe-1.rgw.buckets.index', '.rgw.root', 'cephfs.our-awesome-pool1.data'] debug 2024-10-18T13:23:34.814+0000 7fad971ab640 0 [balancer INFO root] prepared 10/10 upmap changes debug 2024-10-18T13:23:34.814+0000 7fad971ab640 0 [balancer INFO root] Executing plan auto_2024-10-18_13:23:34 debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.148 mappings [{'from': 113, 'to': 94}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.302 mappings [{'from': 138, 'to': 128}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.34e mappings [{'from': 92, 'to': 89}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.504 mappings [{'from': 156, 'to': 94}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.836 mappings [{'from': 148, 'to': 54}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.e4c mappings [{'from': 157, 'to': 54}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.e56 mappings [{'from': 147, 'to': 186}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.e63 mappings [{'from': 79, 'to': 31}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.ed9 mappings [{'from': 158, 'to': 237}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.f1e mappings [{'from': 153, 'to': 237}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer DEBUG root] commands [<mgr_module.CommandResult object at 0x7fae2e05b700>, <mgr_module.CommandResult object at 0x7fad2d889ca0>, <mgr_module.CommandResult object at 0x7fad2d810340>, <mgr_module.CommandResult object at 0x7fad2d88c5e0>, <mgr_module.CommandResult object at 0x7fad2d418130>, <mgr_module.CommandResult object at 0x7fad2d418ac0>, <mgr_module.CommandResult object at 0x7fae2f8d7cd0>, <mgr_module.CommandResult object at 0x7fad2d448520>, <mgr_module.CommandResult object at 0x7fad2d4480d0>, <mgr_module.CommandResult object at 0x7fad2d448fa0>] 162.55.93.25 - - [18/Oct/2024:13:23:35] "GET /metrics HTTP/1.1" 200 7731011 "" "Prometheus/2.43.0" ... debug 2024-10-18T13:23:36.110+0000 7fad971ab640 0 [balancer DEBUG root] done We were suspecting that this might be caused by older ceph-clients connecting and getting identified (wrongly) as Luminous by Ceph due to the fresh upmap-read balancer mode (I think it came with Squid but I might be wrong) that had to do something in the background even when disabled. However setting set-require-min-compat-client did not help our case so we dismissed this assumption. We would be beyond happy for any advice where to look further as having no balancer is sad. If anyone would like to go through detailed logs we are glad to provide. Best, Laimis J.

3 2

CSC Election: Governance amendments and Ceph Executive Council Nominations
by Patrick Donnelly 21 Oct '24

21 Oct '24

The Ceph Steering Committee is voting on a number of governance amendments and nominations to the Ceph Executive Council. The election is public and viewable here: https://vote.heliosvoting.org/helios/elections/e03494ce-e04c-41d0-bb05-ec5c… The election closes on October 28th, 3pm UTC. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Ceph RGW performance guidelines
by Harry Kominos 21 Oct '24

21 Oct '24

Hello Ceph Community! I have the following very interesting problem, for which I found no clear guidelines upstream so I am hoping to get some input from the mailing list. I have a 6PB cluster in operation which is currently half full. The cluster has around 1K OSD, and the RGW data pool has 4096 pgs (and pgp_num). The issue is as follows: Let's say that we have 10 million small objects (4MB) each. 1)Is there a performance difference *when fetching* between storing all 10 million objects in one bucket and storing 1 million in 10 buckets? There should be "some" because of the different number of pgs in use, in the 2 scenarios but it is very hard to quantify. 2) What if I have 100 million objects? Is there some theoretical limit / guideline on the number of objects that I should have in a bucket before I see performance drops? I should mention here that the contents of the bucket *never need to be listed, *The user always knows how to do a curl, to get the contents. Thank you for your help, Harry P.S. The following URLs have been very informative, but they do not answer my question unfortunately. https://www.redhat.com/en/blog/red-hat-ceph-object-store-dell-emc-servers-p… https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond

2 5

How to Speed Up Draining OSDs?
by Alex Hussein-Kershaw (HE/HIM) 21 Oct '24

21 Oct '24

Hi Folks, I'm trying to scale-in a Ceph Cluster. It's running 19.2.0 and is cephadm managed. It's just a test system, so has basically no data and only has 3 OSDs. As part of the scaling-in, I run "ceph orch host drain <hostname> --zap-osd-devices" as per Host Management — Ceph Documentation<https://docs.ceph.com/en/reef/cephadm/host-management/#removing-hosts>. That starts off the OSD draining. However, that drain seems to take an enormous amount of time. My OSD has less than 100MiB raw storage, and I let it run for 2 hours over lunch and it still was not finished, so I cancelled it. I'm not sure how this scales, but I'm assuming at least linearly with data stored, which seems like bad news for doing this on real systems, which may have several TBs per OSD. I had a look at the recovery profiles documentation here mClock Config Reference — Ceph Documentation<https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/> which seemed to indicate I could speed this up (but my impression was maybe I could get a speed up of 2x which seems like it will still take an age). On the other hand, just switching off the host running the OSD and doing an offline host removal ("ceph orch host rm <hostname> --offline") seems much easier, with the trade-off that the Cluster recovers after the loss of the OSD rather than pre-emptively. But that big risk of that seems to be mitigated by "ceph orch host ok-to-stop <hostname>" to check I won't cause any PGs to go offline before hand. Are there any tricks here that I'm missing? Thanks, Alex

2 4

Unable to mount NFS share NFSv3 on windows client.
by Saif Mohammad 21 Oct '24

21 Oct '24

Hello Everyone, We are encountering issues while trying to mount NFSv3-based exports on a Windows 11 client. We made the necessary changes to the configuration file to enable NFSv3 exports, but unable to mount. Below are the details: We created the custom config file named export.conf and applied the changes as follow. ------ { "export_id": 1, "path": "/volumes", "cluster_id": "nfs-cephfs", "pseudo": "/ceph", "access_type": "RW", "squash": "root_squash", "security_label": false, "Enable_UDP": false, "protocols": [ 3 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "cephfs01" }, "clients": [] } ------ On Client machine (Windows 11), we attempted to mount the NFS share using the following command: ------ \Users\Administrator>mount 172.16.100.14:/mnt/share S: Network Error - 53 Type 'NET HELPMSG 53' for more information. ------ we verified there is no issue with the network connectivity, For reference we are sharing the rpcinfo output: ------ C:\Users\Administrator>rpcinfo -p 192.168.2.107 program version protocol port (null) (null) -------------------------------------------------- 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs ------ kindly assist us in identifying what might be missing or suggest any troubleshooting steps? Your assistance will be greatly appreciated. Kind regards, Mohammad Saif

1 0

Re: Ubuntu 24.02 LTS Ceph status warning
by Eugen Block 18 Oct '24

18 Oct '24

Is apparmor configured differently on those hosts? Or is it running only on the misbehaving host? Zitat von Dominique Ramaekers <dominique.ramaekers(a)cometal.be>: > 'ceph config get mgr container_image' gives > quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a => > OK > > 'ceph health detail' gives > HEALTH_WARN failed to probe daemons or devices > [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices > host hvs004 `cephadm gather-facts` failed: cephadm exited with > an error code: 1, stderr: Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File > "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in > <module> > ... > File > "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in > _fetch_apparmor > ValueError: too many values to unpack (expected 2) > host hvs004 `cephadm ceph-volume` failed: cephadm exited with an > error code: 1, stderr: Inferring config > /var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/config/ceph.conf > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File > "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in > <module> > ... > File > "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in > _fetch_apparmor > ValueError: too many values to unpack (expected 2) > > I do think it's a ceph version issue, so I started to compaire the > hvs004 host with a good behaving host hvs001. I did find this: > 'root@hvs001~#:cephadm shell ceph -v' gives > ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable) > 'root@hvs004~#:cephadm shell ceph -v' gives > ceph version 19.3.0-5346-gcc481a63 > (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev) > > It seems only the shell uses a wrong docker image so I took a list > from both of them: > hvs0001 > ---------- > REPOSITORY TAG IMAGE ID CREATED > SIZE > quay.io/ceph/ceph v19 37996728e013 2 weeks > ago 1.28GB > quay.io/ceph/ceph v18.2 2bc0b0f4375d 2 months > ago 1.22GB > quay.io/ceph/ceph v18 a27483cc3ea0 6 months > ago 1.26GB > quay.io/ceph/ceph v17 5a04c8b3735d 9 months > ago 1.27GB > quay.io/ceph/ceph-grafana 9.4.7 954c08fa6188 10 > months ago 633MB > quay.io/ceph/grafana 9.4.12 2bacad6d85d8 17 > months ago 330MB > quay.io/prometheus/prometheus v2.43.0 a07b618ecd1d 19 > months ago 234MB > quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 22 > months ago 65.1MB > quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 22 > months ago 22.5MB > quay.io/ceph/ceph v17.2 0912465dcea5 2 years > ago 1.34GB > quay.io/ceph/ceph v17.2.3 0912465dcea5 2 years > ago 1.34GB > quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 2 years > ago 558MB > quay.ceph.io/ceph-ci/ceph master c5ce177c6a5d 2 years > ago 1.38GB > quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 2 years > ago 204MB > quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years > ago 20.9MB > quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 3 years > ago 57.5MB > quay.io/ceph/ceph-grafana 6.7.4 557c83e11646 3 years > ago 486MB > quay.io/prometheus/prometheus v2.18.1 de242295e225 4 years > ago 140MB > quay.io/prometheus/alertmanager v0.20.0 0881eb8f169f 4 years > ago 52.1MB > quay.io/prometheus/node-exporter v0.18.1 e5a616e4b9cf 5 years > ago 22.9MB > > hvs004 > --------- > REPOSITORY TAG IMAGE ID CREATED > SIZE > quay.ceph.io/ceph-ci/ceph main 6e76ca06f33a 11 days > ago 1.41GB > quay.io/ceph/ceph <none> 37996728e013 2 weeks > ago 1.28GB > quay.io/ceph/ceph v17 9cea3956c04b 18 > months ago 1.16GB > quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 22 > months ago 22.5MB > quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years > ago 20.9MB > > I pulled on hvs004 the v19 tagged image and my cephadm shell ceph -v > gave the correct version. > > It seems my docker images aren't automatically managed by ceph? > > Can I fix this, or do I have to pull the correct images and remove > the wrong ones myself? > > >> -----Oorspronkelijk bericht----- >> Van: Eugen Block <eblock(a)nde.ag> >> Verzonden: vrijdag 11 oktober 2024 13:03 >> Aan: ceph-users(a)ceph.io >> Onderwerp: [ceph-users] Re: Ubuntu 24.02 LTS Ceph status warning >> >> I don't think the warning is related to a specific ceph version. The >> orchestrator uses the default image anyway, you can get it via: >> >> ceph config get mgr container_image >> >> 'ceph health detail' should reveal which host or daemons misbehaves. I >> would then look into cephadm.log on that host to find more hints, what >> exactly goes wrong. You should also look into the active MGR log, >> it could also >> give you hints why that service fails. >> >> Zitat von Dominique Ramaekers <dominique.ramaekers(a)cometal.be>: >> >> > I manage a 4 hosts cluster on Ubuntu 22.04 LTS with ceph installed >> > trough cephad and containers on Docker. >> > >> > Last month, I've migrated to the latest Ceph 19.2. All went great. >> > >> > Last week I've upgraded one of my hosts to Ubuntu 24.04.1 LTS Now I >> > get the following warning in cephadm shell -- ceph status: >> > Failed to apply 1 service(s): osd.all-available-devices failed to >> > probe daemons or devices >> > >> > Outside the ceph shell: >> > Ceph -v results in 'ceph version 19.2.0~git20240301.4c76c50 >> > (4c76c50a73f63ba48ccdf0adccce03b00d1d80c7) squid (dev)' >> > >> > Inside the shell: 'ceph version 19.3.0-5346-gcc481a63 >> > (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)' >> > All osd's, mon's , mgr's and mds's are on 19.2.0` (image id >> > 37996728e013) >> > >> > Do I get the warning because the Ubuntu package of ceph is still on a >> > development version? >> > Or can I have another underling problem? >> > >> > Thanks for the help. >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> > email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email >> to ceph-users-leave(a)ceph.io

2 2

Procedure for temporary evacuation and replacement
by Frank Schilder 18 Oct '24

18 Oct '24

Hi all, a hopefully simple question this time. I would like a second opinion on a procedure for replacing a larger number of disks. We need to replace about 40 disks distributed over all 12 hosts backing a large pool with EC 8+3. We can't do it host by host as it would take way too long (replace disks per host and let recovery rebuild the data). Therefore, we would like to evacuate all data from these disks simultaneously and with as little data movement as possible. This is the procedure that seems to do the trick: 1.) For all OSDs: ceph osd reweight ID 0 # Note: not "osd crush reweight" 2.) Wait for rebalance to finish 3.) Replace disks and deploy OSDs with the same IDs as before per host 4.) Start OSDs and let rebalance back I tested step 1 on Octopus with 1 disk and it seems to work. The reason I ask is that step 1 actually marks the OSDs as OUT. However, they are still UP and I see only misplaced objects, not degraded objects. It is a bit counter-intuitive, but it seems that UP+OUT OSDs still participate in IO. Because it is counter-intuitive, I would like to have a second opinion. I have read before that others reweight to something like 0.001 and hope that this flushes all PGs. I would prefer not to rely on hope and a reweight to 0 apparently is a valid choice here, leading to a somewhat weird state with UP+OUT OSDs. Problems that could arise are timeouts I'm overlooking that will make data chunks on UP+OUT OSDs unavailable after some time. I'm also wondering if UP+OUT OSDs participate in peering in case there is an OSD restart somewhere in the pool. Thanks for your input and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

7 16

Lifecycle Stuck PROCESSING and UNINITIAL
by Reid Guyett 18 Oct '24

18 Oct '24

Hello, I am experiencing an issue where it seems all lifecycles are showing either PROCESSING or UNINITIAL. > # radosgw-admin lc list > [ > { > "bucket": > ":tesra:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.833499554.20", > "shard": "lc.0", > "started": "Thu, 17 Oct 2024 00:00:01 GMT", > "status": "PROCESSING" > }, > { > "bucket": > ":primevideos:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.311333334.886", > "shard": "lc.3", > "started": "Wed, 16 Oct 2024 00:00:01 GMT", > "status": "PROCESSING" > }, > { > "bucket": > ":editorimages:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.311333334.3", > "shard": "lc.4", > "started": "Wed, 16 Oct 2024 00:00:01 GMT", > "status": "PROCESSING" > }, > { > "bucket": > ":osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1", > "shard": "lc.10", > "started": "Thu, 17 Oct 2024 00:00:01 GMT", > "status": "PROCESSING" > }, > { > "bucket": > ":projects0609:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.856877269.1147", > "shard": "lc.10", > "started": "Thu, 01 Jan 1970 00:00:00 GMT", > "status": "UNINITIAL" > }, > ... > I turned up the log level for rgw and see a most have some sort of error: "failed to put head" or "returned error ret==-2" > 2024-10-17T15:08:31.720+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.10 head last stored at Mon Sep > 23 00:00:00 2024 > 2024-10-17T15:08:31.720+0000 7fc9cf968640 16 lifecycle: > RGWLC::expired_session started: 1729123201 interval: 86400(*2==172800) now: > 1729177711 > 2024-10-17T15:08:31.720+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ACTIVE entry: > :osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1::1729123201:1 > index: 10 worker ix: 0 > 2024-10-17T15:08:31.744+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > failed to put head lc.10 > 2024-10-17T15:08:31.779+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 2 worker ix: 0 > 2024-10-17T15:08:31.781+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.2 head last stored at Wed Jun > 19 00:00:02 2024 > 2024-10-17T15:08:31.781+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.782+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 23 worker ix: 0 > 2024-10-17T15:08:31.783+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.23 head last stored at Mon Jan > 8 00:00:00 2024 > 2024-10-17T15:08:31.784+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.784+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 22 worker ix: 0 > 2024-10-17T15:08:31.786+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.22 head last stored at Mon Aug > 19 00:00:00 2024 > 2024-10-17T15:08:31.786+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.787+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 17 worker ix: 0 > 2024-10-17T15:08:31.788+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.17 head last stored at Mon Jul > 22 00:00:00 2024 > 2024-10-17T15:08:31.788+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.789+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 7 worker ix: 0 > 2024-10-17T15:08:31.790+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.7 head last stored at Sat Jun > 8 00:00:02 2024 > 2024-10-17T15:08:31.791+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.791+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 0 worker ix: 0 > 2024-10-17T15:08:31.793+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.0 head last stored at Mon Sep > 23 00:00:00 2024 > 2024-10-17T15:08:31.793+0000 7fc9cf968640 16 lifecycle: > RGWLC::expired_session started: 1729123201 interval: 86400(*2==172800) now: > 1729177711 > 2024-10-17T15:08:31.793+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ACTIVE entry: > :tesra:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.833499554.20::1729123201:1 > index: 0 worker ix: 0 > 2024-10-17T15:08:31.794+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > failed to put head lc.0 > 2024-10-17T15:08:31.795+0000 7fc9cf968640 2 lifecycle: life cycle: stop > I don't see any errors when running the process command but the status doesn't update to COMPLETE: > # radosgw-admin lc process --bucket osbackup --debug_rgw 10/5 > 2024-10-17T15:13:32.826+0000 7f59f15c0840 4 Realm: realname > (5e3b0d1c-78de-499a-9fa3-23e2f9142ef7) > 2024-10-17T15:13:32.826+0000 7f59f15c0840 4 ZoneGroup: us > (1c370840-4d35-4ada-85a4-14bd6eeb2b0a) > 2024-10-17T15:13:32.826+0000 7f59f15c0840 4 Zone: zonename > (5e9bc383-f7bd-4fd1-b607-1e563bfe0011) > 2024-10-17T15:13:32.826+0000 7f59f15c0840 4 using period configuration: > 69f026df-8fa4-42c8-8f62-1b0a902db797:12 > 2024-10-17T15:13:32.900+0000 7f59f15c0840 2 all 8 watchers are set, > enabling cache > 2024-10-17T15:13:32.923+0000 7f59d7536640 2 rgw data changes log: > RGWDataChangesLog::ChangesRenewThread: start > 2024-10-17T15:13:33.609+0000 7f59f15c0840 10 rgw notify: Started > notification manager with: 1 workers > 2024-10-17T15:13:33.610+0000 7f59f15c0840 10 cache get: > name=zonename.rgw.meta+root+osbackup : miss > 2024-10-17T15:13:33.611+0000 7f59f15c0840 10 cache put: > name=zonename.rgw.meta+root+osbackup info.flags=0x11 > 2024-10-17T15:13:33.611+0000 7f59f15c0840 10 > adding zonename.rgw.meta+root+osbackup to cache LRU end > 2024-10-17T15:13:33.611+0000 7f59f15c0840 10 cache get: > name=zonename.rgw.meta+root+.bucket.meta.osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1 > : miss > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 cache put: > name=zonename.rgw.meta+root+.bucket.meta.osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1 > info.flags=0x17 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 > adding zonename.rgw.meta+root+.bucket.meta.osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1 > to cache LRU end > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 updating xattr: > name=ceph.objclass.version bl.length()=42 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 updating xattr: > name=user.rgw.acl bl.length()=153 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 updating xattr: name= > user.rgw.lc bl.length()=208 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 chain_cache_entry: > cache_locator=zonename.rgw.meta+root+.bucket.meta.osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 5 lifecycle: > RGWLC::process_bucket(): ENTER: index: 10 worker ix: 0 > 2024-10-17T15:13:33.641+0000 7f59f15c0840 5 lifecycle: > RGWLC::process_bucket(): ACTIVE entry: > :osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1::1729123201:1 > index: 10 worker ix: 0 > 2024-10-17T15:13:33.815+0000 7f59d7536640 2 rgw data changes log: > RGWDataChangesLog::ChangesRenewThread: start > 2024-10-17T15:13:33.867+0000 7f59f15c0840 2 removed watcher, disabling > cache > I do see that one of the buckets doesn't exist but still is in the lc list. > # radosgw-admin lc list | grep projects0609 > "bucket": > ":projects0609:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.856877269.1147", > # radosgw-admin bucket stats --bucket projects0609 > failure: (2002) Unknown error 2002: Is it possible to rm an LC entry from the LC list if the bucket doesn't exist? Is there anything else to check? I saw https://tracker.ceph.com/issues/68160 and the related tickets in the comments. Maybe it is related. Our RGWs are running in 18.2.4 official containers but our cluster is 17.2.7 debian packages. We upgraded the RGWs due to constant crashing. Thanks, Reid

2 1

Help with "27 osd(s) are not reachable" when also "27 osds: 27 up.. 27 in"
by Harry G Coin 17 Oct '24

17 Oct '24

I need help to remove a useless "HEALTH ERR" in 19.2.0 on a fully dual stack docker setup with ceph using ip v6, public and private nets separated, with a few servers. After upgrading from an error free v18 rev, I can't get rid of the 'health err' owing to the report that all osds are unreachable. Meanwhile ceph -s reports all osds up and in and the cluster otherwise operates normally. I don't care if it's 'a real fix' I just need to remove the false error report. Any ideas? Thanks Harry Coin

5 7

cephadm bootstrap ignoring --skip-firewalld
by Kozakis, Anestis 17 Oct '24

17 Oct '24

As I mentioned in my earlier e-mail, new to Ceph, and trying to set up automation to deploy, configure, and manage a Ceph cluster. We configure our Firewall rules through SaltStack. I am passing the -skip-firewalld option to the cephadm bootstrap command, but cephadm seems to ignore the option and configures the firewall anyway. I have even reconfigured the options order to be the same as cephadm boostrap -help but it still ignores the option and configures the firewall. This creates issues as it configures the public zone, which we don't want changed. Below is the command we are using (with obvious settings changed/removed). cephadm bootstrap --mon-ip 10.0.0.0 --mgr-id host.domain.name --fsid [fsid] --ssh-private-key id_rsa --ssh-public-key id_rsa.pub --ssh-user [user] --cluster-network 192.168.0.0/25 --allow-fqdn-hostname --config=./ceph.conf --initial-dashboard-user admin --initial-dashboard-password SuperS3cr3tPassw0rd --dashboard-password-noupdate --skip-firewalld --with-centralized-logging --apply-spec=spec.yaml What am I missing? Anestis Kozakis Systems Administrator - Multi-Level Security Solutions P: + 61 2 6122 0205 M: +61 4 88 376 339 anestis.kozakis(a)raytheon.com.au<mailto:anestis.kozakis@raytheon.com.au> Raytheon Australia Cybersecurity and Information Assurance 4 Brindabella Cct Brindabella Business Park Canberra Airport, ACT 2609 www.raytheonaustralia.com.au<http://www.raytheonaustralia.com.au/> LinkedIn<https://au.linkedin.com/company/raytheon-australia> | Twitter<https://twitter.com/RaytheonAU> | Facebook<https://www.facebook.com/RaytheonAustralia> | Instagram<https://www.instagram.com/raytheonaustralia/>

2 3

Ubuntu 24.02 LTS Ceph status warning
by Dominique Ramaekers 16 Oct '24

16 Oct '24

I manage a 4 hosts cluster on Ubuntu 22.04 LTS with ceph installed trough cephad and containers on Docker. Last month, I've migrated to the latest Ceph 19.2. All went great. Last week I've upgraded one of my hosts to Ubuntu 24.04.1 LTS Now I get the following warning in cephadm shell -- ceph status: Failed to apply 1 service(s): osd.all-available-devices failed to probe daemons or devices Outside the ceph shell: Ceph -v results in 'ceph version 19.2.0~git20240301.4c76c50 (4c76c50a73f63ba48ccdf0adccce03b00d1d80c7) squid (dev)' Inside the shell: 'ceph version 19.3.0-5346-gcc481a63 (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)' All osd's, mon's , mgr's and mds's are on 19.2.0` (image id 37996728e013) Do I get the warning because the Ubuntu package of ceph is still on a development version? Or can I have another underling problem? Thanks for the help.

3 5

osd won't start
by Erwin Bogaard 16 Oct '24

16 Oct '24

Hi, we're experiencing issues with a few osd's. They had a crash, but now won't start anymore. Nothing seems wrong with them, but they keep hanging with apparent 100% i/o wait on the machine when starting the osd. This is on ceph 18.2.4 This is the log (edited a bit, as it's too long): 2024-10-16T13:48:05.075+0200 7fc771a10640 0 set uid:gid to 167:167 (ceph:ceph) 2024-10-16T13:48:05.075+0200 7fc771a10640 0 ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process ceph-osd, pid 1223 2024-10-16T13:48:05.076+0200 7fc771a10640 0 pidfile_write: ignore empty --pid-file 2024-10-16T13:48:05.086+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:05.086+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:05.087+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:05.088+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:05.088+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:05.088+0200 7fc771a10640 0 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:05.089+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:05.089+0200 7fc771a10640 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 1.8 TiB 2024-10-16T13:48:05.089+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:05.352+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:05.605+0200 7fc771a10640 0 starting osd.3 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2024-10-16T13:48:05.605+0200 7fc771a10640 -1 Falling back to public interface 2024-10-16T13:48:05.611+0200 7fc771a10640 0 load: jerasure load: lrc 2024-10-16T13:48:05.612+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:05.612+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:05.612+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:05.612+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:05.612+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:05.886+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:05.886+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:05.887+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:05.888+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:05.888+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:06.157+0200 7fc771a10640 0 osd.3:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:06.157+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:06.157+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:06.157+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:06.159+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:06.159+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:06.422+0200 7fc771a10640 0 osd.3:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:06.422+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:06.422+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:06.422+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:06.424+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:06.424+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:06.684+0200 7fc771a10640 0 osd.3:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:06.684+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:06.684+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:06.684+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:06.686+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:06.687+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:06.947+0200 7fc771a10640 0 osd.3:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:06.947+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:06.947+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:06.948+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:06.949+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:06.949+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) close 2024-10-16T13:48:07.214+0200 7fc771a10640 0 osd.3:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=wpq, cutoff=196) 2024-10-16T13:48:07.219+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:07.219+0200 7fc771a10640 0 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:07.219+0200 7fc771a10640 1 bdev(0x5630304bee00 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:07.219+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2024-10-16T13:48:07.219+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block 2024-10-16T13:48:07.219+0200 7fc771a10640 0 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument 2024-10-16T13:48:07.220+0200 7fc771a10640 1 bdev(0x5630304bf180 /var/lib/ceph/osd/ceph-3/block) open size 1999995142144 (0x1d1a9000000, 1.8 TiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-10-16T13:48:07.220+0200 7fc771a10640 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 1.8 TiB 2024-10-16T13:48:07.220+0200 7fc771a10640 1 bluefs mount 2024-10-16T13:48:07.221+0200 7fc771a10640 1 bluefs _init_alloc shared, id 1, capacity 0x1d1a9000000, block size 0x10000 2024-10-16T13:48:07.335+0200 7fc771a10640 1 bluefs mount shared_bdev_used = 0 2024-10-16T13:48:07.335+0200 7fc771a10640 1 bluestore(/var/lib/ceph/osd/ceph-3) _prepare_db_environment set db_paths to db,1899995385036 db.slow,1899995385036 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: RocksDB version: 7.9.2 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Git sha 0 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Compile date 2024-07-12 14:24:06 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: DB SUMMARY 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: DB Session ID: FNW6XLKJV5A5TZMHCHLU 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: CURRENT file: CURRENT 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: IDENTITY file: IDENTITY 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: MANIFEST file: MANIFEST-020722 size: 59265 Bytes 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: SST files in db dir, Total Num: 109, files: 021121.sst 021122.sst 021123.sst 021124.sst 021125.sst 021126.sst 021127.sst 021128.sst 021129.sst 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Write Ahead Log file in db.wal: 021323.log size: 15835454 ; 021325.log size: 12760055 ; 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Options.error_if_exists: 0 2024-10-16T13:48:07.345+0200 7fc771a10640 4 rocksdb: Options.create_if_missing: 0 ... 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.compaction_readahead_size: 2097152 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.max_background_flushes: -1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Compression algorithms supported: 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kZSTD supported: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kXpressCompression supported: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kBZip2Compression supported: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kLZ4Compression supported: 1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kZlibCompression supported: 1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kLZ4HCCompression supported: 1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: kSnappyCompression supported: 1 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: [db/db_impl/db_impl_readonly.cc:25] Opening the db in read only mode 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: db/MANIFEST-020722 2024-10-16T13:48:07.346+0200 7fc771a10640 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.compaction_filter: None 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.compaction_filter_factory: None 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.sst_partitioner_factory: None 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.table_factory: BlockBasedTable 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5630304b89a0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 checksum: 4 no_block_cache: 0 block_cache: 0x563030498dd0 block_cache_name: BinnedLRUCache block_cache_options: capacity : 483183820 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: bloomfilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 5 enable_index_compression: 1 block_align: 0 max_auto_readahead_size: 262144 prepopulate_block_cache: 0 initial_auto_readahead_size: 8192 num_file_reads_for_auto_readahead: 2 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.write_buffer_size: 16777216 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.max_write_buffer_number: 64 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.compression: LZ4 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.bottommost_compression: Disabled 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.prefix_extractor: nullptr ... 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.blob_file_starting_level: 0 2024-10-16T13:48:07.346+0200 7fc771a10640 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 Then is hangs here seemingly indefinitely. Hope someone can shed some light!

2 2

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

SLOW_OPS problems
by Tim Sauerbein 15 Oct '24

15 Oct '24

Dear list, I have a small cluster (Reef 18.2.4) with 7 hosts and 3-4 OSDs each (960GB/1.92TB mixed Intel D3-S4610, Samsung SM883, PM897 SSDs): cluster: id: ecff3ce8-539b-443e-a492-da428f4aa9e9 health: HEALTH_OK services: mon: 5 daemons, quorum titan,mangan,kalium,argon,chromium (age 2w) mgr: mangan(active, since 2w), standbys: titan, argon osd: 22 osds: 22 up (since 2w), 22 in (since 3M) data: pools: 2 pools, 513 pgs objects: 2.76M objects, 7.0 TiB usage: 16 TiB used, 15 TiB / 31 TiB avail pgs: 513 active+clean On that cluster RBD volumes for virtual machines are stored. For a couple of months now the cluster reports slow ops for some OSDs and some PGs as laggy. This happens once or twice a day, sometimes more and sometimes not at all for a few days, at completely random times, independent of when snapshots are deleted and trimmed and independent of the I/O load or load on the hosts. After about 30 seconds, during which the write speed goes to zero on the VMs, everything returns to normal. I cannot reproduce the slow ops manually by creating write load on the cluster. Even writing continuously with 300-400 MB/s full speed for 20 minutes does not create any problems. See attached log file for an example of a typical occurrence. I have also measured write load on the disks during the problems with iostat which just shows how writes stall, see also attached. The OSDs with slow ops are completely random, any of the disks would show up once in a while. Current config (I've tried optimising snaptrim and scrub which didn't help): # ceph config dump WHO MASK LEVEL OPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced bdev_async_discard true global advanced bdev_enable_discard true global advanced public_network 10.0.4.0/24 * mon advanced auth_allow_insecure_global_id_reclaim false mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/mode upmap mgr unknown mgr/pg_autoscaler/autoscale_profile scale-up * osd basic osd_memory_target 4294967296 osd advanced osd_pg_max_concurrent_snap_trims 1 osd advanced osd_scrub_begin_hour 23 osd advanced osd_scrub_end_hour 4 osd advanced osd_scrub_sleep 1.000000 osd advanced osd_snap_trim_priority 1 osd advanced osd_snap_trim_sleep 2.000000 osd.0 basic osd_mclock_max_capacity_iops_ssd 29199.674019 osd.1 basic osd_mclock_max_capacity_iops_ssd 31554.530141 osd.10 basic osd_mclock_max_capacity_iops_ssd 25949.821194 osd.11 basic osd_mclock_max_capacity_iops_ssd 26300.596265 osd.12 basic osd_mclock_max_capacity_iops_ssd 25167.331294 osd.13 basic osd_mclock_max_capacity_iops_ssd 21606.610828 osd.14 basic osd_mclock_max_capacity_iops_ssd 27894.095121 osd.15 basic osd_mclock_max_capacity_iops_ssd 25929.047047 osd.16 basic osd_mclock_max_capacity_iops_ssd 15423.600235 osd.17 basic osd_mclock_max_capacity_iops_ssd 25097.493934 osd.18 basic osd_mclock_max_capacity_iops_ssd 25966.188007 osd.19 basic osd_mclock_max_capacity_iops_ssd 23628.746459 osd.2 basic osd_mclock_max_capacity_iops_ssd 32157.280832 osd.20 basic osd_mclock_max_capacity_iops_ssd 22722.682745 osd.3 basic osd_mclock_max_capacity_iops_ssd 33951.086556 osd.4 basic osd_mclock_max_capacity_iops_ssd 22736.907664 osd.5 basic osd_mclock_max_capacity_iops_ssd 21916.777510 osd.6 basic osd_mclock_max_capacity_iops_ssd 29984.954749 osd.7 basic osd_mclock_max_capacity_iops_ssd 26757.965797 osd.8 basic osd_mclock_max_capacity_iops_ssd 22738.921429 osd.9 basic osd_mclock_max_capacity_iops_ssd 24635.156413 Any help would be much appreciated! Thanks, Tim

9 21

Announcing go-ceph v0.30.0
by John Mulligan 15 Oct '24

15 Oct '24

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.30.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, rgw, and other subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulliga(a)ibm.com / jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

1 0

Ceph User + Dev October Meetup Details
by Laura Flores 14 Oct '24

14 Oct '24

Hi all, Please see the following link for details about the User + Dev October Meetup that will be happening this week Wednesday! https://www.meetup.com/ceph-user-group/events/303960311/ Joseph Mundackal will be presenting a proposal for introducing a new S3-focused user group. Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Question about mounting cephFS on MacOS
by Baijia Ye 14 Oct '24

14 Oct '24

Dear ceph developers, I hope this email finds you well. I am an MSCS student at NYU and I am trying to use ceph as our university's research network storage system. We have built the server with ceph, may I ask how to mount cephFS on MacOS as a client? I hope this is the correct place to ask questions. If not, could you instruct me the write way? Thanks for your help. Best, Baijia Ye

1 0

Membership additions/removals from the Ceph Steering Committee
by Patrick Donnelly 14 Oct '24

14 Oct '24

Greetings, The Ceph Steering Committee [1] (CSC) -- formerly the "CLT" or Ceph Leadership Team -- was formed 3 years ago with the adoption of our new governance model [2]. The CSC is responsible [4] for electing the Ceph Executive Council [3] (CEC), amending the governance model, and deciding on tactical issues /strategic direction for the Ceph project. After voting for the inaugural members of the CEC (Dan van der Ster, Josh Durgin, and Neha Ohja), we have primarily focused on week-to-week tactical issues confronting the Ceph project during our weekly meetings. This was a natural continuation of our former "Ceph Leadership Team" meetings where we discussed any technical issues or planning for the project. We have not had another election for the Ceph Executive Council membership or any other governance amendment since but are overdue for one according to our governance charter. To begin such an election, it seemed prudent to first vote on any membership changes to the CSC due to natural attrition (e.g. folks have moved on to other projects) or expansion (e.g. new team leads, interested community members, etc.). The CSC weekly call has had several individuals present that are all but officially on the committee. This announcement is a call for anyone (developer, community member, etc.) interested in participating in the governance and strategic initiatives of the Ceph project to please put your or someone else's name forward. There is broad interest in the CSC to have representatives from newer components in Ceph present. If you are on the current list of members and would like to resign from the CSC, please let us know. Some members who have not been present in any recent calls have already been proposed to be removed. Any addition or removal to the CSC must be proposed as a motion with a second and then approved by a 2/3 supermajority vote. We hope to have this election for CSC membership in the coming weeks. It will be performed remotely and offline using the Helios election system (also a FOSS project!). It is deemed necessary to finalize this in advance of any other election topic, including the CEC membership. [1] https://docs.ceph.com/en/latest/governance/#ceph-steering-committee [2] https://docs.ceph.com/en/latest/governance/ [3] https://docs.ceph.com/en/latest/governance/#ceph-executive-council [4] https://docs.ceph.com/en/latest/governance/#id2 [5] https://vote.heliosvoting.org/ Cheers, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 2

What is the problem with many PGs per OSD
by Frank Schilder 14 Oct '24

14 Oct '24

Hi all, during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing. As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing. Yet, I'm holding back on an increase of PG count due to these rumors. My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450. I heard in related rumors that some users have 1000+ PGs per OSD without problems. I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do. I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object" or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors. An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500? Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

8 24

Reduced data availability: 3 pgs inactive, 3 pgs down
by Shain Miley 13 Oct '24

13 Oct '24

Hello, I am seeing the following information after reviewing ‘ceph health detail’: [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 3 pgs down pg 0.1a is down, acting [234,35] pg 0.20 is down, acting [226,267] pg 0.2f is down, acting [227,161] When I query each of those pgs I see the following message on each of them: "peering_blocked_by": [ { "osd": 233, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } Osd.233 crashed a while ago and when I try to start it the log shows some sort of issue with the filesystem: ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) 1: (()+0x12980) [0x7f2779617980] 2: (gsignal()+0xc7) [0x7f27782c9fb7] 3: (abort()+0x141) [0x7f27782cb921] 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x556ebe773ddf] 5: (FileStore::_do_transaction(ceph::os::Transaction&, unsigned long, int, ThreadPool::TPHandle*, char const*)+0x62b3) [0x556ebebe2753] 6: (FileStore::_do_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, unsigned long, ThreadPool::TPHandle*, char const*)+0x48) [0x556ebebe3f38] 7: (JournalingObjectStore::journal_replay(unsigned long)+0x105a) [0x556ebebfc56a] 8: (FileStore::mount()+0x438a) [0x556ebebda82a] 9: (OSD::init()+0x4d1) [0x556ebe80fdc1] 10: (main()+0x3f8c) [0x556ebe77ad2c] 11: (__libc_start_main()+0xe7) [0x7f27782acbf7] 12: (_start()+0x2a) [0x556ebe78fc4a] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. At this point I am thinking about either running an xfs repair on osd.233 and trying to see if I can get it back up (once the pgs are healthy again I would likey zap/readd or replace the drive). Another option it sounds like is to mark the osd as lost. I am just looking for advice on what exactly I should do next to try to minimize the chances of any data loss. Here is the query output for each of those pgs: https://pastebin.com/YbfnpZGC Thank you, Shain

2 4

GUI Block Images --> Restart mgr
by przemek.kuczynski＠gmail.com 12 Oct '24

12 Oct '24

When i want open block images in GUI mgr beeing restarted on 18.2.4 . After image conteiner is restarted i cannot access crash log Oct 11 13:46:38 hostname bash[27409]: debug -41> 2024-10-11T11:46:38.389+0000 7f4e2f823640 5 librbd::DiffIterate: fast diff enabled Oct 11 13:46:38 hostname bash[27409]: debug -40> 2024-10-11T11:46:38.389+0000 7f4e2f823640 5 librbd::DiffIterate: diff_iterate from 0 to 12 size from 0 to 2361393152 Oct 11 13:46:38 hostname bash[27409]: debug -39> 2024-10-11T11:46:38.389+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff121f3ef0 register_dispatch: dispatch_layer=3 Oct 11 13:46:38 hostname bash[27409]: debug -38> 2024-10-11T11:46:38.389+0000 7f518e504640 10 monclient: get_auth_request con 0x55ff103b6000 auth_method 0 Oct 11 13:46:38 hostname bash[27409]: debug -37> 2024-10-11T11:46:38.393+0000 7f4e2f823640 5 librbd::DiffIterate: fast diff enabled Oct 11 13:46:38 hostname bash[27409]: debug -36> 2024-10-11T11:46:38.393+0000 7f4e2f823640 5 librbd::DiffIterate: diff_iterate from 12 to 18446744073709551614 size from 2361393152 to 2361393152 Oct 11 13:46:38 hostname bash[27409]: debug -35> 2024-10-11T11:46:38.397+0000 7f4e2f823640 5 librbd::Watcher: 0x55ff132fac00 block_notifies: blocked_count=1 Oct 11 13:46:38 hostname bash[27409]: debug -34> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 librbd::io::Dispatcher: 0x55ff121f3ef0 shut_down_dispatch: dispatch_layer=3 Oct 11 13:46:38 hostname bash[27409]: debug -33> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 librbd::io::Dispatcher: 0x55ff121f3ef0 shut_down: Oct 11 13:46:38 hostname bash[27409]: debug -31> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 librbd::io::ObjectDispatch: 0x55ff107bde10 shut_down: Oct 11 13:46:38 hostname bash[27409]: debug -29> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 librbd::cache::WriteAroundObjectDispatch: 0x55ff1253a900 shut_down: Oct 11 13:46:38 hostname bash[27409]: debug -28> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 asok(0x55ff051a8000) unregister_commands rbd cache flush Images/0408a651-e894-4d01-9f49-65b2418d1505 Oct 11 13:46:38 hostname bash[27409]: debug -26> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=10 Oct 11 13:46:38 hostname bash[27409]: debug -25> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::QueueImageDispatch: 0x55ff1370c0e0 QueueImageDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -24> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=1 Oct 11 13:46:38 hostname bash[27409]: debug -23> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::QosImageDispatch: 0x55ff13050100 QosImageDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -22> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=2 Oct 11 13:46:38 hostname bash[27409]: debug -21> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::RefreshImageDispatch: 0x55ff107bdd60 RefreshImageDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -20> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=4 Oct 11 13:46:38 hostname bash[27409]: debug -19> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::WriteBlockImageDispatch: 0x55ff0e6540a0 WriteBlockImageDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -18> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=7 Oct 11 13:46:38 hostname bash[27409]: debug -17> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff0ea65000 register_dispatch: dispatch_layer=6 Oct 11 13:46:38 hostname bash[27409]: debug -16> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 asok(0x55ff051a8000) register_command rbd cache flush Images/10f3af9e-1766-4dbf-9cdb-416436027b23 hook 0x55ff13050f00 Oct 11 13:46:38 hostname bash[27409]: debug -15> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 asok(0x55ff051a8000) register_command rbd cache invalidate Images/10f3af9e-1766-4dbf-9cdb-416436027b23 hook 0x55ff13050f00 Oct 11 13:46:38 hostname bash[27409]: debug -14> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 librbd::ImageCtx: 0x55ff11696000: disabling zero-copy writes Oct 11 13:46:38 hostname bash[27409]: debug -12> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 librbd::cache::WriteAroundObjectDispatch: 0x55ff1253a900 init: Oct 11 13:46:38 hostname bash[27409]: debug -11> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 librbd::io::Dispatcher: 0x55ff0ea65000 register_dispatch: dispatch_layer=1 Oct 11 13:46:38 hostname bash[27409]: debug -10> 2024-10-11T11:46:38.405+0000 7f4e1ac7b640 5 librbd::io::SimpleSchedulerObjectDispatch: 0x55ff1304c6c0 SimpleSchedulerObjectDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -9> 2024-10-11T11:46:38.405+0000 7f4e1ac7b640 5 librbd::io::SimpleSchedulerObjectDispatch: 0x55ff1304c6c0 init: Oct 11 13:46:38 hostname bash[27409]: debug -8> 2024-10-11T11:46:38.405+0000 7f4e1ac7b640 5 librbd::io::Dispatcher: 0x55ff0ea65000 register_dispatch: dispatch_layer=5 Oct 11 13:46:38 hostname bash[27409]: debug -6> 2024-10-11T11:46:38.405+0000 7f4e1ac7b640 5 librbd::io::Dispatcher: 0x55ff13076090 shut_down_dispatch: dispatch_layer=3 Oct 11 13:46:38 hostname bash[27409]: debug -5> 2024-10-11T11:46:38.405+0000 7f4e1a47a640 5 librbd::io::WriteBlockImageDispatch: 0x55ff0e6540a0 unblock_writes: 0x55ff11696000, num=0 Oct 11 13:46:38 hostname bash[27409]: debug -3> 2024-10-11T11:46:38.409+0000 7f4e1a47a640 5 librbd::io::WriteBlockImageDispatch: 0x55ff0e6540a0 unblock_writes: 0x55ff11696000, num=0 Oct 11 13:46:38 hostname bash[27409]: debug -2> 2024-10-11T11:46:38.409+0000 7f4e2f823640 5 librbd::DiffIterate: fast diff enabled Oct 11 13:46:38 hostname bash[27409]: debug -1> 2024-10-11T11:46:38.409+0000 7f4e2f823640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/librbd/api/DiffIterate.cc: In function 'int librbd::api::DiffIterate<ImageCtxT>::execute() [with ImageCtxT = librbd::ImageCtx]' thread 7f4e2f823640 time 2024-10-11T11:46:38.414077+0000 Oct 11 13:46:38 hostname bash[27409]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/librbd/api/DiffIterate.cc: 341: FAILED ceph_assert(object_diff_state.size() == end_object_no - start_object_no) Oct 11 13:46:38 hostname bash[27409]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Oct 11 13:46:38 hostname bash[27409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f51910e504d] Oct 11 13:46:38 hostname bash[27409]: 4: /lib64/librbd.so.1(+0x51ada7) [0x7f5181bf1da7] Oct 11 13:46:38 hostname bash[27409]: 6: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x630bc) [0x7f5181e7c0bc] Oct 11 13:46:38 hostname bash[27409]: 8: PyVectorcall_Call() Oct 11 13:46:38 hostname bash[27409]: 9: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x44d50) [0x7f5181e5dd50] Oct 11 13:46:38 hostname bash[27409]: 10: _PyObject_MakeTpCall() Oct 11 13:46:38 hostname bash[27409]: 11: /lib64/libpython3.9.so.1.0(+0x125133) [0x7f5191c0a133] Oct 11 13:46:38 hostname bash[27409]: 12: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 14: _PyFunction_Vectorcall() Oct 11 13:46:38 hostname bash[27409]: 17: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7f5191c01b73] Oct 11 13:46:38 hostname bash[27409]: 18: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 19: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 20: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7f5191c01b73] Oct 11 13:46:38 hostname bash[27409]: 21: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 22: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 23: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: 25: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 26: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 27: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: 29: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 30: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 31: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: debug 0> 2024-10-11T11:46:38.413+0000 7f4e2f823640 -1 *** Caught signal (Aborted) ** Oct 11 13:46:38 hostname bash[27409]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5190a8e6f0] Oct 11 13:46:38 hostname bash[27409]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5190adb94c] Oct 11 13:46:38 hostname bash[27409]: 3: raise() Oct 11 13:46:38 hostname bash[27409]: 4: abort() Oct 11 13:46:38 hostname bash[27409]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f51910e50a7] Oct 11 13:46:38 hostname bash[27409]: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7f51910e520b] Oct 11 13:46:38 hostname bash[27409]: 7: /lib64/librbd.so.1(+0x193403) [0x7f518186a403] Oct 11 13:46:38 hostname bash[27409]: 9: rbd_diff_iterate2() Oct 11 13:46:38 hostname bash[27409]: 11: /lib64/libpython3.9.so.1.0(+0x11d7a1) [0x7f5191c027a1] Oct 11 13:46:38 hostname bash[27409]: 13: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x44d50) [0x7f5181e5dd50] Oct 11 13:46:38 hostname bash[27409]: 14: _PyObject_MakeTpCall() Oct 11 13:46:38 hostname bash[27409]: 16: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 17: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: 18: _PyFunction_Vectorcall() Oct 11 13:46:38 hostname bash[27409]: 19: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 20: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 21: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7f5191c01b73] Oct 11 13:46:38 hostname bash[27409]: 22: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 24: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7f5191c01b73] Oct 11 13:46:38 hostname bash[27409]: 26: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 27: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: 29: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 30: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 0/ 5 none Oct 11 13:46:38 hostname bash[27409]: 0/ 1 context Oct 11 13:46:38 hostname bash[27409]: 1/ 5 mds_balancer Oct 11 13:46:38 hostname bash[27409]: 1/ 5 mds_log Oct 11 13:46:38 hostname bash[27409]: 1/ 5 mds_log_expire Oct 11 13:46:38 hostname bash[27409]: 1/ 5 mds_migrator Oct 11 13:46:38 hostname bash[27409]: 0/ 1 buffer Oct 11 13:46:38 hostname bash[27409]: 0/ 1 timer Oct 11 13:46:38 hostname bash[27409]: 0/ 1 objecter Oct 11 13:46:38 hostname bash[27409]: 0/ 5 rados Oct 11 13:46:38 hostname bash[27409]: 0/ 5 rbd_mirror Oct 11 13:46:38 hostname bash[27409]: 0/ 5 rbd_replay Oct 11 13:46:38 hostname bash[27409]: 0/ 5 rbd_pwl Oct 11 13:46:38 hostname bash[27409]: 0/ 5 journaler Oct 11 13:46:38 hostname bash[27409]: 0/ 5 immutable_obj_cache Oct 11 13:46:38 hostname bash[27409]: 1/ 5 osd Oct 11 13:46:38 hostname bash[27409]: 0/ 5 objclass Oct 11 13:46:38 hostname bash[27409]: 0/ 0 ms Oct 11 13:46:38 hostname bash[27409]: 0/10 monc Oct 11 13:46:38 hostname bash[27409]: 1/ 5 paxos Oct 11 13:46:38 hostname bash[27409]: 0/ 5 tp Oct 11 13:46:38 hostname bash[27409]: 1/ 5 crypto Oct 11 13:46:38 hostname bash[27409]: 1/ 5 heartbeatmap Oct 11 13:46:38 hostname bash[27409]: 1/ 5 rgw_sync Oct 11 13:46:38 hostname bash[27409]: 1/ 5 rgw_datacache Oct 11 13:46:38 hostname bash[27409]: 1/ 5 rgw_flight Oct 11 13:46:38 hostname bash[27409]: 1/ 5 asok Oct 11 13:46:38 hostname bash[27409]: 1/ 1 throttle Oct 11 13:46:38 hostname bash[27409]: 0/ 0 refs Oct 11 13:46:38 hostname bash[27409]: 1/ 5 compressor Oct 11 13:46:38 hostname bash[27409]: 1/ 5 bluestore Oct 11 13:46:38 hostname bash[27409]: 1/ 5 kstore Oct 11 13:46:38 hostname bash[27409]: 4/ 5 rocksdb Oct 11 13:46:38 hostname bash[27409]: 1/ 5 fuse Oct 11 13:46:38 hostname bash[27409]: 2/ 5 mgr Oct 11 13:46:38 hostname bash[27409]: 0/ 5 test Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_onode Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_odata Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_t Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_cleaner Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_epm Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_lba Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_fixedkv_tree Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_cache Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_device Oct 11 13:46:38 hostname bash[27409]: 0/ 5 cyanstore Oct 11 13:46:38 hostname bash[27409]: 1/ 5 ceph_exporter Oct 11 13:46:38 hostname bash[27409]: -2/-2 (syslog threshold) Oct 11 13:46:38 hostname bash[27409]: 99/99 (stderr threshold) Oct 11 13:46:38 hostname bash[27409]: 7f4e1a47a640 / io_context_pool Oct 11 13:46:38 hostname bash[27409]: 7f4e1b47c640 / safe_timer Oct 11 13:46:38 hostname bash[27409]: 7f4e1e000640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f4e1f803640 / io_context_pool Oct 11 13:46:38 hostname bash[27409]: 7f4e2b01a640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e2c01c640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2c81d640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2e020640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2e821640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2f022640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2f823640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e31827640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e32829640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e3402c640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e36030640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e36831640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e38034640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e3a9b8640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e3e1bf640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e42b08640 / safe_timer Oct 11 13:46:38 hostname bash[27409]: 7f4e43b0a640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f4e453cd640 / io_context_pool Oct 11 13:46:38 hostname bash[27409]: 7f4e45c0e640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e47c12640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e4a417640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e4ac18640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e4c41b640 / safe_timer Oct 11 13:46:38 hostname bash[27409]: 7f4e4d41d640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f4e4ec20640 / io_context_pool Oct 11 13:46:38 hostname bash[27409]: 7f4e51465640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e5552d640 / pg_autoscaler Oct 11 13:46:38 hostname bash[27409]: 7f4e5652f640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e58d34640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e5a537640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e5bd7a640 / devicehealth Oct 11 13:46:38 hostname bash[27409]: 7f4e5fd82640 / crash Oct 11 13:46:38 hostname bash[27409]: 7f4e60d84640 / cephadm Oct 11 13:46:38 hostname bash[27409]: 7f4e62587640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e64e0c640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e65e0e640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e66e10640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e67611640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e68613640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e68e14640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e69e96640 / balancer Oct 11 13:46:38 hostname bash[27409]: 7f4e6dede640 / cmdfin Oct 11 13:46:38 hostname bash[27409]: 7f4e6f6e1640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f51864f4640 / safe_timer Oct 11 13:46:38 hostname bash[27409]: 7f518acfd640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f518e504640 / msgr-worker-1 Oct 11 13:46:38 hostname bash[27409]: 7f518ed05640 / msgr-worker-0 Oct 11 13:46:38 hostname bash[27409]: max_recent 10000 Oct 11 13:46:38 hostname bash[27409]: max_new 1000 Oct 11 13:46:38 hostname bash[27409]: log_file /var/lib/ceph/crash/2024-10-11T11:46:38.415833Z_b3978f24-6697-44f5-80dc-4915b5ec144d/log Oct 11 13:46:38 hostname bash[27409]: --- end dump of recent events --- ^C

2 1

Questions about inactive pgs. thank you
by 苏察哈尔灿 12 Oct '24

12 Oct '24

I am deleting two OSD nodes on a node, and ceph health detail indicates that there are 15 pgs inactive, as follows: [WRN] PG_AVAILABILITY: Reduced data availability: 15 pgs inactive     pg 37.4 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [NONE,40,38,88,0,22,62,32,NONE,21]     pg 37.a is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [38,103,NONE,57,20,NONE,63,51,8,18]     pg 37.c is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [58,7,18,38,52,101,80,NONE,62,NONE]     pg 37.62 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [67,101,NONE,57,62,NONE,7,26,14,20]     pg 37.97 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [35,19,NONE,13,58,17,20,6,25,NONE]     pg 37.9d is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [103,22,NONE,63,NONE,34,45,56,90,14]     pg 37.b4 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [61,NONE,NONE,46,86,60,9,66,21,35]     pg 37.136 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [10,NONE,24,78,48,89,NONE,56,8,51]     pg 37.13a is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [71,33,NONE,14,32,27,NONE,5,25,57]     pg 37.149 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [6,69,65,84,27,25,NONE,0,14,NONE]     pg 37.15c is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [103,66,6,23,NONE,NONE,61,45,8,42]     pg 37.16d is stuck inactive for 113m, current state recovering+undersized+remapped+peered, last acting [79,68,26,66,58,NONE,52,8,NONE,60]     pg 37.17b is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [80,47,27,14,24,NONE,67,35,NONE,5]     pg 37.1d0 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [0,76,68,57,NONE,10,48,85,101,NONE]     pg 37.1ea is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [28,24,52,20,10,43,56,NONE,NONE,80] My pool is used using 8+2 erasure-code configuration: ceph osd erasure-code-profile get my-ec-profile02 crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 In theory, it should be possible to access the corresponding pool, but in fact, the client cannot access normally, or it will be stuck in the middle of the access. Why? Then, when using the command "rados -p cephfs-backup.data-ec ls" to display the pool object, it also stuck 10000000e74.00000cd8 1000000080f.00004436 10000000e3d.000012e3 10000000e50.000018a2 1000000081b.00005b22 100000003d3.0000777a 100000003d7.00003c64 Cursor motionless thank you!

2 1

The ceph monitor crashes every few days
by 李明 10 Oct '24

10 Oct '24

Hello, ceph version is 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) and the rbd info command is also slow, some times it needs 6 seconds. rbd snap create command takes 17 seconds. There is another cluster with the same configuration it only takes less than 1 second. crash log -2> 2024-10-09 09:52:50.782 7f30ddcb4700 5 prioritycache tune_memory target: 2147483648 mapped: 1286012928 unmapped: 1695842304 heap: 2981855232 old mem: 1020054731 new mem: 1020054731 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/mon/OSDMonitor.cc: 4112: ceph_abort_msg("abort() called") ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f30ea6c2a66] 2: (OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned long)+0x89f) [0x563b0175382f] 3: (OSDMonitor::send_incremental(unsigned int, MonSession*, bool, boost::intrusive_ptr<MonOpRequest>)+0x11d) [0x563b01753e6d] 4: (OSDMonitor::check_osdmap_sub(Subscription*)+0xca) [0x563b0175a53a] 5: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xbbe) [0x563b0162c9ee] 6: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x525) [0x563b0164b0a5] 7: (Monitor::_ms_dispatch(Message*)+0xcdb) [0x563b0164cd9b] 8: (Monitor::ms_dispatch(Message*)+0x26) [0x563b0167ad96] 9: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x563b01677776] 10: (DispatchQueue::entry()+0x1699) [0x7f30ea8e7699] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f30ea99564d] 12: (()+0x7ea5) [0x7f30e74acea5] 13: (clone()+0x6d) [0x7f30e636fb0d] 0> 2024-10-09 09:52:55.260 7f30db4af700 -1 *** Caught signal (Aborted) ** in thread 7f30db4af700 thread_name:ms_dispatch ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (()+0xf630) [0x7f30e74b4630] 2: (gsignal()+0x37) [0x7f30e62a7387] 3: (abort()+0x148) [0x7f30e62a8a78] 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f30ea6c2b2e] 5: (OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned long)+0x89f) [0x563b0175382f] 6: (OSDMonitor::send_incremental(unsigned int, MonSession*, bool, boost::intrusive_ptr<MonOpRequest>)+0x11d) [0x563b01753e6d] 7: (OSDMonitor::check_osdmap_sub(Subscription*)+0xca) [0x563b0175a53a] 8: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xbbe) [0x563b0162c9ee] 9: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x525) [0x563b0164b0a5] 10: (Monitor::_ms_dispatch(Message*)+0xcdb) [0x563b0164cd9b] 11: (Monitor::ms_dispatch(Message*)+0x26) [0x563b0167ad96] 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x563b01677776] 13: (DispatchQueue::entry()+0x1699) [0x7f30ea8e7699] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f30ea99564d] 15: (()+0x7ea5) [0x7f30e74acea5] 16: (clone()+0x6d) [0x7f30e636fb0d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this ceph -s cluster: id: 6e36443f-326b-4344-b013-c03a4b7bb8d7 health: HEALTH_WARN noout flag(s) set 314 pgs not deep-scrubbed in time 660 pgs not scrubbed in time 23 daemons have recently crashed services: mon: 3 daemons, quorum rbd02,rbd01,rbd03 (age 81m) mgr: rbd03(active, since 9d), standbys: rbd01, rbd02 osd: 360 osds: 360 up (since 9d), 360 in (since 12M) flags noout data: pools: 1 pools, 8192 pgs objects: 109.45M objects, 274 TiB usage: 488 TiB used, 3.4 PiB / 3.8 PiB avail pgs: 8116 active+clean 76 active+clean+scrubbing+deep io: client: 61 MiB/s rd, 34 MiB/s wr, 74.51k op/s rd, 2.06k op/s wr

3 2

Erasure coding scheme 2+4 = good idea?
by Andre Tann 10 Oct '24

10 Oct '24

Hi all, I'd like your thoughts and comments on this idea: Setup: - two fault domains = DCs - connected with 100 GBit, < 1 ms - 80 NVME SSDs on each side Goal: One fault domain can be lost, and then there's still have some redundancy. Option 1: Replicated pool with size = 4. This gives me two copies on each side, thus meets the goal. But the efficiency is only 25%. Option 2: Erasure coded pool with 2 + 4 scheme. This gives me 3 chunks on each side. If I lose one side, I still have 3 chunks left, where I only need 2. Thus the goal is also met. Efficiency is 33%. Even though I did a lot of googling, I couldn't find anything about a similar setup. In all profiles, there is k <= m. What do you think about 2+4, is it a good idea or a bad one, or do I miss something and it doesn't work at all? In particular: is it possible to recover two data chunks out of 2 coding chunks? As I read the documentation, this should be no problem, just want to confirm. -- Andre Tann

4 5

About 100g network card for ceph
by Phong Tran Thanh 10 Oct '24

10 Oct '24

Hi ceph users I have a 100G network card with dual ports for a Ceph node with NVMe disks. Should I bond them or not? Should I bond 200G for both the public and cluster networks, or separate it: one port for the public network and one for the cluster? Thank ceph users -- Email: tranphong079(a)gmail.com Skype: tranphong079

3 2

Packets Drops in bond interface
by Alfredo Rezinovsky 10 Oct '24

10 Oct '24

I'm having lots of those warning in all nodes: CephNodeNetworkPacketDrops Node ceph-06 experiences packet drop > 0.5% or > 10 packets/s on interface bond0. bonds are 802.3ad with Mikrotik switches. Physical interfaces are OK, without drops. They are Intel e1000 dual PCIe NICs Running with kernel 6.8.0-45-generic ubuntu 22.04 -- Alfrenovsky

1 0

Forced upgrade OSD from Luminous to Pacific
by Alex Rydzewski 10 Oct '24

10 Oct '24

Hello, dear community! I kindly ask for your help in resolving my issue. I have a server with a single-node CEPH setup with 5 OSDs. This server has been powered off for about two years, and when I needed the data from it, I found that the SSD where the system was installed had died. I tried to recover the cluster. First, assuming the old CEPH is there I installed Debian 10 with CEPH 12.2.11, mounted the OSDs to /var/lib/ceph/osd/ceph-xx and assembled the monitor, as described here https://forum.proxmox.com/threads/recover-ceph-from-osds-only.113699/. However, the monitor wouldn't start, giving an error I don't remember. Then I made a series of mistakes, upgrading the system and CEPH first to Nautilus and then to Pacific. Eventually, I managed to start the monitor, but a compatibility issue with the OSDs remains. When the OSDs start, I see the message: /check_osdmap_features require_osd_release unknown -> luminous /At the same time, the monitor log shows: /disallowing boot of octopus+ OSD osd.xx. /After starting, the OSD remains in the state: /tick checking mon for new map/ Then I enabledmsgrv2 protocol and tried enabling RocksDB sharding for the OSD, as described here https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#b…, but it didn’t help. Attempts to start the OSD with lower versions of CEPH event with Octopus end with the error: /2024-10-08 10:45:38.402975 7fba61b34ec0 -1 bluefs _replay 0x0: stop: unrecognized op 12 2024-10-08 10:45:38.402992 7fba61b34ec0 -1 bluefs mount failed to replay log: (5) Input/output error/ So, currently, I have CEPH 16.2.15, and the OSD is in the following state: /"/var/lib/ceph/osd/ceph-1/block": { "osd_uuid": "2bb56721-28c7-45cc-9344-6cc5c699a642", "size": 4000681103360, "btime": "2018-06-02 13:16:57.042205", "description": "main", "bfm_blocks": "61045632", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "65536", "bfm_size": "4000681099264", "bluefs": "1", "ceph_fsid": "96b6ff1d-25bf-403f-be3d-78c2fb0ff747", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "ready": "ready", "require_osd_release": "12", "whoami": "1" }/ with modified RocksDB to enable sharding. Suggest me, please, Is there a way to upgrade such OSDs so they can run with this version of Ceph? If you need more information here, let me know and I will provide whatever is needed. -- Alexander Rydzewski

3 10

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Can't pull container images from quay-quay-quay.apps.os.sepia.ceph.com
by John Robert Mendoza 09 Oct '24

09 Oct '24

Hello, I am trying to set up a 3-node Ceph cluster using cephadm and bootstrapping fails because podman can't pull ceph container images from quay-quay-quay.apps.os.sepia.ceph.com. Excerpt of the error message I encountered: ... /usr/bin/podman: stderr time="2024-10-09T21:57:22+08:00" level=warning msg="Failed, retrying in 1s ... (3/3). Error: parsing image configuration: Get \"https://quay-quay-quay.apps.os.sepia.ceph.com/_storage_proxy/ZXlKaGJHY2lPaU…": dial tcp 172.21.0.91:443: i/o timeout" ... There must be a mistake with quay-quay-quay.apps.os.sepia.ceph.com DNS record because it's resolving to a RFC1918 address (172.16/12 prefix). I tried various online DNS tools (including Google's Admin Toolbox) to help isolate the issue and they all return the same IP address (172.21.0.91). Anyone else having this problem?

2 1

About scrub and deep-scrub
by Phong Tran Thanh 09 Oct '24

09 Oct '24

Hi ceph users! How about the disable scrub and deep-scrub, i want to disable it because of its effect on many I/O of my cluster. If I disable scrub how will it affect my cluster? If enabled, scrubbing is not complete and takes a long time. Thank Skype: tranphong079

4 9

The ceph monitor crashes every few days
by 李明 09 Oct '24

09 Oct '24

Hello, ceph version is 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) and the rbd info command is also slow, some times it needs 6 seconds. rbd snap create command takes 17 seconds. There is another cluster with the same configuration it only takes less than 1 second. crash log -2> 2024-10-09 09:52:50.782 7f30ddcb4700 5 prioritycache tune_memory target: 2147483648 mapped: 1286012928 unmapped: 1695842304 heap: 2981855232 old mem: 1020054731 new mem: 1020054731 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/mon/OSDMonitor.cc: 4112: ceph_abort_msg("abort() called") ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f30ea6c2a66] 2: (OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned long)+0x89f) [0x563b0175382f] 3: (OSDMonitor::send_incremental(unsigned int, MonSession*, bool, boost::intrusive_ptr<MonOpRequest>)+0x11d) [0x563b01753e6d] 4: (OSDMonitor::check_osdmap_sub(Subscription*)+0xca) [0x563b0175a53a] 5: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xbbe) [0x563b0162c9ee] 6: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x525) [0x563b0164b0a5] 7: (Monitor::_ms_dispatch(Message*)+0xcdb) [0x563b0164cd9b] 8: (Monitor::ms_dispatch(Message*)+0x26) [0x563b0167ad96] 9: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x563b01677776] 10: (DispatchQueue::entry()+0x1699) [0x7f30ea8e7699] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f30ea99564d] 12: (()+0x7ea5) [0x7f30e74acea5] 13: (clone()+0x6d) [0x7f30e636fb0d] 0> 2024-10-09 09:52:55.260 7f30db4af700 -1 *** Caught signal (Aborted) ** in thread 7f30db4af700 thread_name:ms_dispatch ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (()+0xf630) [0x7f30e74b4630] 2: (gsignal()+0x37) [0x7f30e62a7387] 3: (abort()+0x148) [0x7f30e62a8a78] 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f30ea6c2b2e] 5: (OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned long)+0x89f) [0x563b0175382f] 6: (OSDMonitor::send_incremental(unsigned int, MonSession*, bool, boost::intrusive_ptr<MonOpRequest>)+0x11d) [0x563b01753e6d] 7: (OSDMonitor::check_osdmap_sub(Subscription*)+0xca) [0x563b0175a53a] 8: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xbbe) [0x563b0162c9ee] 9: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x525) [0x563b0164b0a5] 10: (Monitor::_ms_dispatch(Message*)+0xcdb) [0x563b0164cd9b] 11: (Monitor::ms_dispatch(Message*)+0x26) [0x563b0167ad96] 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x563b01677776] 13: (DispatchQueue::entry()+0x1699) [0x7f30ea8e7699] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f30ea99564d] 15: (()+0x7ea5) [0x7f30e74acea5] 16: (clone()+0x6d) [0x7f30e636fb0d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this ceph -s cluster: id: 6e36443f-326b-4344-b013-c03a4b7bb8d7 health: HEALTH_WARN noout flag(s) set 314 pgs not deep-scrubbed in time 660 pgs not scrubbed in time 23 daemons have recently crashed services: mon: 3 daemons, quorum rbd02,rbd01,rbd03 (age 81m) mgr: rbd03(active, since 9d), standbys: rbd01, rbd02 osd: 360 osds: 360 up (since 9d), 360 in (since 12M) flags noout data: pools: 1 pools, 8192 pgs objects: 109.45M objects, 274 TiB usage: 488 TiB used, 3.4 PiB / 3.8 PiB avail pgs: 8116 active+clean 76 active+clean+scrubbing+deep io: client: 61 MiB/s rd, 34 MiB/s wr, 74.51k op/s rd, 2.06k op/s wr

1 0

Administrative test, please ignore
by Anthony D'Atri 08 Oct '24

08 Oct '24

Testing

1 0

Ceph.io down?!
by ceph＠elchaka.de 08 Oct '24

08 Oct '24

Hi, Not sure of this is the right path.. but the site Ceph.io Is since at least 1,5h down for me... Intendet? Mehmet

1 0

Question about bucket / object policy
by Andrea Martra 08 Oct '24

08 Oct '24

Hello ceph-users, I need to prevent the use of public objects in some Ceph S3 clusters where clients access directly via haproxy as balancer and backend on radosgw. I would like to know if anyone has found a solution to prevent entire Buckets or even single objects from being made accessible without authentication and that they can be exposed with a specific URL. I tried to set the policies in 'deny' for "s3:PutBucketPolicy", "s3:PutObjectAcl" and the objects uploaded to the bucket after the policy is applied cannot be made public. However, I would need to have a way to set the Deny on all buckets when they are created by clients because otherwise I have to iterate on all existing buckets and apply the policy, but in this way if the client creates a bucket, uploads some objects and makes them public before the Deny is applied I no longer have control. I should then also execute a policy on all objects but it becomes a very long operation. If anyone has any ideas on this it would be greatly appreciated. Thank you very much. Andrea

1 0

Ceph Steering Committee Team Meeting 2024-10-07
by Ernesto Puerta 07 Oct '24

07 Oct '24

Hi Cephers, The big topic today has been on the *Ceph project Governance*, especifically, *Ceph Steering Committee (CSC) changes *in the context of the ongoing* Ceph Executive Council (CEC) renewal*. Please check this email thread <https://www.spinics.net/lists/ceph-users/msg84293.html> and this etherpad <https://pad.ceph.com/p/css-vote-2024q4> for more details. Additionally: - We reviewed the *new candidates* proposed to the CSC (removals have been deferred), - Discussed about the *CSC vs. CLT naming* (decision to continue with CSC), - Proposed an *alternating CSC/CLT meeting* (+12h) to facilitate everyone around the Globe to join (e.g: APAC). - Proposed to *limit the max number of CSC members* (deferred). The first ballot <https://vote.heliosvoting.org/helios/elections/f276a15a-84c5-11ef-a0e4-b69e…> is now open and will run until next Monday. --- Also: @Radoslaw Zarzynski <rzarzyns(a)redhat.com> brought up the following topics: - *Facilities for testing intended FTBFSes*: driving factor: https://github.com/ceph/ceph/pull/60159/files - *Namespaces in ceph-dencoder*: let's discuss next week, CC @Ilya Dryomov <idryomov(a)redhat.com> @Nizamudeen A <nia(a)redhat.com> communicated *he's stepping down from the Ceph Dashboard team* leadership, and that @Afreen Misbah <afrahman(a)redhat.com>* will take over*. Welcome and good luck, Afreen! @John Zachary Dover <zac.dover(a)gmail.com> announced that *Ceph Quarterly <https://ceph.io/en/community/cq/> #6 will be published on the 15th of October*, and asked for Squid highlights. @Casey Bodley <cbodley(a)redhat.com> asked about the *status of the latest Quincy* release, and @Yuri Weinstein <yweinste(a)redhat.com> confirmed that it is still *in progress*, with ~8 PRs pending. --- We wanted to remind you that the CSC/CLT weekly meeting (Mondays 11am ET) is open to anyone actively involved in the project's both short and long-term challenges (feel free to review and suggest topics in https://pad.ceph.com/p/clt-weekly-minutes) Kind Regards, Ernesto

1 0

Ceph Meetup in Berlin, Germany and Frankfurt/Main, Germany - interested people welcome !
by Matthias Muench 07 Oct '24

07 Oct '24

Hi people in Germany, we run two meetup groups in Berlin and Frankfurt with meetings nearly quarterly. For each location, we provide also a quick introduction to Ceph before the main event. Currently, we have a new set of appointments. You can join us by RSVP to the events: - Ceph Frankfurt: 09.10.2024 in Frankfurt/Main, Germany Introduction: https://www.meetup.com/de-DE/ceph-frankfurt/events/303545788/ Main meetup: https://www.meetup.com/de-DE/ceph-frankfurt/events/303545742/ - Ceph Berlin: 14.10.2024 in Berlin, Germany Introduction: https://www.meetup.com/de-DE/ceph-berlin/events/303546142/ Main meetup: https://www.meetup.com/de-DE/ceph-berlin/events/303546178/ To get the invites also for any future meetings, one could also join the meetup groups: - Ceph Frankfurt: https://www.meetup.com/de-DE/ceph-frankfurt/ - Ceph Berlin: https://www.meetup.com/de-DE/ceph-berlin/ Happy to see you soon, -matt -- —————————————————— Matthias Muench Principal Specialist Solution Architect, OpenShift EMEA Storage Specialist matthias.muench(a)redhat.com Phone: +49-160-92654111 Red Hat GmbH Technopark II Werner-von-Siemens-Ring 12 85630 Grasbrunn Germany _______________________________________________________________________ Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

1 0

cephadm crush_device_class not applied
by Eugen Block 04 Oct '24

04 Oct '24

Hi, I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec: cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore I see that cephadm has stored it correctly: ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}} And it has the OSDSPEC_AFFINITY set: cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd'] But the OSDs still are created with hdd device class: ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000 I have tried it with two different indentations: spec: crush_device_class: hdd-ec and as seen above: crush_device_class: hdd-ec placement: label: osd spec: According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation: 2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2. Trying to create it manually also doesn't work as expected: soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph' soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000 This is in the mgr debug output from the manual creation: 2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')) So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here? Thanks! Eugen [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-…

4 8

Compatibility check before updating from Centros stream 8 ceph 16.2.15 to 17.2.7
by xadhoom76＠gmail.com 04 Oct '24

04 Oct '24

Anyone knowns if there is any issue to upgrade from 16.2.16 to 17.2.7 (containerized ceph) on centos stream 8 ? Best regards

1 0

v19 & IPv6: unable to convert chosen address to string
by Sascha Frey 04 Oct '24

04 Oct '24

Hi, after upgrading our Ceph cluster from 18.2.2 to 19.2.0, I get the following error messages: 'ceph status' shows HEALTH_ERR 504 osds(s) are not reachable but, luckily, everything works fine. I see these related messages in the monitor logs (these messages show up for each OSD): 2024-10-01T16:26:36.185+0200 7ffbd5600640 -1 log_channel(cluster) log [ERR] : osd.101's public address is not in '2001:638:504:2011::/64' subnet ... 2024-10-01T16:34:01.011+0200 7ffbdba00640 -1 unable to convert chosen address to string: 2001:638:504:2011:9:4:1:1 ... IPv6 is enabled in ceph.conf: ms bind ipv4 = false ms bind ipv6 = true public network = 2001:638:504:2011::/64 Is this a bug? Thanks, Sascha

3 3

Monitors for two different cluster
by Michel Niyoyita 03 Oct '24

03 Oct '24

Hello Team, I have a running cluster deployed using ceph-ansible pacific version and ubuntu OS , with three mons and 3 osds servers , the cluster is well running , now I want to make another cluster wich will consist of 3 osds servers , can the new cluster be deployed using cephadm and using the existing Mons for the first cluster? Best regards Michel

3 3

MDS stuck in replay and continually crashing during replay
by Ivan Clayson 03 Oct '24

03 Oct '24

Hello everyone, We're having an issue with our backup filesystem running Reef (18.2.4) on an Alma9.4 (version 5.14 kernel) cluster where are our filesystem "ceph_spare" is in a constant degraded state: root@pebbles-n4 11:49 [~]: ceph fs status ceph_spare - 360 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay pebbles-s2 14.1M 6858k 240k 0 POOL TYPE USED AVAIL mds_spare_fs metadata 1451G 3216G ec82_spare_primary_fs_data data 0 3216G ec82pool_spare data 8820T 4626T STANDBY MDS pebbles-s1 pebbles-s2 MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) root@pebbles-n4 11:49 [~]: ceph health detail HEALTH_WARN 1 filesystem is degraded; 1 large omap objects; 1 MDSs report oversized cache; 1 MDSs behind on trimming [WRN] FS_DEGRADED: 1 filesystem is degraded fs ceph_spare is degraded [WRN] LARGE_OMAP_OBJECTS: 1 large omap objects 1 large objects found in pool 'mds_spare_fs' Search the cluster log for 'Large omap object found' for more details. [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.pebbles-s2(mds.0): MDS cache is too large (24GB/4GB); 0 inodes in use by clients, 0 stray files [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.pebbles-s2(mds.0): Behind on trimming (69293/128) max_segments: 128, num_segments: 69293 Our MDS is stuck in replay constantly and crashing before finishing despite a restart, reboot, and wiping of the client sessions ("ceph config set mds mds_wipe_sessions true"). When following the replay, we've noticed that the MDS always crashes just before the "journal_read_pos" reaches the "journal_write_pos" where the MDS logs with a 10/20 debug report the following: ... -14> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay dir 0x1002a7c9f3a.010001* -13> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x1002a7c9f3a.010001* /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_Sma llHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/ [2,head] auth v=283167 cv=0/0 state=1073741824 f(v0 m2024-09-30T02:20:11.052706+0100 4262=4262+0) n(v37 rc2024-09-30T02:20:11.052706+0100 b4168080204 4262=4262+0) hs=877+853,ss=3+0 dirty=1733 | child=1 0x5642e8d5d680] -12> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.cache.dir(0x1002a7c9f3a.010001*) lookup_exact_snap (head, 'FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks _plot_0012.ef980630.partial') -11> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.cache.dir(0x1002a7c9f3a.010001*) lookup_exact_snap (head, 'FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks _plot_0012.ef980630.partial') -10> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 12 mds.0.cache.dir(0x1002a7c9f3a.010001*) add_null_dentry [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallH ole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lo ck) pv=0 v=283167 ino=(nil) state=1073741824 0x564f219fca00] -9> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_P u_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lock) v=28316 6 ino=(nil) state=1610612736 | dirty=1 0x564f219fca00] -8> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 12 mds.0.cache.dir(0x1002a7c9f3a.010001*) link_primary_inode [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_Sma llHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_46164 89_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lock) v=283166 ino=(nil) state=1610612736 | dirty=1 0x564f219fca00] [inode 0x1002c5f1d10 [126,head] #1002c5f1d10 auth v283166 s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} 0x564f21a00000] -7> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay added [inode 0x1002c5f1d10 [126,head] /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial auth v283166 s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} 0x564f21a00000] -6> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.cache.ino(0x1002c5f1d10) mark_dirty_parent -5> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay noting opened inode [inode 0x1002c5f1d10 [126,head] /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial auth v283166 DIRTYPARENT s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} | dirtyparent=1 dirty=1 0x564f21a00000] -4> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay inotable tablev 1481899 <= table 1481899 -3> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay sessionmap v 746010300, table 746010299 prealloc [] used 0x1002c5f1d10 -2> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.journal (session prealloc [0x1002bdf9f74~0x22,0x1002bdfb51d~0x7c,0x1002c59def8~0xfb,0x1002c5eec0c~0x86,0x1002c5eee87~0x1f5]) -1> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7fa8b6d95640 time 2024-10-02T15:20:52.495403+0100 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa8c416b04d] 2: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fa8c416b20b] 3: /usr/bin/ceph-mds(+0x1ef0fe) [0x5642daf490fe] 4: /usr/bin/ceph-mds(+0x1ef145) [0x5642daf49145] 5: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bad) [0x5642db17b67d] 6: (EUpdate::replay(MDSRank*)+0x5d) [0x5642db18477d] 7: (MDLog::_replay_thread()+0x75e) [0x5642db12d00e] 8: /usr/bin/ceph-mds(+0x13c561) [0x5642dae96561] 9: /lib64/libc.so.6(+0x89c02) [0x7fa8c3889c02] 10: /lib64/libc.so.6(+0x10ec40) [0x7fa8c390ec40] 0> 2024-10-02T15:20:52.495+0100 7fa8b6d95640 -1 *** Caught signal (Aborted) ** in thread 7fa8b6d95640 thread_name:md_log_replay ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa8c383e6f0] 2: /lib64/libc.so.6(+0x8b94c) [0x7fa8c388b94c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa8c416b0a7] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fa8c416b20b] 7: /usr/bin/ceph-mds(+0x1ef0fe) [0x5642daf490fe] 8: /usr/bin/ceph-mds(+0x1ef145) [0x5642daf49145] 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bad) [0x5642db17b67d] 10: (EUpdate::replay(MDSRank*)+0x5d) [0x5642db18477d] 11: (MDLog::_replay_thread()+0x75e) [0x5642db12d00e] 12: /usr/bin/ceph-mds(+0x13c561) [0x5642dae96561] 13: /lib64/libc.so.6(+0x89c02) [0x7fa8c3889c02] 14: /lib64/libc.so.6(+0x10ec40) [0x7fa8c390ec40] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 10/20 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/ 5 rgw_access 1/ 5 rgw_dbstore 1/ 5 rgw_flight 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_t 0/ 5 seastore_cleaner 0/ 5 seastore_epm 0/ 5 seastore_lba 0/ 5 seastore_fixedkv_tree 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 seastore_backref 0/ 5 alienstore 1/ 5 mclock 0/ 5 cyanstore 1/ 5 ceph_exporter 1/ 5 memstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7fa8b6d95640 / md_log_replay max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.pebbles-s3.log --- end dump of recent events --- Our MDS then starts at the beginning of the replay process and continually re-replays the journal until it crashes again at the same point. It looks like our journal has gotten corrupted at this file from what I understand and our journal (worryingly) is exceptionally large where we've had to use a 2 TiB machine just to try and export it. What is causing this issue? Can we do small modifications to the journal or similar to rectify this issue or move the faulty object in the journal out of the bulk object-store to fail (and thus skip) the transaction? We really do not want to go through disaster recovery again (https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#disaster-re…) as this is the 2nd time this has happened to this cluster in the last 4 months and it took over a month to recover the data last time Kindest regards, Ivan -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N249 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

1 0

Using XFS and LVM backends together on the same cluster and hosts
by Özkan Göksu 02 Oct '24

02 Oct '24

Hello folks! I hope you are doing well :) I have a general question about XFS and LVM backend OSD performance and possible effects if they are used together in the same pool. I built a cluster 5 years ago with Nautilus and I used the XFS backend for OSD's. After 5 years they reached me back with ERR state and I see 10K++ slow ops, 29 incomplete PG, 2 inconsistent PG, 34300 unfound objects and 1 osd down due to compactions problem. At the first check-up I found some drives were replaced by others and they used LVM backend for the replaced drives without using wal+db on nvme. In the cluster I have mostly XFS backend drives wal+db on nvme and some LVM drives without wal+db. We have Cephfs and RBD pools on SSD drives and 8+2 EC pool for RGW S3 workload. RGW index stored on SSD pool. 10 nodes: with - 21 x 16TB Toshiba MG08CA16TEY, Firmware:EJ09 | 8+2 EC RGW DATA POOL - 3 x 960GB MZILS960HEHP/007 Firm: GXL0 | Rep 2 RGW index pool - 2 x PM1725B 1.6T PCI-E NVME | 50G WAL+DB for 21x HDD Total HDD raw size 2.8PiB, SSD size 26TiB I started fixing all the problems one by one and I'm gonna recreate these LVM drives without wal+db and I wonder 2 questions: 1- Are there any speed or latency differences on XFS and LVM backend OSD's for 16TB 7200rpm NL-SAS drives. 2- Mixing XFS and LVM backend on the same cluster does have any negative effect or problems? Best regards: Extra note: If you wonder please check the LSBLK output for 1/10 server: NODE-01# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 14.6T 0 disk ├─sda1 8:1 0 100M 0 part /var/lib/ceph/osd/ceph-180 └─sda2 8:2 0 14.6T 0 part sdb 8:16 0 14.6T 0 disk ├─sdb1 8:17 0 100M 0 part /var/lib/ceph/osd/ceph-181 └─sdb2 8:18 0 14.6T 0 part sdc 8:32 0 14.6T 0 disk ├─sdc1 8:33 0 100M 0 part /var/lib/ceph/osd/ceph-182 └─sdc2 8:34 0 14.6T 0 part sdd 8:48 0 14.6T 0 disk ├─sdd1 8:49 0 100M 0 part /var/lib/ceph/osd/ceph-183 └─sdd2 8:50 0 14.6T 0 part sde 8:64 0 14.6T 0 disk ├─sde1 8:65 0 100M 0 part /var/lib/ceph/osd/ceph-185 └─sde2 8:66 0 14.6T 0 part sdf 8:80 0 14.6T 0 disk └─ceph--ef5bd394--8dc9--46a8--a244--0c5d3c1400e3-osd--block--b69c0802--9634--43a5--b4a9--0f36cd8690c5 253:2 0 14.6T 0 lvm sdg 8:96 0 14.6T 0 disk ├─sdg1 8:97 0 100M 0 part /var/lib/ceph/osd/ceph-186 └─sdg2 8:98 0 14.6T 0 part sdh 8:112 0 14.6T 0 disk ├─sdh1 8:113 0 100M 0 part /var/lib/ceph/osd/ceph-187 └─sdh2 8:114 0 14.6T 0 part sdi 8:128 0 14.6T 0 disk ├─sdi1 8:129 0 100M 0 part /var/lib/ceph/osd/ceph-188 └─sdi2 8:130 0 14.6T 0 part sdj 8:144 0 14.6T 0 disk ├─sdj1 8:145 0 100M 0 part /var/lib/ceph/osd/ceph-189 └─sdj2 8:146 0 14.6T 0 part sdk 8:160 0 14.6T 0 disk ├─sdk1 8:161 0 100M 0 part /var/lib/ceph/osd/ceph-190 └─sdk2 8:162 0 14.6T 0 part sdl 8:176 0 14.6T 0 disk ├─sdl1 8:177 0 100M 0 part /var/lib/ceph/osd/ceph-191 └─sdl2 8:178 0 14.6T 0 part sdm 8:192 0 14.6T 0 disk ├─sdm1 8:193 0 100M 0 part /var/lib/ceph/osd/ceph-192 └─sdm2 8:194 0 14.6T 0 part sdn 8:208 0 14.6T 0 disk ├─sdn1 8:209 0 100M 0 part /var/lib/ceph/osd/ceph-193 └─sdn2 8:210 0 14.6T 0 part sdo 8:224 0 14.6T 0 disk ├─sdo1 8:225 0 100M 0 part /var/lib/ceph/osd/ceph-194 └─sdo2 8:226 0 14.6T 0 part sdp 8:240 0 14.6T 0 disk ├─sdp1 8:241 0 100M 0 part /var/lib/ceph/osd/ceph-195 └─sdp2 8:242 0 14.6T 0 part sdq 65:0 0 14.6T 0 disk ├─sdq1 65:1 0 100M 0 part /var/lib/ceph/osd/ceph-196 └─sdq2 65:2 0 14.6T 0 part sdr 65:16 0 14.6T 0 disk ├─sdr1 65:17 0 100M 0 part /var/lib/ceph/osd/ceph-197 └─sdr2 65:18 0 14.6T 0 part sds 65:32 0 14.6T 0 disk ├─sds1 65:33 0 100M 0 part /var/lib/ceph/osd/ceph-198 └─sds2 65:34 0 14.6T 0 part sdt 65:48 0 894.3G 0 disk ├─sdt1 65:49 0 100M 0 part /var/lib/ceph/osd/ceph-199 └─sdt2 65:50 0 894.2G 0 part sdu 65:64 0 894.3G 0 disk └─ceph--e115ee52--429b--4631--92f8--a4282cb43938-osd--block--f4375a1a--6a0e--4797--a496--0b9ba0448607 253:3 0 894.3G 0 lvm sdv 65:80 0 894.3G 0 disk └─ceph--bb838dc5--4eab--4461--a0cf--6439fa12706a-osd--block--f5d007a0--86b2--43ff--8750--0b24c4f823e4 253:1 0 894.3G 0 lvm sdw 65:96 0 14.6T 0 disk └─ceph--d2858c65--6b29--43e8--ad4f--d92b004c2f37-osd--block--905c999e--dd4e--463c--a0e0--93a69ccc7845 253:0 0 14.6T 0 lvm sdx 65:112 0 223.5G 0 disk ├─sdx1 65:113 0 1M 0 part └─sdx2 65:114 0 223.5G 0 part rbd0 252:0 0 15G 0 disk rbd1 252:16 0 6T 0 disk nvme0n1 259:2 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md nvme1n1 259:3 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md

2 2

Optimizations on "high" latency Ceph clusters
by Victor Rodriguez 02 Oct '24

02 Oct '24

Hello, I'm trying to get the most from a Ceph Reef 3 node clusters, with 6 NVMe OSD each. Each node is between 0'6 and 1 millisecond RTT. Obviously performance isn't as good as with local clusters, usually around ~0'2ms. 4k in Q1D1 I/O write request are the most affected by this, as expected, with a performance loss of around 70% from an all local cluster with similar hardware. In general, measurements match the increased latency. Are there any guidelines about what to tune or adjust to get the most of this kind of setups with "high" latencies? Thanks in advance! Victor

3 4

Is there a way to throttle faster osds due to slow ops?
by Szabo, Istvan (Agoda) 02 Oct '24

02 Oct '24

Hi, We have extended our clusters with some new nodes and currently it is impossible to remove from any old node the nvme drive which holding the index pool in the cluster without generating slow ops and cluster performance degradation. Currently how I want to remove is in quincy non cephadm cluster is to crush reweight to 0 and remove. This data movement makes slow ops all the way during the nvme osd out. In my opinion it might be generated from the faster drives harder push on the old servers nvme which makes high iowait on the old nvmes so I want to somehow throttle the new nvmes. Not sure with mclock or with any wait is it possible? (max backfill, osd recover ops and recovery ops priotity is already 1 and balancer max misplaced ratio 0.01). This some of the slow osd says during remove: https://gist.github.com/Badb0yBadb0y/15b51e524a47dfbd2728bbabc18238fc#file-… 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.583707809s, txc = 0x55af2bd2e300 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.541916847s, txc = 0x55af1a035b00 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.533919334s, txc = 0x55af19fafb00 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.904534340s, txc = 0x55af49814c00 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.911001205s, txc = 0x55af24b19800 2024-10-01T11:46:29.601+0700 7f29bf4f8640 0 bluestore(/var/lib/ceph/osd/ceph-91) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.597061634s, txc = 0x55af4fe0fb00 2024-10-01T11:46:30.889+0700 7f29becf7640 4 rocksdb: [db/db_impl/db_impl_write.cc:1736] [default] New memtable created with log file: #280327. Immutable memtables: 0. 2024-10-01T11:46:30.889+0700 7f29becf7640 4 rocksdb: [db/column_family.cc:983] [default] Increasing compaction threads because we have 18 level-0 files 2024-10-01T11:46:30.889+0700 7f29c4512640 4 rocksdb: (Original Log Time 2024/10/01-11:46:30.893378) [db/db_impl/db_impl_compaction_flush.cc:2394] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 2, flush slots scheduled 1, compaction slots scheduled 2 2024-10-01T11:46:30.889+0700 7f29c4512640 4 rocksdb: [db/flush_job.cc:335] [default] [JOB 5604] Flushing memtable with next log file: 280327 2024-10-01T11:46:30.889+0700 7f29c4512640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1727757990893428, "job": 5604, "event": "flush_started", "num_memtables": 1, "num_entries": 2437269, "num_deletes": 2384624, "total_data_size": 233695787, "memory_usage": 278437952, "flush_reason": "Write Buffer Full"} 2024-10-01T11:46:30.889+0700 7f29c4512640 4 rocksdb: [db/flush_job.cc:364] [default] [JOB 5604] Level-0 flush table #280328: started Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 4

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5