September 2024 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

CRC Bad Signature when using KRBD
by jsterr＠deckplosion.de 19 Dec '24

19 Dec '24

Hello Ceph Users, * Problem: we get the following errors when using krbd, we are using rbd for vms. * Workaround: by switching to librbd the errors disappear. * Software: ** Kernel: 6.8.8-2 (parameters: intel_iommu=on iommu=pt pcie_aspm.policy=performance) ** Ceph: 18.2.2 Description/Details: Errors from using krbd with ceph. Side-effects: [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000015af2284 data crc 1221767919 != exp. 282251377 [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000066b200ab data crc 3817026135 != exp. 3925662391 [Wed Aug 21 03:04:17 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:17 2024] libceph: osd13 (1)10.1.4.13:6809 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000008a131738 data crc 2612835980 != exp. 917302924 [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000005160776b data crc 2965872045 != exp. 563323792 [Wed Aug 21 03:04:21 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: osd6 (1)10.1.4.12:6843 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000007e548354 data crc 1265032637 != exp. 2426281931 [Wed Aug 21 03:06:44 2024] libceph: osd0 (1)10.1.4.11:6835 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000009214d802 data crc 2596010853 != exp. 1221875667 [Wed Aug 21 03:06:44 2024] libceph: osd10 (1)10.1.4.12:6809 bad crc/signature [Wed Aug 21 03:06:47 2024] libceph: read_partial_message 000000000f9edc73 data crc 1326019705 != exp. 3079604517 [Wed Aug 21 03:06:47 2024] libceph: osd3 (1)10.1.4.11:6803 bad crc/signature [Wed Aug 21 03:06:50 2024] libceph: read_partial_message 000000004769da61 data crc 3421275194 != exp. 4183754554 [Wed Aug 21 03:06:50 2024] libceph: osd8 (1)10.1.4.12:6835 bad crc/signature [Wed Aug 21 03:06:51 2024] libceph: read_partial_message 0000000044db9a59 data crc 2603270708 != exp. 4150529351 [Wed Aug 21 03:06:51 2024] libceph: osd14 (1)10.1.4.13:6806 bad crc/signature Description/Details 2: vms get problems with buffer i/o errors on rbd-backed virtual disks: Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=885s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=847s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Add. Sense: I/O process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 CDB: Write(10) 2a 00 34 87 48 08 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 CDB: Write (10) 2a 00 34 81 52 70 00 00 58 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281032 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 880890480 op 0x1: (WRITE) flags 0x103000 phys_seg 11 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: EXT4-fs warning (device sdb1): ext4_end_bio:343: I/0 error 10 writing to inode 27525908 starting Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111054, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: buffer_io_error: 21 callbacks suppressed Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on device sdbl, logical block 110159873 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on dev sdbl, logical block 110111055, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=875s Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111056, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111057, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111058, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 CDB: Write(10) 2a 00 34 87 48 10 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111059, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281040 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111060, lost async page write Thanks for helping out. Greetings Jonas

3 4

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

RGW Graphs in cephadm setup
by bkennedy＠cfl.rr.com 26 Oct '24

26 Oct '24

We recently upgraded all our clusters to rocky 9.4 and reef 18.2.4. Two of the clusters show the rgw metrics in the ceph dashboard and the other two don't. I made sure the firewalls were open for ceph-exporter and that Prometheus was gathering the stats on all 4 clusters. For the clusters that are not working, when I do a search on the Prometheus web console at port 9095 on the performance monitoring node using "{instance="rgw1"}, I noticed that the instance_id shows a long number. I did the same search on the clusters that show rgw stats in grafana and they have an instance_id that looks like the drop down in the Grafana dashboard. E.g. "rgw.rgw.rgw1.biewae". So it seems that for some reason, on the clusters that are not working, the ceph-exporter running on the rgw nodes is not pulling the proper instance_id rgw instance. Is there a way to fix the instance_id that ceph-exporter is pulling? I can see this instance id when browsing to the servers http://rgw1:9926/metrics portal. If I run a podman ps, I can see the container running and it has the unique id .biewae showing. I used cephadm to enter the ceph-exporter container on the rgw to see what the logs were saying and I see the same thing repeated: 1 pid path is empty; process metrics won't be fetched for: ceph-client.rgw.rgw.rgw1.biewae.2.94759457265864 Its only the rgw's that show this error in the ceph-exporter log file, the mons and osd nodes show no errors in the log and show up in grafana properly with data. I redeployed ceph-exporter and rgw to the same node in that order, but no change. I did set the default image container to 18.2.4 prior to redeploy cause I ran into that issue as well. Regards, -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw

3 3

cephadm bootstrap ignoring --skip-firewalld
by Kozakis, Anestis 17 Oct '24

17 Oct '24

As I mentioned in my earlier e-mail, new to Ceph, and trying to set up automation to deploy, configure, and manage a Ceph cluster. We configure our Firewall rules through SaltStack. I am passing the -skip-firewalld option to the cephadm bootstrap command, but cephadm seems to ignore the option and configures the firewall anyway. I have even reconfigured the options order to be the same as cephadm boostrap -help but it still ignores the option and configures the firewall. This creates issues as it configures the public zone, which we don't want changed. Below is the command we are using (with obvious settings changed/removed). cephadm bootstrap --mon-ip 10.0.0.0 --mgr-id host.domain.name --fsid [fsid] --ssh-private-key id_rsa --ssh-public-key id_rsa.pub --ssh-user [user] --cluster-network 192.168.0.0/25 --allow-fqdn-hostname --config=./ceph.conf --initial-dashboard-user admin --initial-dashboard-password SuperS3cr3tPassw0rd --dashboard-password-noupdate --skip-firewalld --with-centralized-logging --apply-spec=spec.yaml What am I missing? Anestis Kozakis Systems Administrator - Multi-Level Security Solutions P: + 61 2 6122 0205 M: +61 4 88 376 339 anestis.kozakis(a)raytheon.com.au<mailto:anestis.kozakis@raytheon.com.au> Raytheon Australia Cybersecurity and Information Assurance 4 Brindabella Cct Brindabella Business Park Canberra Airport, ACT 2609 www.raytheonaustralia.com.au<http://www.raytheonaustralia.com.au/> LinkedIn<https://au.linkedin.com/company/raytheon-australia> | Twitter<https://twitter.com/RaytheonAU> | Facebook<https://www.facebook.com/RaytheonAustralia> | Instagram<https://www.instagram.com/raytheonaustralia/>

2 3

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

SLOW_OPS problems
by Tim Sauerbein 15 Oct '24

15 Oct '24

Dear list, I have a small cluster (Reef 18.2.4) with 7 hosts and 3-4 OSDs each (960GB/1.92TB mixed Intel D3-S4610, Samsung SM883, PM897 SSDs): cluster: id: ecff3ce8-539b-443e-a492-da428f4aa9e9 health: HEALTH_OK services: mon: 5 daemons, quorum titan,mangan,kalium,argon,chromium (age 2w) mgr: mangan(active, since 2w), standbys: titan, argon osd: 22 osds: 22 up (since 2w), 22 in (since 3M) data: pools: 2 pools, 513 pgs objects: 2.76M objects, 7.0 TiB usage: 16 TiB used, 15 TiB / 31 TiB avail pgs: 513 active+clean On that cluster RBD volumes for virtual machines are stored. For a couple of months now the cluster reports slow ops for some OSDs and some PGs as laggy. This happens once or twice a day, sometimes more and sometimes not at all for a few days, at completely random times, independent of when snapshots are deleted and trimmed and independent of the I/O load or load on the hosts. After about 30 seconds, during which the write speed goes to zero on the VMs, everything returns to normal. I cannot reproduce the slow ops manually by creating write load on the cluster. Even writing continuously with 300-400 MB/s full speed for 20 minutes does not create any problems. See attached log file for an example of a typical occurrence. I have also measured write load on the disks during the problems with iostat which just shows how writes stall, see also attached. The OSDs with slow ops are completely random, any of the disks would show up once in a while. Current config (I've tried optimising snaptrim and scrub which didn't help): # ceph config dump WHO MASK LEVEL OPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced bdev_async_discard true global advanced bdev_enable_discard true global advanced public_network 10.0.4.0/24 * mon advanced auth_allow_insecure_global_id_reclaim false mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/mode upmap mgr unknown mgr/pg_autoscaler/autoscale_profile scale-up * osd basic osd_memory_target 4294967296 osd advanced osd_pg_max_concurrent_snap_trims 1 osd advanced osd_scrub_begin_hour 23 osd advanced osd_scrub_end_hour 4 osd advanced osd_scrub_sleep 1.000000 osd advanced osd_snap_trim_priority 1 osd advanced osd_snap_trim_sleep 2.000000 osd.0 basic osd_mclock_max_capacity_iops_ssd 29199.674019 osd.1 basic osd_mclock_max_capacity_iops_ssd 31554.530141 osd.10 basic osd_mclock_max_capacity_iops_ssd 25949.821194 osd.11 basic osd_mclock_max_capacity_iops_ssd 26300.596265 osd.12 basic osd_mclock_max_capacity_iops_ssd 25167.331294 osd.13 basic osd_mclock_max_capacity_iops_ssd 21606.610828 osd.14 basic osd_mclock_max_capacity_iops_ssd 27894.095121 osd.15 basic osd_mclock_max_capacity_iops_ssd 25929.047047 osd.16 basic osd_mclock_max_capacity_iops_ssd 15423.600235 osd.17 basic osd_mclock_max_capacity_iops_ssd 25097.493934 osd.18 basic osd_mclock_max_capacity_iops_ssd 25966.188007 osd.19 basic osd_mclock_max_capacity_iops_ssd 23628.746459 osd.2 basic osd_mclock_max_capacity_iops_ssd 32157.280832 osd.20 basic osd_mclock_max_capacity_iops_ssd 22722.682745 osd.3 basic osd_mclock_max_capacity_iops_ssd 33951.086556 osd.4 basic osd_mclock_max_capacity_iops_ssd 22736.907664 osd.5 basic osd_mclock_max_capacity_iops_ssd 21916.777510 osd.6 basic osd_mclock_max_capacity_iops_ssd 29984.954749 osd.7 basic osd_mclock_max_capacity_iops_ssd 26757.965797 osd.8 basic osd_mclock_max_capacity_iops_ssd 22738.921429 osd.9 basic osd_mclock_max_capacity_iops_ssd 24635.156413 Any help would be much appreciated! Thanks, Tim

9 21

Membership additions/removals from the Ceph Steering Committee
by Patrick Donnelly 14 Oct '24

14 Oct '24

Greetings, The Ceph Steering Committee [1] (CSC) -- formerly the "CLT" or Ceph Leadership Team -- was formed 3 years ago with the adoption of our new governance model [2]. The CSC is responsible [4] for electing the Ceph Executive Council [3] (CEC), amending the governance model, and deciding on tactical issues /strategic direction for the Ceph project. After voting for the inaugural members of the CEC (Dan van der Ster, Josh Durgin, and Neha Ohja), we have primarily focused on week-to-week tactical issues confronting the Ceph project during our weekly meetings. This was a natural continuation of our former "Ceph Leadership Team" meetings where we discussed any technical issues or planning for the project. We have not had another election for the Ceph Executive Council membership or any other governance amendment since but are overdue for one according to our governance charter. To begin such an election, it seemed prudent to first vote on any membership changes to the CSC due to natural attrition (e.g. folks have moved on to other projects) or expansion (e.g. new team leads, interested community members, etc.). The CSC weekly call has had several individuals present that are all but officially on the committee. This announcement is a call for anyone (developer, community member, etc.) interested in participating in the governance and strategic initiatives of the Ceph project to please put your or someone else's name forward. There is broad interest in the CSC to have representatives from newer components in Ceph present. If you are on the current list of members and would like to resign from the CSC, please let us know. Some members who have not been present in any recent calls have already been proposed to be removed. Any addition or removal to the CSC must be proposed as a motion with a second and then approved by a 2/3 supermajority vote. We hope to have this election for CSC membership in the coming weeks. It will be performed remotely and offline using the Helios election system (also a FOSS project!). It is deemed necessary to finalize this in advance of any other election topic, including the CEC membership. [1] https://docs.ceph.com/en/latest/governance/#ceph-steering-committee [2] https://docs.ceph.com/en/latest/governance/ [3] https://docs.ceph.com/en/latest/governance/#ceph-executive-council [4] https://docs.ceph.com/en/latest/governance/#id2 [5] https://vote.heliosvoting.org/ Cheers, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 2

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Using XFS and LVM backends together on the same cluster and hosts
by Özkan Göksu 02 Oct '24

02 Oct '24

Hello folks! I hope you are doing well :) I have a general question about XFS and LVM backend OSD performance and possible effects if they are used together in the same pool. I built a cluster 5 years ago with Nautilus and I used the XFS backend for OSD's. After 5 years they reached me back with ERR state and I see 10K++ slow ops, 29 incomplete PG, 2 inconsistent PG, 34300 unfound objects and 1 osd down due to compactions problem. At the first check-up I found some drives were replaced by others and they used LVM backend for the replaced drives without using wal+db on nvme. In the cluster I have mostly XFS backend drives wal+db on nvme and some LVM drives without wal+db. We have Cephfs and RBD pools on SSD drives and 8+2 EC pool for RGW S3 workload. RGW index stored on SSD pool. 10 nodes: with - 21 x 16TB Toshiba MG08CA16TEY, Firmware:EJ09 | 8+2 EC RGW DATA POOL - 3 x 960GB MZILS960HEHP/007 Firm: GXL0 | Rep 2 RGW index pool - 2 x PM1725B 1.6T PCI-E NVME | 50G WAL+DB for 21x HDD Total HDD raw size 2.8PiB, SSD size 26TiB I started fixing all the problems one by one and I'm gonna recreate these LVM drives without wal+db and I wonder 2 questions: 1- Are there any speed or latency differences on XFS and LVM backend OSD's for 16TB 7200rpm NL-SAS drives. 2- Mixing XFS and LVM backend on the same cluster does have any negative effect or problems? Best regards: Extra note: If you wonder please check the LSBLK output for 1/10 server: NODE-01# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 14.6T 0 disk ├─sda1 8:1 0 100M 0 part /var/lib/ceph/osd/ceph-180 └─sda2 8:2 0 14.6T 0 part sdb 8:16 0 14.6T 0 disk ├─sdb1 8:17 0 100M 0 part /var/lib/ceph/osd/ceph-181 └─sdb2 8:18 0 14.6T 0 part sdc 8:32 0 14.6T 0 disk ├─sdc1 8:33 0 100M 0 part /var/lib/ceph/osd/ceph-182 └─sdc2 8:34 0 14.6T 0 part sdd 8:48 0 14.6T 0 disk ├─sdd1 8:49 0 100M 0 part /var/lib/ceph/osd/ceph-183 └─sdd2 8:50 0 14.6T 0 part sde 8:64 0 14.6T 0 disk ├─sde1 8:65 0 100M 0 part /var/lib/ceph/osd/ceph-185 └─sde2 8:66 0 14.6T 0 part sdf 8:80 0 14.6T 0 disk └─ceph--ef5bd394--8dc9--46a8--a244--0c5d3c1400e3-osd--block--b69c0802--9634--43a5--b4a9--0f36cd8690c5 253:2 0 14.6T 0 lvm sdg 8:96 0 14.6T 0 disk ├─sdg1 8:97 0 100M 0 part /var/lib/ceph/osd/ceph-186 └─sdg2 8:98 0 14.6T 0 part sdh 8:112 0 14.6T 0 disk ├─sdh1 8:113 0 100M 0 part /var/lib/ceph/osd/ceph-187 └─sdh2 8:114 0 14.6T 0 part sdi 8:128 0 14.6T 0 disk ├─sdi1 8:129 0 100M 0 part /var/lib/ceph/osd/ceph-188 └─sdi2 8:130 0 14.6T 0 part sdj 8:144 0 14.6T 0 disk ├─sdj1 8:145 0 100M 0 part /var/lib/ceph/osd/ceph-189 └─sdj2 8:146 0 14.6T 0 part sdk 8:160 0 14.6T 0 disk ├─sdk1 8:161 0 100M 0 part /var/lib/ceph/osd/ceph-190 └─sdk2 8:162 0 14.6T 0 part sdl 8:176 0 14.6T 0 disk ├─sdl1 8:177 0 100M 0 part /var/lib/ceph/osd/ceph-191 └─sdl2 8:178 0 14.6T 0 part sdm 8:192 0 14.6T 0 disk ├─sdm1 8:193 0 100M 0 part /var/lib/ceph/osd/ceph-192 └─sdm2 8:194 0 14.6T 0 part sdn 8:208 0 14.6T 0 disk ├─sdn1 8:209 0 100M 0 part /var/lib/ceph/osd/ceph-193 └─sdn2 8:210 0 14.6T 0 part sdo 8:224 0 14.6T 0 disk ├─sdo1 8:225 0 100M 0 part /var/lib/ceph/osd/ceph-194 └─sdo2 8:226 0 14.6T 0 part sdp 8:240 0 14.6T 0 disk ├─sdp1 8:241 0 100M 0 part /var/lib/ceph/osd/ceph-195 └─sdp2 8:242 0 14.6T 0 part sdq 65:0 0 14.6T 0 disk ├─sdq1 65:1 0 100M 0 part /var/lib/ceph/osd/ceph-196 └─sdq2 65:2 0 14.6T 0 part sdr 65:16 0 14.6T 0 disk ├─sdr1 65:17 0 100M 0 part /var/lib/ceph/osd/ceph-197 └─sdr2 65:18 0 14.6T 0 part sds 65:32 0 14.6T 0 disk ├─sds1 65:33 0 100M 0 part /var/lib/ceph/osd/ceph-198 └─sds2 65:34 0 14.6T 0 part sdt 65:48 0 894.3G 0 disk ├─sdt1 65:49 0 100M 0 part /var/lib/ceph/osd/ceph-199 └─sdt2 65:50 0 894.2G 0 part sdu 65:64 0 894.3G 0 disk └─ceph--e115ee52--429b--4631--92f8--a4282cb43938-osd--block--f4375a1a--6a0e--4797--a496--0b9ba0448607 253:3 0 894.3G 0 lvm sdv 65:80 0 894.3G 0 disk └─ceph--bb838dc5--4eab--4461--a0cf--6439fa12706a-osd--block--f5d007a0--86b2--43ff--8750--0b24c4f823e4 253:1 0 894.3G 0 lvm sdw 65:96 0 14.6T 0 disk └─ceph--d2858c65--6b29--43e8--ad4f--d92b004c2f37-osd--block--905c999e--dd4e--463c--a0e0--93a69ccc7845 253:0 0 14.6T 0 lvm sdx 65:112 0 223.5G 0 disk ├─sdx1 65:113 0 1M 0 part └─sdx2 65:114 0 223.5G 0 part rbd0 252:0 0 15G 0 disk rbd1 252:16 0 6T 0 disk nvme0n1 259:2 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md nvme1n1 259:3 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md

2 2

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Ceph Steering Committee (a.k.a. CLT) Meeting Minutes 2024-09-30
by Patrick Donnelly 30 Sep '24

30 Sep '24

Hello, Today we only discussed the upcoming election [5]. - A test election was conducted using Helios: [3] - A date for the election was discussed. The eventual consensus was that we should hold the election offline and meet during our normal call for any points of discussion. - The executive council's practical responsibilities have been expanded, thanks to Neha, Josh, and Dan. It is available at [4]. - The remaining time was used to discuss proposals [5] for membership [6] changes on the Ceph Steering Committee in advance of the vote on other agenda items (including the Ceph Executive Council membership). + The focus was on removing members who are no longer active on the project. We did not finish. + Many discussions were had on the mechanisms/criteria for membership addition/removal. These included: lack of attendance as a reason for removal; whether a vote should be had at all for removal; or, the number of representatives for each Ceph component. No consensus was reached. + Mark Nelson will be following up with individuals on the CSC who've been proposed to be removed from the membership roll. (Mainly, the criteria used seemed to be lack of attendance.) In general, I don't think anyone wants to vote for removal of someone who wishes to participate in the CSC. + Next week we will focus on proposals for adding members to the CSC. I will be sending an email shortly concerning that. Minutes: https://pad.ceph.com/p/clt-weekly-minutes There is a Ceph Developer Monthly meeting on Wednesday: [2] [1] https://pad.ceph.com/p/clt-weekly-minutes [2] https://www.meetup.com/ceph-user-group/events/303725485/ [3] https://vote.heliosvoting.org/helios/e/CLT-Test-Election/vote [4] https://pad.ceph.com/p/ceph-exec-council-responsibilities [5] https://pad.ceph.com/p/css-vote-2024q4 [6] https://docs.ceph.com/en/latest/governance/#id4 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Dashboard: frequent queries for balancer status
by Eugen Block 30 Sep '24

30 Sep '24

Hi, I just noticed across different Ceph versions that when browsing the dashboard, the MGR is logging lots of prometheus queries for the balancer status: Sep 30 11:15:55 host2 ceph-mgr[3993215]: log_channel(cluster) log [DBG] : pgmap v25341: 381 pgs: 381 active+clean; 3.9 GiB data, 69 GiB used, 311 GiB / 380 GiB avail; 954 B/s rd, 1 op/s Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:56 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:57 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch Sep 30 11:15:59 host2 conmon[3993190]: 192.168.168.62 - - [30/Sep/2024:09:15:59] "GET /metrics HTTP/1.1" 200 71320 "" "Prometheus/2.33.4" Sep 30 11:16:02 host2 ceph-mgr[3993215]: log_channel(audit) log [DBG] : from='mon.1 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch At least I assume that this is prometheus if I interpret [0] correctly. In this case, the balancer is even disabled. Why is it necessary to query the balancer status that often? I couldn't find any useful config related to this. This was noticed on Pacific (16.2.15) and Squid (19.1.1), both with Prometheus version 2.33.4. I haven't tried other Prometheus versions yet. I will check it with a newer Prometheus version as well. Thanks, Eugen [0] https://github.com/ceph/ceph/blob/2d93ad5a87a720c9756ea1eac9d95e1067b80976/…

1 0

device_health_metrics pool automatically recreated
by Eugen Block 29 Sep '24

29 Sep '24

Hi, I just looked into one customer cluster that we upgraded some time ago from Octopus to Quincy (17.2.6) and I'm wondering why there are still both pools, "device_health_metrics" and ".mgr". According to the docs [0], it's supposed to be renamed: > Prior to Quincy, the devicehealth module created a > device_health_metrics pool to store device SMART statistics. With > Quincy, this pool is automatically renamed to be the common manager > module pool. Now only .mgr has data while device_health_metrics is empty, but it has a newer ID: ses01:~ # ceph df | grep -E "device_health|.mgr" .mgr 1 1 68 MiB 18 204 MiB 0 254 TiB device_health_metrics 15 1 0 B 0 0 B 0 254 TiB On a test cluster (meanwhile upgraded to latest Reef) I see the same: ceph01:~ # ceph df | grep -E "device_health_metrics|.mgr" .mgr 38 1 577 KiB 2 1.7 MiB 0 71 GiB device_health_metrics 45 1 0 B 0 0 B 0 71 GiB Since there are still many users who haven't upgraded to >= Quincy yet, this should be clarified/fixed. I briefly checked tracker.ceph.com, but didn't find anything related to this. I'm currently trying to reproduce it on a one-node test cluster which I upgraded from Pacific to Quincy, but no results yet, only that the renaming was successful. But for the other clusters I don't have enough logs to find out how/why the device_health_metrics pool had been recreated. Thanks, Eugen [0] https://docs.ceph.com/en/latest/mgr/administrator/#module-pool

2 3

RGW returning HTTP 500 during resharding
by Floris Bos 29 Sep '24

29 Sep '24

Hi, I am doing an attempt at loading a large number (billions) of tiny objects from other existing system into Ceph (18.2.4 reef) RadosGW/S3. I wrote a little program for that using the AWS S3 Java SDK (using CRT/S3AsyncClient), and am using 250 simultanous connections to get the data in. However I am experiencing problems when it starts to dynamically reshard. The first problem I was experiencing is that the SDK then complains connection throughput is too low, and errors out: == Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Http connectio n channel shut down due to failure to meet throughput minimum == While I configured the SDK to retry several times on errors, that does not seem to be happening. Or perhaps it does retry, but using one of the other connections, which obviously will fail as well for the same reason. Anyway, I shoved this issue under the carpet by configuring a very high timeout (creating a S3CrtConnectionHealthConfiguration and setting minimumThroughputTimeout() ) That seemed to work around it for a while. However I found out not much later that if the resharding takes longer than some minutes, RadosGW starts returning HTTP status code 500 to PUT requests: == Sep 27 18:14:33 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:15:03 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:21:59 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:22:04 myserver radosgw[6395]: req 9716400394278448257 450.223876953s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 9716400394278448257 450.223876953s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-16f7e. ret=(2 300) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: req 14148855427882423133 450.231872559s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 14148855427882423133 450.231872559s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-15b3. ret=(23 00) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: req 2645819454948921814 450.239868164s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 2645819454948921814 450.239868164s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-147ea. ret=(23 00) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfc7865710 op status=-2300 http_status=500 latency=450.223876953s ====== Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfc7ff4710 op status=-2300 http_status=500 latency=450.231872559s ====== Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfca7c3710 op status=-2300 http_status=500 latency=450.239868164s ====== Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfc7865710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.851 +0000] "PUT /datastore/some-obj-16f7e HTTP/1.1" 500 905 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.223876953s Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfc7ff4710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.843 +0000] "PUT /datastore/some-obj-15b3 HTTP/1.1" 500 293 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.231872559s Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfca7c3710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.835 +0000] "PUT /datastore/some-obj-147ea HTTP/1.1" 500 318 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.239868164s S == AWS SDK considers 500 quite fatal: == Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response code indicates internal server error == No retries. Is it expected that resharding can take so long? (in a setup with all NVMe drives) And is it correct behavior that it returns HTTP response code 500, instead of something that could indicate it is a retry'able condition? If I would add my own code that does retry for a very long time, is there any way I can detect the 500 is due to the resharding, instead of some other condition that do is fatal? Also, is there any more efficient way to get a large amount of objects into Ceph than individual PUTs? Yours sincerely, Floris Bos

3 5

Re: Syncing Error - (4) Incoming failed! ceph-users@ceph.io
by ceph.io 29 Sep '24

29 Sep '24

1 0

Undeliverable: Incoming Messages Failure
by Postmaster 29 Sep '24

29 Sep '24

1 0

RGW Graphs in cephadm setup
by brentk＠cfl.rr.com 28 Sep '24

28 Sep '24

We recently upgraded all our clusters to rocky 9.4 and reef 18.2.4. Two of the clusters show the rgw metrics in the ceph dashboard and the other two don't. I made sure the firewalls were open for ceph-exporter and that Prometheus was gathering the stats on all 4 clusters. For the clusters that are not working, when I do a search on the Prometheus web console at port 9095 on the performance monitoring node using "{instance="rgw1"}, I noticed that the instance_id shows a long number. I did the same search on the clusters that show rgw stats in grafana and they have an instance_id that looks like the drop down in the Grafana dashboard. E.g. "rgw.rgw.rgw1.biewae". So it seems that for some reason, on the clusters that are not working, the ceph-exporter running on the rgw nodes is not pulling the proper instance_id rgw instance. Is there a way to fix the instance_id that ceph-exporter is pulling? I can see this instance id when browsing to the servers http://rgw1:9926/metrics portal. If I run a podman ps, I can see the container running and it has the unique id .biewae showing. I used cephadm to enter the ceph-exporter container on the rgw to see what the logs were saying and I see the same thing repeated: 1 pid path is empty; process metrics won't be fetched for: ceph-client.rgw.rgw.rgw1.biewae.2.94759457265864 Its only the rgw's that show this error in the ceph-exporter log file, the mons and osd nodes show no errors in the log and show up in grafana properly with data. I redeployed ceph-exporter and rgw to the same node in that order, but no change. I did set the default image container to 18.2.4 prior to redeploy cause I ran into that issue as well. Regards, -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw

1 0

Old MDS container version when: Ceph orch apply mds
by opositorvlc＠mail.com 28 Sep '24

28 Sep '24

Hi All, I migrated my CEPH 18.2.2 cluster from a non cephadm configuration. All goes fine except MDS service was deployed in a old version: 17.0.0 I'm trying to deploy MDS daemons using ceph orch but CEPH always download an old MDS image from docker. How could I deploy the MDS service in the same 18.2.2 version that the rest of services? [root@master1 ~]# ceph orch apply mds datafs --placement="2 master1 master2" [root@master1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.datafs.master1.gcpovr master1 running (36m) 6m ago 36m 37.2M - 17.0.0-7183-g54142666 75e3d7089cea 96682779c7ad mds.datafs.master2.oqaxuy master2 running (36m) 6m ago 36m 33.1M - 17.0.0-7183-g54142666 75e3d7089cea a9a647f87c83 mgr.master master1 running (16h) 6m ago 17h 448M - 18.2.2 3c937764e6f5 70f06fa05b70 mgr.master2 master2 running (16h) 6m ago 17h 524M - 18.2.2 3c937764e6f5 2d0d5376d8b3 mon.master master1 running (16h) 6m ago 17h 384M 2048M 18.2.2 3c937764e6f5 66a65017ce29 mon.master2 master2 running (16h) 6m ago 17h 380M 2048M 18.2.2 3c937764e6f5 51d783a9e36c osd.0 osd00 running (16h) 3m ago 17h 432M 4096M 18.2.2 3c937764e6f5 fedff66f5ed2 osd.1 osd00 running (16h) 3m ago 17h 475M 4096M 18.2.2 3c937764e6f5 24e24a1a22e6 osd.2 osd00 running (16h) 3m ago 17h 516M 4096M 18.2.2 3c937764e6f5 ccd05451b739 osd.3 osd00 running (16h) 3m ago 17h 454M 4096M 18.2.2 3c937764e6f5 f6d8f13c8aaf osd.4 master1 running (16h) 6m ago 17h 525M 4096M 18.2.2 3c937764e6f5 a2dcf9f1a9b7 osd.5 master2 running (16h) 6m ago 17h 331M 4096M 18.2.2 3c937764e6f5 b0011e8561a4 [root@master1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mds.datafs 2/2 6m ago 46s master1;master2;count:2 mgr 2/0 6m ago - <unmanaged> mon 2/0 6m ago - <unmanaged> osd 6 6m ago - <unmanaged> [root@master1 ~]# ceph versions { "mon": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 2 }, "mgr": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 2 }, "osd": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 6 }, "mds": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2 }, "overall": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 10 } } [root@master1 ~]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/ceph/ceph v18.2.2 3c937764e6f5 7 weeks ago 1.28 GB quay.io/ceph/ceph v18 3c937764e6f5 7 weeks ago 1.28 GB registry.access.redhat.com/ubi8 latest c70d72aaebb4 3 months ago 212 MB quay.io/ceph/ceph v16 0d668911f040 23 months ago 1.27 GB quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 2 years ago 571 MB quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 2 years ago 205 MB quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years ago 22.3 MB quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 2 years ago 58.9 MB docker.io/ceph/daemon-base latest-master-devel 75e3d7089cea 2 years ago 1.29 GB

3 2

17.2.8 release date?
by Szabo, Istvan (Agoda) 28 Sep '24

28 Sep '24

Hi, Do we know roughly when the 17.2.8 Quincy is going to be released? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

v19.2.0 Squid released
by Laura Flores 27 Sep '24

27 Sep '24

We're very happy to announce the first stable release of the Squid series. We express our gratitude to all members of the Ceph community who contributed by proposing pull requests, testing this release, providing feedback, and offering valuable suggestions. Highlights: RADOS * BlueStore has been optimized for better performance in snapshot-intensive workloads. * BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. * Other improvements include more flexible EC configurations, an OpTracker to help debug mgr module issues, and better scrub scheduling. Dashboard * Improved navigation layout CephFS * Support for managing CephFS snapshots and clones, as well as snapshot schedule management * Manage authorization capabilities for CephFS resources * Helpers on mounting a CephFS volume RBD * diff-iterate can now execute locally, bringing a dramatic performance improvement for QEMU live disk synchronization and backup use cases. * Support for cloning from non-user type snapshots is added. * rbd-wnbd driver has gained the ability to multiplex image mappings. RGW * The User Accounts feature unlocks several new AWS-compatible IAM APIs for the self-service management of users, keys, groups, roles, policy and more. Crimson/Seastore * Crimson's first tech preview release! Supporting RBD workloads on Replicated pools. For more information please visit: https://ceph.io/en/news/crimson We encourage you to read the full release notes at https://ceph.io/en/news/blog/2024/v19-2-0-squid-released/ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 16063ff2022298c9300e49a547a16ffda59baf13 -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 2

WAL on NVMe/SSD not used after OSD/HDD replace
by mailing-lists 27 Sep '24

27 Sep '24

Dear Ceph-users, I have a problem that I'd like to have your input for. Preface: I have got a test-cluster and a productive-cluster. Both are setup the same and both are having the same "issue". I am running Ubuntu 22.04 and deployed ceph 17.2.3 via cephadm. Upgraded to 17.2.7 later on, which is the version we are currently running. Since the issue seem to be the exact same on the test-cluster, I will post test-cluster-outputs here for better readability. The issue: I have replaced disks and after the replacement, it does not show that it would use the NVMe as WAL device anymore. The LV still exists, but the metadata of the osd does not show it, as it would be with any other osd/hdd, that hasnt been replaced. ODS.1 (incorrect, bluefs_dedicated_wal: "0") ``` { "id": 1, "arch": "x86_64", "back_addr": "[v2:192.168.6.241:6802/3213655489,v1:192.168.6.241:6803/3213655489]", "back_iface": "", "bluefs": "1", "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev_node": "/dev/dm-3", "bluestore_bdev_devices": "sdd", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_optimal_io_size": "0", "bluestore_bdev_partition_path": "/dev/dm-3", "bluestore_bdev_rotational": "1", "bluestore_bdev_size": "17175674880", "bluestore_bdev_support_discard": "1", "bluestore_bdev_type": "hdd", "bluestore_min_alloc_size": "4096", "ceph_release": "quincy", "ceph_version": "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)", "ceph_version_short": "17.2.7", "ceph_version_when_created": "", "container_hostname": "bi-ubu-srv-ceph2-01", "container_image": "quay.io/ceph/ceph@sha256:28323e41a7d17db238bdcc0a4d7f38d272f75c1a499bc30f59b0b504af132c6b", "cpu": "AMD EPYC 75F3 32-Core Processor", "created_at": "", "default_device_class": "hdd", "device_ids": "sdd=QEMU_HARDDISK_drive-scsi3", "device_paths": "sdd=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:3:0", "devices": "sdd", "distro": "centos", "distro_description": "CentOS Stream 8", "distro_version": "8", "front_addr": "[v2:<my_public_ip>.241:6800/3213655489,v1:<my_public_ip>.241:6801/3213655489]", "front_iface": "", "hb_back_addr": "[v2:192.168.6.241:6806/3213655489,v1:192.168.6.241:6807/3213655489]", "hb_front_addr": "[v2:<my_public_ip>.241:6804/3213655489,v1:<my_public_ip>.241:6805/3213655489]", "hostname": "bi-ubu-srv-ceph2-01", "journal_rotational": "1", "kernel_description": "#132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024", "kernel_version": "5.15.0-122-generic", "mem_swap_kb": "4018172", "mem_total_kb": "5025288", "network_numa_unknown_ifaces": "back_iface,front_iface", "objectstore_numa_unknown_devices": "sdd", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-1", "osd_objectstore": "bluestore", "osdspec_affinity": "dashboard-admin-1661853488642", "rotational": "1" } ``` ODS.0 (correct, bluefs_dedicated_wal: "1") ``` { "id": 0, "arch": "x86_64", "back_addr": "[v2:192.168.6.241:6810/3249286142,v1:192.168.6.241:6811/3249286142]", "back_iface": "", "bluefs": "1", "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "1", "bluefs_single_shared_device": "0", "bluefs_wal_access_mode": "blk", "bluefs_wal_block_size": "4096", "bluefs_wal_dev_node": "/dev/dm-0", "bluefs_wal_devices": "sdb", "bluefs_wal_driver": "KernelDevice", "bluefs_wal_optimal_io_size": "0", "bluefs_wal_partition_path": "/dev/dm-0", "bluefs_wal_rotational": "0", "bluefs_wal_size": "4290772992", "bluefs_wal_support_discard": "1", "bluefs_wal_type": "ssd", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev_node": "/dev/dm-2", "bluestore_bdev_devices": "sdc", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_optimal_io_size": "0", "bluestore_bdev_partition_path": "/dev/dm-2", "bluestore_bdev_rotational": "1", "bluestore_bdev_size": "17175674880", "bluestore_bdev_support_discard": "1", "bluestore_bdev_type": "hdd", "bluestore_min_alloc_size": "4096", "ceph_release": "quincy", "ceph_version": "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)", "ceph_version_short": "17.2.7", "ceph_version_when_created": "", "container_hostname": "bi-ubu-srv-ceph2-01", "container_image": "quay.io/ceph/ceph@sha256:28323e41a7d17db238bdcc0a4d7f38d272f75c1a499bc30f59b0b504af132c6b", "cpu": "AMD EPYC 75F3 32-Core Processor", "created_at": "", "default_device_class": "hdd", "device_ids": "sdb=QEMU_HARDDISK_drive-scsi1,sdc=QEMU_HARDDISK_drive-scsi2", "device_paths": "sdb=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:1:0,sdc=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:2:0", "devices": "sdb,sdc", "distro": "centos", "distro_description": "CentOS Stream 8", "distro_version": "8", "front_addr": "[v2:<my_public_ip>.241:6808/3249286142,v1:<my_public_ip>.241:6809/3249286142]", "front_iface": "", "hb_back_addr": "[v2:192.168.6.241:6814/3249286142,v1:192.168.6.241:6815/3249286142]", "hb_front_addr": "[v2:<my_public_ip>.241:6812/3249286142,v1:<my_public_ip>.241:6813/3249286142]", "hostname": "bi-ubu-srv-ceph2-01", "journal_rotational": "0", "kernel_description": "#132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024", "kernel_version": "5.15.0-122-generic", "mem_swap_kb": "4018172", "mem_total_kb": "5025288", "network_numa_unknown_ifaces": "back_iface,front_iface", "objectstore_numa_unknown_devices": "sdb,sdc", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-0", "osd_objectstore": "bluestore", "osdspec_affinity": "dashboard-admin-1661853488642", "rotational": "1" } ``` The LV that should contain the WAL is existing: ``` :/# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368 -wi-ao---- <16.00g osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35 -wi-ao---- <16.00g osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd ceph-a84cbc55-33b5-4c31-8426-66e375fc157b -wi-ao---- <4.00g osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431 ceph-a84cbc55-33b5-4c31-8426-66e375fc157b -wi-a----- <4.00g ``` We can also see there, that there is no (o) as in open in the LV attributes. For good measure, this is the pattern for the orch osd service. ``` service_type: osd service_id: dashboard-admin-1661853488642 service_name: osd.dashboard-admin-1661853488642 placement: host_pattern: '*' spec: data_devices: size: 16GB db_devices: rotational: false filter_logic: AND objectstore: bluestore wal_devices: rotational: false status: created: '2022-08-30T09:58:11.210571Z' last_refresh: '2024-09-27T12:19:14.619434Z' running: 16 size: 16 ``` "ceph-volume lvm list" looks fine, in my opinion, as it should be: ``` { "0": [ { "devices": [ "/dev/sdc" ], "lv_name": "osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "lv_path": "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "lv_size": "17175674880", "lv_tags": "ceph.block_device=/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.block_uuid=3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.osd_id=0,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=block,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd,ceph.wal_uuid=BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et", "lv_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk", "name": "osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "path": "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "tags": { "ceph.block_device": "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "ceph.block_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk", "ceph.cephx_lockbox_secret": "", "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a", "ceph.cluster_name": "ceph", "ceph.crush_device_class": "", "ceph.encrypted": "0", "ceph.osd_fsid": "d472bf9f-c17d-4939-baf5-514a07db66bc", "ceph.osd_id": "0", "ceph.osdspec_affinity": "dashboard-admin-1661853488642", "ceph.type": "block", "ceph.vdo": "0", "ceph.wal_device": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "ceph.wal_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et" }, "type": "block", "vg_name": "ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368" }, { "devices": [ "/dev/sdb" ], "lv_name": "osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "lv_path": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "lv_size": "4290772992", "lv_tags": "ceph.block_device=/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.block_uuid=3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.osd_id=0,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=wal,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd,ceph.wal_uuid=BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et", "lv_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et", "name": "osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "path": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "tags": { "ceph.block_device": "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "ceph.block_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk", "ceph.cephx_lockbox_secret": "", "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a", "ceph.cluster_name": "ceph", "ceph.crush_device_class": "", "ceph.encrypted": "0", "ceph.osd_fsid": "d472bf9f-c17d-4939-baf5-514a07db66bc", "ceph.osd_id": "0", "ceph.osdspec_affinity": "dashboard-admin-1661853488642", "ceph.type": "wal", "ceph.vdo": "0", "ceph.wal_device": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "ceph.wal_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et" }, "type": "wal", "vg_name": "ceph-a84cbc55-33b5-4c31-8426-66e375fc157b" } ], "1": [ { "devices": [ "/dev/sdd" ], "lv_name": "osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "lv_path": "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "lv_size": "17175674880", "lv_tags": "ceph.block_device=/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f,ceph.block_uuid=QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d94bda82-e59f-4d3d-81cd-28ea69c5e02f,ceph.osd_id=1,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=block,ceph.vdo=0", "lv_uuid": "QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO", "name": "osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "path": "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "tags": { "ceph.block_device": "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "ceph.block_uuid": "QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO", "ceph.cephx_lockbox_secret": "", "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a", "ceph.cluster_name": "ceph", "ceph.crush_device_class": "", "ceph.encrypted": "0", "ceph.osd_fsid": "d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "ceph.osd_id": "1", "ceph.osdspec_affinity": "dashboard-admin-1661853488642", "ceph.type": "block", "ceph.vdo": "0" }, "type": "block", "vg_name": "ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35" }, { "devices": [ "/dev/sdb" ], "lv_name": "osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "lv_path": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "lv_size": "4290772992", "lv_tags": "ceph.block_device=/dev/ceph-179656b5-6783-4b4e-bb7e-695d4523072f/osd-block-7a1d0007-71ff-4011-8a18-e6de1499cbdf,ceph.block_uuid=gDt63k-v0Qd-89WE-bOGO-cmZv-Sc8v-tYOagJ,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=7a1d0007-71ff-4011-8a18-e6de1499cbdf,ceph.osd_id=1,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=wal,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431,ceph.wal_uuid=UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq", "lv_uuid": "UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq", "name": "osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "path": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "tags": { "ceph.block_device": "/dev/ceph-179656b5-6783-4b4e-bb7e-695d4523072f/osd-block-7a1d0007-71ff-4011-8a18-e6de1499cbdf", "ceph.block_uuid": "gDt63k-v0Qd-89WE-bOGO-cmZv-Sc8v-tYOagJ", "ceph.cephx_lockbox_secret": "", "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a", "ceph.cluster_name": "ceph", "ceph.crush_device_class": "", "ceph.encrypted": "0", "ceph.osd_fsid": "7a1d0007-71ff-4011-8a18-e6de1499cbdf", "ceph.osd_id": "1", "ceph.osdspec_affinity": "dashboard-admin-1661853488642", "ceph.type": "wal", "ceph.vdo": "0", "ceph.wal_device": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "ceph.wal_uuid": "UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq" }, "type": "wal", "vg_name": "ceph-a84cbc55-33b5-4c31-8426-66e375fc157b" } ] } ``` We can see two devices for osd.1, but the osd does not seem to use the WAL device. How can i tell osd.1 to use its WAL again? Why does the orch osd service not do it anyways? When i was replacing the disks i was using ``` ceph orch osd rm 1 --replace ``` The osd got back up and refilled, but without WAL. While I was diggin into this, i realized that the DB is not on the NVME, but on the HDD itself. Even though, i was specifying it in the pattern above: db_devices: rotational: false Shouldnt the db then be located on the NVMe/SSD where the wal is at? At least that is what the dashboard suggest, when creating OSDs. Of course I chose the NVME/SSD for DB and WAL. I hope this is somewhat readable and coherent. Thank you in advance! Best wishes and a nice weekend!

4 3

Mds daemon damaged - assert failed
by Kyriazis, George 27 Sep '24

27 Sep '24

Hello ceph users, I am in the unfortunate situation of having a status of “1 mds daemon damaged”. Looking at the logs, I see that the daemon died with an assert as follows: ./src/osdc/Journaler.cc: 1368: FAILED ceph_assert(trim_to > trimming_pos) ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12a) [0x73a83189d7d9] 2: /usr/lib/ceph/libceph-common.so.2(+0x29d974) [0x73a83189d974] 3: (Journaler::_trim()+0x671) [0x57235caa70b1] 4: (Journaler::_finish_write_head(int, Journaler::Header&, C_OnFinisher*)+0x171) [0x57235caaa8f1] 5: (Context::complete(int)+0x9) [0x57235c716849] 6: (Finisher::finisher_thread_entry()+0x16d) [0x73a83194659d] 7: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x73a8310a8134] 8: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x73a8311287dc] 0> 2024-09-23T14:10:26.490-0500 73a822c006c0 -1 *** Caught signal (Aborted) ** in thread 73a822c006c0 thread_name:MR_Finisher ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x73a83105b050] 2: /lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c) [0x73a8310a9e2c] 3: gsignal() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x185) [0x73a83189d834] 6: /usr/lib/ceph/libceph-common.so.2(+0x29d974) [0x73a83189d974] 7: (Journaler::_trim()+0x671) [0x57235caa70b1] 8: (Journaler::_finish_write_head(int, Journaler::Header&, C_OnFinisher*)+0x171) [0x57235caaa8f1] 9: (Context::complete(int)+0x9) [0x57235c716849] 10: (Finisher::finisher_thread_entry()+0x16d) [0x73a83194659d] 11: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x73a8310a8134] 12: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x73a8311287dc] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. As listed above, I am running 18.2.2 on a proxmox cluster with a hybrid hdd/sdd setup. 2 cephfs filesystems. The mds responsible for the hdd filesystem is the one that died. Output of ceph -s follows: root@vis-mgmt:~/bin# ceph -s cluster: id: ec2c9542-dc1b-4af6-9f21-0adbcabb9452 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 5 pgs not scrubbed in time 1 daemons have recently crashed services: mon: 5 daemons, quorum vis-hsw-01,vis-skx-01,vis-clx-15,vis-clx-04,vis-icx-00 (age 6m) mgr: vis-hsw-02(active, since 13d), standbys: vis-skx-02, vis-hsw-04, vis-clx-08, vis-clx-02 mds: 1/2 daemons up, 5 standby osd: 97 osds: 97 up (since 3h), 97 in (since 4d) data: volumes: 1/2 healthy, 1 recovering; 1 damaged pools: 14 pools, 1961 pgs objects: 223.70M objects, 304 TiB usage: 805 TiB used, 383 TiB / 1.2 PiB avail pgs: 1948 active+clean 9 active+clean+scrubbing+deep 4 active+clean+scrubbing io: client: 86 KiB/s rd, 5.5 MiB/s wr, 64 op/s rd, 26 op/s wr I tried restarting all the mds deamons but they are all marked as “standby”. I also tried restarting all the mons and then the mds daemons again, but that didn’t help. Much help is appreciated! Thank you! George

4 8

All monitors fall down simultaneously when I try to map rbd on client
by Alex from North 27 Sep '24

27 Sep '24

Hello everybody, found intresting thing: for some reason ALL the monitors crash when I try to rbd map on client host. here is my pool: root@ceph1:~# ceph osd pool ls iotest Here is my rbd in this pool: root@ceph1:~# rbd ls -p iotest test1 this is a client creds to connect to this pool: [client.iotest] key = AQASVfZm5bPGLBAAXyPWqJvNMBsXsJQcFrSAhg== caps mgr = "profile rbd pool=iotest" caps mon = "profile rbd" caps osd = "profile rbd pool=iotest" This is rbmap file on a client host: root@node-stat:/etc/ceph# cat rbdmap # RbdDevice Parameters #poolname/imagename id=client,keyring=/etc/ceph/ceph.client.keyring iotest/test1 id=iotest,keyring=/etc/ceph/ceph.client.iotest.keyring So, when I press Enter on command rbd map iotest/test1 --id iotest in the same moment ALL the mons go down. I pul log on pastebin as it is quite long https://pastebin.com/iCr8pY1r All the hints are appreciated. Thanks in advance.

3 5

Restore a pool from snapshot
by Pavel Kaygorodov 27 Sep '24

27 Sep '24

Hi! May be a dumb question, sorry, but how I can restore a whole pool from a snapshot? I have made a snapshot with 'rados mksnap', but there is no command to restore a whole snapshot, only one object may be specified for rollback. Is it possible to restore all objects? Thanks in advance, Pavel.

2 1

Quincy: osd_pool_default_crush_rule being ignored?
by Florian Haas 27 Sep '24

27 Sep '24

Hello everyone, my cluster has two CRUSH rules: the default replicated_rule (rule_id 0), and another rule named rack-aware (rule_id 1). Now, if I'm not misreading the config reference, I should be able to define that all future-created pools use the rack-aware rule, by setting osd_pool_default_crush_rule to 1. I've verified that this option is defined in src/common/options/global.yaml.in, so the "global" configuration section should be the applicable one (I did try with "mon" and "osd" also, for good measure). However, setting this option, in Quincy, apparently has no effect: # ceph config set global osd_pool_default_crush_rule 1 # ceph osd pool create foo pool 'foo' created # ceph osd pool ls detail | grep foo # pool 9 'foo' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 264 flags hashpspool stripe_width 0 I am seeing this behaviour in 17.2.7. After an upgrade to Reef (18.2.4) it is gone, the option behaves as documented, and new pools are created with a crush_rule of 1: # ceph osd pool create bar pool 'bar' created # ceph osd pool ls detail | grep bar pool 10 'bar' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 302 flags hashpspool stripe_width 0 read_balance_score 4.00 However, the documentation at https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#con… asserts that osd_pool_default_crush_rule should already work in Quincy, and the Reef release notes at https://docs.ceph.com/en/latest/releases/reef/ don't mention a fix covering this. Am I doing something wrong? Is this a documentation bug, and the option can't work in Quincy? Was this "accidentally" fixed at some point in the Reef cycle? Thanks in advance for any insight you might be able to share. Cheers, Florian

3 9

ceph can list volumes from a pool but can not remove the volume
by bryansoong21＠gmail.com 26 Sep '24

26 Sep '24

We have a volume in our cluster: [root(a)ceph-1.lab-a ~]# rbd ls volume-ssd volume-8a30615b-1c91-4e44-8482-3c7d15026c28 [root(a)ceph-1.lab-a ~]# rbd rm volume-ssd/volume-8a30615b-1c91-4e44-8482-3c7d15026c28 Removing image: 0% complete...failed. rbd: error opening image volume-8a30615b-1c91-4e44-8482-3c7d15026c28: (2) No such file or directory rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. Any ideas on how can I remove the volume? Thanks

2 1

Cephalocon 2024 Agenda Announced – Sponsorship Opportunities Still Available!
by Dan van der Ster 26 Sep '24

26 Sep '24

Dear Ceph Community, We are excited to announce that the agenda for Cephalocon 2024 <https://events.linuxfoundation.org/cephalocon/program/schedule/> is now live! This year’s event, hosted at CERN, the birthplace of Ceph at scale, promises to be a unique opportunity to learn from experts, network with the community, and explore the latest innovations in software-defined storage. Whether you're new to Ceph or a long-time expert, there will be sessions and workshops tailored to all levels. We also want to highlight that sponsorship opportunities are still available. Sponsoring Cephalocon is an excellent way to increase your visibility in this influential community while supporting the ongoing growth and development of Ceph. Check out the full agenda on the Cephalocon website <https://events.linuxfoundation.org/cephalocon/> and get in touch with us at Sponsor | LF Events <https://events.linuxfoundation.org/cephalocon/sponsor/> to learn more about how you can become a sponsor. We look forward to seeing you in Geneva! Best regards, Neha, Josh, & Dan

2 1

cephfs +inotify = caps problem?
by Burkhard Linke 26 Sep '24

26 Sep '24

Hi, we are currently trying to debug and understand a problem with cephfs and inotify watchers. A user is running Visual Studio Code with a workspace on a cephfs mount. VSC uses inotify for monitoring files and directories in the workspace: root@cli:~# ./inotify-info ------------------------------------------------------------------------------ INotify Limits: max_queued_events 16,384 max_user_instances 128 max_user_watches 1,048,576 ------------------------------------------------------------------------------ Pid Uid App Watches Instances 3599940 1236 node 1,681 1 1 0 systemd 106 5 3600170 1236 node 54 1 874797 0 udevadm 17 1 3599118 0 systemd 7 3 3599707 1236 systemd 7 3 3599918 1236 node 6 1 2047 100 dbus-daemon 3 1 2054 0 sssd 2 1 2139 0 systemd-logind (deleted) 1 1 2446 0 agetty 1 1 3600001 1236 node 1 1 ------------------------------------------------------------------------------ Total inotify Watches: 1886 Total inotify Instances: 20 ------------------------------------------------------------------------------ root@cli:~# cat /sys/kernel/debug/ceph/XYZ.client354064780/caps | wc -l 1773083 root@cli:~# uname -a Linux cli 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux So roughly 1.700 watchers result in over 1.7 million caps. (some of the watchers might for files on different filesystems). I've also checked this on the MDS side, it also reports a very high number of caps for that client. Running tools like lsof on the host as root only reports very few open files (<50). So inotify seems to be responsible for the massive caps build up. Terminating VSC results in a sharp drop of the caps (just a few open files / directories left afterwards). Is this a known problem? Best regards, Burkhard Linke

2 1

[Ceph incident] PG stuck in peering.
by HARROUIN Loan (PRESTATAIRE CA-GIP) 26 Sep '24

26 Sep '24

Hello dear ceph community, We are facing a strange issue this weekend with a pg (13.6a) that is stuck in peering. Because of that we got lot of ops stuck of course. We are running a ceph in Pacific version 16.2.10, we have only SSD disk and are using erasure coding. cluster: id: f5c69b4a-89e0-4055-95f7-eddc6800d4fe health: HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg peering 256 slow ops, oldest one blocked for 5274 sec, osd.20 has slow ops services: mon: 3 daemons, quorum cos1-dal-ceph-mon-01,cos1-dal-ceph-mon-02,cos1-dal-ceph-mon-03 (age 17h) mgr: cos1-dal-ceph-mon-02(active, since 17h), standbys: cos1-dal-ceph-mon-03, cos1-dal-ceph-mon-01 osd: 647 osds: 646 up (since 27m), 643 in (since 2h) data: pools: 7 pools, 1921 pgs objects: 432.65M objects, 1.6 PiB usage: 2.4 PiB used, 2.0 PiB / 4.4 PiB avail pgs: 0.052% pgs not active 1916 active+clean 2 active+clean+scrubbing 2 active+clean+scrubbing+deep 1 peering The ‘ceph pg 13.6a query’ hung, so we must restart one of the osd that are part of this PG to temporary unhung the query (because during some seconds the pg isn’t peering yet). In that case, the query only retrieves the information about the shard that was hosted on the OSD that we restart. The result of the query is in attachment (shard 0). First when the issue occurs, we check the logs and restart all the osd linked to this PG. Sadly, it didn’t fix anything. We try to investigate the peering state to understand what was going on the primary OSD. We put the OSD in debug but at first glance anything seems strange (we are not use to deep dive that much into ceph). We find that CERN faced something similar a long time ago: https://indico.cern.ch/event/617118/contributions/2490930/attachments/14227… After reading it, we try to do the empty OSD method that they tried (diapo7). We identify that the shard0 seem in a weird state (and was primary) so it was our candidate. We wipe the OSD 11, 148 and 280 (one by one and waiting of course the peering to avoid data loss on other PGs). After that, the OSD.20 was now elected as the primary but still, the PG stay huge in peering and now all OPS are stuck on OSD.20. We are now in the dark. We plan to maybe deep dive deeper into the log of this new OSD.20, and see if we can plan to upgrade our ceph in order to have the most recent version. Any help or suggestion is welcome 😊

2 3

Backup strategies for rgw s3
by Adam Prycki 26 Sep '24

26 Sep '24

Hi, I'm currently working on a project which requires us to backup 2 separate s3 zones/realms and retain it for few months. Requirements were written by someone who doesn't know ceph rgw capabilities. We have to do incremental and full backups. Each type of backup has separate retention period. Is there a way to accomplish this with in a sensible way? My fist idea would be to create multisite replication to archive-zone. But I cannot really enforce data retention on archive zone. It would require us to overwrite lifecycle policies created by our users. As far as I know it's not possible to create zone level lifecycle policy. Users get their accounts are provisioned via openstack swift. Second idea would be to create custom backup script and copy all the buckets in the cluster to different s3 zone. Destination buckets could be all versioned to have desired retention. But this option feels very hackish and messy. Backing up 2 separate s3 zones to one could cause collision in bucket names. Prefixing bucket names with additional information is not safe because buckets have fixed name length. Prefixing object key name is also not ideal. Best regards Adam Prycki

6 6

Re: CephFS snaptrim bug?
by Linkriver Technology 26 Sep '24

26 Sep '24

Hello, We recently upgraded to Quincy (17.2.7) and I can see in the ceph logs many messages of the form: 1713256584.3135679 osd.28 (osd.28) 66398 : cluster 4 osd.28 found snap mapper error on pg 7.284 oid 7:214b503b:::100125de9b8.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256584.3136106 osd.28 (osd.28) 66399 : cluster 4 osd.28 found snap mapper error on pg 7.284 oid 7:214b4f95:::1001654390d.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256584.3136535 osd.28 (osd.28) 66400 : cluster 4 osd.28 found snap mapper error on pg 7.284 oid 7:214b4f3f:::1001549ed54.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256584.9496887 osd.29 (osd.29) 70001 : cluster 4 osd.29 found snap mapper error on pg 7.b4 oid 7:2d089bdc:::10016105140.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256590.9785151 osd.28 (osd.28) 66401 : cluster 4 osd.28 found snap mapper error on pg 7.284 oid 7:214b5179:::100128b85a0.00000cfe:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256598.6286905 osd.29 (osd.29) 70002 : cluster 4 osd.29 found snap mapper error on pg 7.17c oid 7:3e877f95:::100151d8670.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired ... A cursory reading of the code involved suggests that the scrubber in Quincy has acquired the capacity of detecting and removing the lost snapshots from Octopus, if I understand it correctly. Cheers, Linkriver Technology On Sat, 2022-06-25 at 19:36 +0000, Kári Bertilsson wrote: > Hello > > I am also having this issue after having > set osd_pg_max_concurrent_snap_trims = 0 previously to pause the > snaptrim. I upgraded to ceph 17.2.0. Have tried restarting, > repeering, deep-scrubbing all OSD's, so far nothing works. > > For one of the affected pools `cephfs_10k` I have tested removing > ALL data and it's still showing 26% usage. All snapshots have been > deleted and all pg's for the pool remain at SNAPTRIMQ_LEN = 0. All > pg's are active+clean. > > The pool still shows 589k object usage. When testing `rados get > object` on all the objects, it only works for 2.420 of them. The rest > seem to be in some kind of limbo and can not be read or deleted using > rados. > > # rados -p cephfs_10k listsnaps 10010539c22.00000000 > > > 10010539c22.00000000: > cloneid snaps size overlap > 288 288 30767656 [] > > # rados -p cephfs_10k get 10010539c22.00000000 10010539c22.00000000 > error getting cephfs_10k/10010539c22.00000000: (2) No such file or > directory > > # rados -p cephfs_10k rm 10010539c22.00000000 > error removing cephfs_10k>10010539c22.00000000: (2) No such file or > directory > > Is there some way to make the snap trimmer rediscover these objects > and remove them ? > > On Fri, Mar 18, 2022 at 2:21 PM Linkriver Technology > <technology(a)linkriver-capital.com> wrote: > > Hello, > > > > If I understand my issue correctly, it is in fact unrelated to > > CephFS itself, > > rather the problem happens at a lower level (in Ceph itself). IOW, > > it affects > > all kind of snapshots, not just CephFS ones. I believe my FS is > > healthy > > otherwise. In any case, here is the output of the command you > > asked: > > > > I ran it a few hours ago: > > > > "num_strays": 235, > > "num_strays_delayed": 38, > > "num_strays_enqueuing": 0, > > "strays_created": 5414436, > > "strays_enqueued": 5405983, > > "strays_reintegrated": 17892, > > "strays_migrated": 0, > > > > And just now: > > > > "num_strays": 186, > > "num_strays_delayed": 0, > > "num_strays_enqueuing": 0, > > "strays_created": 5540016, > > "strays_enqueued": 5531494, > > "strays_reintegrated": 18128, > > "strays_migrated": 0, > > > > > > Regards, > > > > LRT > > > > -----Original Message----- > > From: Arnaud M <arnaud.meauzoone(a)gmail.com> > > To: Linkriver Technology <technology(a)linkriver-capital.com> > > Cc: Dan van der Ster <dvanders(a)gmail.com>, Ceph Users > > <ceph-users(a)ceph.io> > > Subject: [ceph-users] Re: CephFS snaptrim bug? > > Date: Thu, 17 Mar 2022 21:48:18 +0100 > > > > Hello Linkriver > > > > I might have an issue close to your > > > > Can you tell us if your strays dirs are full ? > > > > What does this command output to you ? > > > > ceph tell mds.0 perf dump | grep strays > > > > Does the value change over time ? > > > > All the best > > > > Arnaud > > > > Le mer. 16 mars 2022 à 15:35, Linkriver Technology < > > technology(a)linkriver-capital.com> a écrit : > > > > > Hi, > > > > > > Has anyone figured whether those "lost" snaps are rediscoverable > > / > > > trimmable? > > > All pgs in the cluster have been deep scrubbed since my previous > > email and > > > I'm > > > not seeing any of that wasted space being recovered. > > > > > > Regards, > > > > > > LRT > > > > > > -----Original Message----- > > > From: Dan van der Ster <dvanders(a)gmail.com> > > > To: technology(a)linkriver-capital.com > > > Cc: Ceph Users <ceph-users(a)ceph.io>, Neha Ojha <nojha(a)redhat.com> > > > Subject: Re: [ceph-users] CephFS snaptrim bug? > > > Date: Thu, 24 Feb 2022 09:48:04 +0100 > > > > > > See https://tracker.ceph.com/issues/54396 > > > > > > I don't know how to tell the osds to rediscover those trimmed > > snaps. > > > Neha does that possible? > > > > > > Cheers, Dan > > > > > > On Thu, Feb 24, 2022 at 9:27 AM Dan van der Ster > > <dvanders(a)gmail.com> > > > wrote: > > > > > > > > Hi, > > > > > > > > I had a look at the code -- looks like there's a flaw in the > > logic: > > > > the snaptrim queue is cleared if > > osd_pg_max_concurrent_snap_trims = 0. > > > > > > > > I'll open a tracker and send a PR to restrict > > > > osd_pg_max_concurrent_snap_trims to >= 1. > > > > > > > > Cheers, Dan > > > > > > > > On Wed, Feb 23, 2022 at 9:44 PM Linkriver Technology > > > > <technology(a)linkriver-capital.com> wrote: > > > > > > > > > > Hello, > > > > > > > > > > I have upgraded our Ceph cluster from Nautilus to Octopus > > (15.2.15) > > > over the > > > > > weekend. The upgrade went well as far as I can tell. > > > > > > > > > > Earlier today, noticing that our CephFS data pool was > > approaching > > > capacity, I > > > > > removed some old CephFS snapshots (taken weekly at the root > > of the > > > filesystem), > > > > > keeping only the most recent one (created today, 2022-02-21). > > As > > > expected, a > > > > > good fraction of the PGs transitioned from active+clean to > > > active+clean+snaptrim > > > > > or active+clean+snaptrim_wait. In previous occasions when I > > removed a > > > snapshot > > > > > it took a few days for snaptrimming to complete. This would > > happen > > > without > > > > > noticeably impacting other workloads, and would also free up > > an > > > appreciable > > > > > amount of disk space. > > > > > > > > > > This time around, after a few hours of snaptrimming, users > > complained > > > of high IO > > > > > latency, and indeed Ceph reported "slow ops" on a number of > > OSDs and > > > on the > > > > > active MDS. I attributed this to the snaptrimming and decided > > to > > > reduce it by > > > > > initially setting osd_pg_max_concurrent_snap_trims to 1, > > which didn't > > > seem to > > > > > help much, so I then set it to 0, which had the surprising > > effect of > > > > > transitioning all PGs back to active+clean (is this > > intended?). I also > > > restarted > > > > > the MDS which seemed to be struggling. IO latency went back > > to normal > > > > > immediately. > > > > > > > > > > Outside of users' working hours, I decided to resume > > snaptrimming by > > > setting > > > > > osd_pg_max_concurrent_snap_trims back to 1. Much to my > > surprise, > > > nothing > > > > > happened. All PGs remained (and still remain at time of > > writing) in > > > the state > > > > > active+clean, even after restarting some of them. This > > definitely seems > > > > > abnormal, as I mentioned earlier, snaptrimming this FS > > previously > > > would take in > > > > > the order of multiple days. Moreover, if snaptrim were truly > > complete, > > > I would > > > > > expect pool usage to have dropped by appreciable amounts (at > > least a > > > dozen > > > > > terabytes), but that doesn't seem to be the case. > > > > > > > > > > A du on the CephFS root gives: > > > > > > > > > > # du -sh /mnt/pve/cephfs > > > > > 31T /mnt/pve/cephfs > > > > > > > > > > But: > > > > > > > > > > # ceph df > > > > > <snip> > > > > > --- POOLS --- > > > > > POOL ID PGS STORED OBJECTS USED > > %USED MAX > > > AVAIL > > > > > cephfs_data 7 512 43 TiB 190.83M 147 TiB > > 93.22 > > > 3.6 TiB > > > > > cephfs_metadata 8 32 89 GiB 694.60k 266 GiB > > 1.32 > > > 6.4 TiB > > > > > <snip> > > > > > > > > > > ceph pg dump reports a SNAPTRIMQ_LEN of 0 on all PGs. > > > > > > > > > > Did CephFS just leak a massive 12 TiB worth of objects...? It > > seems to > > > me that > > > > > the snaptrim operation did not complete at all. > > > > > > > > > > Perhaps relatedly: > > > > > > > > > > # ceph daemon mds.choi dump snaps > > > > > { > > > > > "last_created": 93, > > > > > "last_destroyed": 94, > > > > > "snaps": [ > > > > > { > > > > > "snapid": 93, > > > > > "ino": 1, > > > > > "stamp": "2022-02-21T00:00:01.245459+0800", > > > > > "name": "2022-02-21" > > > > > } > > > > > ] > > > > > } > > > > > > > > > > How can last_destroyed > last_created? The last snapshot to > > have been > > > taken on > > > > > this FS is indeed #93, and the removed snapshots were all > > created on > > > previous > > > > > weeks. > > > > > > > > > > Could someone shed some light please? Assuming that snaptrim > > didn't > > > run to > > > > > completion, how can I manually delete objects from now- > > removed > > > snapshots? I > > > > > believe this is what the Ceph documentation calls a > > "backwards scrub" > > > - but I > > > > > didn't find anything in the Ceph suite that can run such a > > scrub. This > > > pool is > > > > > filling up fast, I'll throw in some more OSDs for the moment > > to buy > > > some time, > > > > > but I certainly would appreciate your help! > > > > > > > > > > Happy to attach any logs or info you deem necessary. > > > > > > > > > > Regards, > > > > > > > > > > LRT > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy
by Alex Hussein-Kershaw (HE/HIM) 25 Sep '24

25 Sep '24

Thanks 🙂 I've raised: Bug #68102: rgw: "radosgw-admin topic list" may contain duplicated data and redundant nesting - rgw - Ceph<https://tracker.ceph.com/issues/68102?next_issue_id=984> Enhancement #68104: rgw: Add a "disable replication" flag to bucket notification configuration - rgw - Ceph<https://tracker.ceph.com/issues/68104> Also re-including the mailing list as it was dropped. ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com> Sent: Tuesday, September 17, 2024 10:36 AM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy ok, got it. not even sure we support attribute filtering for these types of events... i think you have to go with v1 for now. would be great if you also submit a tracker for the v1 topic list On Tue, Sep 17, 2024 at 12:24 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Indeed, I am using the s3:Replication:Create event (which I think is equivalent to s3:ObjectSynced:Create). But this does not solve the problem of the event being added to both topics on the site that received the replication. ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Tuesday, September 17, 2024 10:21 AM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy if you want to get a notification when an object is synced, you should use a different type of notification. the fact that an object is uploaded to siteB does not mean it is immediately synced to siteA. would recommend using s3:ObjectSynced:Create event type in this case. you will get this event only when an object is synced. On Tue, Sep 17, 2024 at 12:05 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Using the sites sounds sensible, and probably better than using my suggestion of application deployment name as they are not subject to changing (and actually we do genuinely use siteA and siteB exclusively as the zone names,). I want the siteA application to be notified of replication changes synced across from PUTs made on siteB, so I need the flip of what you are suggesting, i.e. create notification on siteA with filter "x-amz-metadata-site" == "siteB". That's fine I think (and doesn't really complicate the scenario), but requires the application to now be aware of the zone it is, which isn't information it has without some plumbing (and doesn't seem like something an application would typically be aware of?). That's on top of the plumbing to add the metadata header into every place where I do an S3 PUT. I still think I prefer using v1 and attempting to contribute an enhancement here to set a flag to disable multisite on a per notification basis, given the above; assuming you agree that this is a sensible enhancement. ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Monday, September 16, 2024 6:15 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy regarding the filter. i don't really follow. on siteA create a notification (with id "notifA") with filter "x-amz-metadata-site" == "siteA" that point to topicA (reachable in site A) and on siteB create a notification (with id "notifB") with filter "x-amz-metadata-site" == "siteB" that point to topicB (reachable in siteB) assuming we are in v2 and both notifications exist on the bucket. when app is uploading an object to a bucket in siteA, it should set "siteA" in the "x-amz-metadata-site" attribute. notifA will match the filter and send the notification to topicA, while it won't match the filter of notifB. when uploading an object to that bucket in siteB the opposite will happen. On Mon, Sep 16, 2024 at 7:14 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: This is really an application level problem, but it's not trivial for me to determine the name of the remote site. So while I could add a metadata header and include the site name, then use "x-amz-metadata-site" == remote_site_name as my filter, it would be much more practical to say "x-amz-metadata-site" != local_site_name, if that were possible. I suspect that makes it firmly not your problem 🙂 but it does push me towards v1. That's what I assumed re the notifications. However, I did delete all the notifications and topics before disabling the feature and did still end up with the nesting shown below in the "ragosgw-admin topic list" command output. I also reproduced that installing a fresh Cluster at V19.1.1, immediately disabling notifications_v2 and then creating topics. although without the duplication this time. $ radosgw-admin topic list { "topics": [ { "topics": [ { "owner": "zone.user", "name": "ahk2", "dest": { "push_endpoint": "https://1.2.3.4:1234/", "push_endpoint_args": "verify-ssl=false", "push_endpoint_topic": "ahk2", "stored_secret": false, "persistent": true, "persistent_queue": ":ahk2", "time_to_live": "None", "max_retries": "None", "retry_sleep_duration": "10" }, "arn": "arn:aws:sns:geored_zg::ahk2", "opaqueData": "", "policy": "" } ] } ] } So maybe a bug in Squid? ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Monday, September 16, 2024 4:42 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Cc: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy inline On Mon, Sep 16, 2024 at 5:13 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Yes - that's correct. Thanks for the suggestions. I think the metadata suggestion probably does work, however it doesn't come easy for me as it seems not possible to do a negative match filter; what I would really like to do is setup a filter for objects where "x-amz-metadata-site" != "local-site-name". Seems that's an AWS S3 limitation. why do you need a negative match? the filter determines what is being notified not what to filter out. Likely leaning to disabling notifications_v2 for now and submitting a tracker for adding the flag as you suggested. I tried that but got some internal error from the tracker system, so will try again later. I'm keen to have a go at implementing as it will give us a path forward when v1 is removed. I'm curious if it's supported to go from notifications_v2 to v1? while we automatically migrate topics and notifications from v1 to v2, we do not do the opposite. if you had something on v2 and want to disable, best option is to delete all topics and notifications, disable v2 and define them again. Suspect it might not be and the bottom is an artifact of that. I tested disabling notifications_v2 and seem to have confused the "radosgw-admin topic list" command, which now returns some heavily nested JSON which looks a bit odd. Functionally might still be OK. I'm on the 19.1.1 RC, so not a stable release. I've replaced the actual topic data with "..." below, but it appears I am listing 4 copies of the same 2 topics. e.g.: $ radosgw-admin topic list { "topics": [ { "topics": [ ..., ... ], }, "topics": [ { "topics": [ ..., ... ], }, "topics": [ { "topics": [ ..., ... ], }, "topics": [ { "topics": [ ..., ... ], } ] } Thanks, Alex ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Monday, September 16, 2024 1:25 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Cc: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy just to see that i got it right. you are asking to disable the topic and notification replication. as you want to send to different topics based on the zone that got the update to the bucket? * one option is to disable "notifications_v2" on the zonegroup. but, this is probably not a good idea - there are other benefits to v2, and the v1 format will be deprecated eventually. * if you can control the client apps on each site, maybe you can add a metadata attribute (e.g. z-amz-...) to the requests (or maybe you already have one?) and filter the notifications based on that attribute? * please consider submitting a tracker for adding a "replication" flag to the notification configuration. so we can disable the replication (without disabling v2 notifications) on a granular level On Mon, Sep 16, 2024 at 1:18 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Following up on this, I've run into another issue during my prototyping. I have two Ceph Clusters, with a zone each, sharing a zonegroup and realm. I have a local application to each, that needs informed of replication changes. So I've created a topic and notification per site. Given the notification_v2 feature, these are replicated and both sites are aware of both topics and both notifications. I'm testing by writing to the bucket on the "siteA" Cluster, and I can see Replication:Create events pushed to both topics on the "siteB" Cluster. However, therein lies my problem; I don't want the Ceph Cluster on siteB to attempt to notify siteA of the replication (in-fact it can't route to it in the network at all, so it fails, and a queue builds up, which will eventually block subsequent writes if not dealt with). I think this is new in the notification_v2 feature, as a side-effect of the notifications being multisite. I had a look into filtering rules to see if there was a way to say filter by events in a specific zone, but didn't find anything. Is there a solution to this? Any suggestions very welcome 🙂 Kind regards, Alex ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Tuesday, September 3, 2024 2:56 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Cc: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy responded inline On Tue, Sep 3, 2024 at 3:05 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Hi Yuval, Thanks for the response. I did managed to disable the feature, however I hope you can understand my hesitancy to design our move away from pubsub onto a deprecated replacement (i.e. "notifications v1"). > agree. notification v2 provides better observability, e.g. list of buckets that use topic etc. The difference between this and the other operations that require forwarding to the master site, for me, is that I don't rely on any of those operations for day-to-day function. For example, S3 user creation I do at deployment time and never again; so if I lose the ability to do that in the event of a site failure that's OK. Perhaps I just need to implement a process to ensure the remaining zone is quickly promoted on a site failure. > this is probably the correct approach in multisite A further concern of the new feature... My intention is to use the bucket notification feature to send events generated by the multisite sync to my application server. If the topic is now multisite, will all kicks be duplicated? For example consider: SiteA receives an S3 write for object SiteA writes to the event queue and asynchronously delivers the event SiteB syncs object from siteA we do not send regular notifications upon sync. note that we have other type of notifications: s3:ObjectSynced:*/s3:Replication:* that are sent only when an object is synced SiteB writes to the event queue and asynchronously delivers the event Really I only want the second event in this example, which is what I get with the pubsub module, and I think is what I will get with "notifications_v1". Best Wishes, Alex ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Tuesday, September 3, 2024 12:15 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Cc: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> Subject: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy Hi Alex, It should be possible to disable the v2 feature through an admin command. radosgw-admin onegroup modify --disable-feature=notification_v2 Also note that when creating a new squid cluster, v2 is enabled by default. But, when upgrading an existing cluster, you need to call: radosgw-admin onegroup modify --enable-feature=notification_v2 to enable v2. However, we do encourage moving to v2, as there are many benefits to the new data model (besides site syncing). Regarding the problem you encountered when the "master" site is down, how is this different from other operations that are forwarded to "master"? Yuval On Tue, Sep 3, 2024 at 11:31 AM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Hi Folks, I see in the pending release notes for Squid a description of the "notification_v2" feature. ceph/PendingReleaseNotes at main · ceph/ceph (github.com<http://github.com/>)<https://github.com/ceph/ceph/blob/main/PendingReleaseNotes#L43> I have some concerns about the multisite nature of this feature, I am using Ceph multisite to provide geographic redundancy for my application data. That seems to be a mainline use case. My application is designed to cope if an entire zone fails. My application currently relies on pubsub (and soon to be replaced by bucket notifications) to provide service. More specifically I rely on the ability to semi-regularly automatically reconfigure topics to adjust the HTTP endpoint. I believe with the "notification_v2" feature all adjustments of the topic are routed through the master for metadata site (much like creating an S3 user). Fundamentally I think this breaks the combination of Ceph multisite, notifications_v2 and reconfiguring a topic; you cannot reconfigure a topic unless both sites are up. I tested this, by setting up a deployment, shutting down a site and trying to reconfigure a topic. I get a 500 HTTP response to my createTopic, and can see this in the RGW logs: debug 2024-09-03T08:24:39.997+0000 7f53a3372640 20 ERROR: curl error: Couldn't connect to server req_data->error_buf=Failed to connect to 10.225.20.19 port 7480: No route to host debug 2024-09-03T08:24:39.997+0000 7f5329a7f640 4 req 2968544598970267460 0.686004043s sns:pubsub_topic_create CreateTopic forward_request_to_master returned ret = -2200 Have the implications of this been considered? For now I may be able to disable the notifications_v2 feature, however in the release note the predecessor is deprecated in Squid. Hopefully I'm missing something obvious. Kind regards, Alex _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 2

Quincy: osd_pool_default_crush_rule being ignored?
by Florian Haas 24 Sep '24

24 Sep '24

Hello everyone, my cluster has two CRUSH rules: the default replicated_rule (rule_id 0), and another rule named rack-aware (rule_id 1). Now, if I'm not misreading the config reference, I should be able to define that all future-created pools use the rack-aware rule, by setting osd_pool_default_crush_rule to 1. I've verified that this option is defined in src/common/options/global.yaml.in, so the "global" configuration section should be the applicable one (I did try with "mon" and "osd" also, for good measure). However, setting this option, in Quincy, apparently has no effect: # ceph config set global osd_pool_default_crush_rule 1 # ceph osd pool create foo pool 'foo' created # ceph osd pool ls detail | grep foo # pool 9 'foo' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 264 flags hashpspool stripe_width 0 I am seeing this behaviour in 17.2.7. After an upgrade to Reef (18.2.4) it is gone, the option behaves as documented, and new pools are created with a crush_rule of 1: # ceph osd pool create bar pool 'bar' created # ceph osd pool ls detail | grep bar pool 10 'bar' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 302 flags hashpspool stripe_width 0 read_balance_score 4.00 However, the documentation at https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#con… asserts that osd_pool_default_crush_rule should already work in Quincy, and the Reef release notes at https://docs.ceph.com/en/latest/releases/reef/ don't mention a fix covering this. Am I doing something wrong? Is this a documentation bug, and the option can't work in Quincy? Was this "accidentally" fixed at some point in the Reef cycle? Thanks in advance for any insight you might be able to share. Cheers, Florian

1 0

Multisite sync: is metadata transferred in plain text?
by maryzhang0920＠gmail.com 24 Sep '24

24 Sep '24

Hi, We have a multisite Ceph configuration, with http (not https) sync endpoints. Are all sync traffic in plain text? We have concerns about metadata. For example, when syncing a newly created user and its access key and secret key from the Master zone to a secondary zone, is this traffic in plain text? If so, what are options to encrypt it? Thank you, Mary

4 4

High usage (DATA column) on dedicated for OMAP only OSDs
by Александр Руденко 24 Sep '24

24 Sep '24

Hi! We have S3/CephFS cluster with dedicated SSDs for bucket indexes, CephFS metadata and for a few small rgw metadata pools. We have special crush rules for these pools. All of our pools (pool naming is old because it's very old cluster) that are placed on SSDs: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 32 1.3 KiB 5 108 KiB 0 690 GiB .rgw.control 2 32 0 B 8 0 B 0 690 GiB .rgw 3 32 2.0 MiB 8.02k 328 MiB 0.02 690 GiB .rgw.gc 4 64 1.6 GiB 64 4.9 GiB 0.24 690 GiB .users.uid 5 32 1.8 MiB 2.63k 67 MiB 0 690 GiB .users 6 32 275 KiB 7.02k 379 MiB 0.02 690 GiB .usage 7 32 30 MiB 64 30 MiB 0 690 GiB .intent-log 8 64 0 B 0 0 B 0 690 GiB .log 9 64 6.6 GiB 34.29k 21 GiB 1.01 690 GiB .rgw.buckets.index 12 4096 1.9 TiB 248.96k 1.9 TiB 48.02 690 GiB .users.email 13 32 64 KiB 1.61k 72 MiB 0 690 GiB fs1_meta 14 64 467 MiB 35.39k 829 MiB 0.04 690 GiB But on all our SSDs we can see high DATA usage, for example: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 57 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 96 GiB 1.4 GiB 40 GiB 90.96 1.37 138 up osd.57 19 ssd 0.00005 1.00000 447 GiB 402 GiB 309 GiB 92 GiB 1.4 GiB 44 GiB 90.09 1.35 145 up osd.19 10 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 97 GiB 1.1 GiB 40 GiB 91.01 1.37 137 up osd.10 12 ssd 0.00005 1.00000 447 GiB 401 GiB 309 GiB 91 GiB 1.3 GiB 46 GiB 89.77 1.35 134 up osd.12 All SSDs are bluestore. Ceph 16.3.11 and many SSDs were deployed on 16.x. As I understand, the majority of these pools contain only "OMAP" which is stored in RocksDB and tracked in stats as *OMAP*. And I don't understand why we have such high *DATA* and we can see that DATA usage grows by 1-2 GB/day for the last 90 days (we don't have more monitoring data)! I have checked crush rules and all SSDs for PGs from other pools and can't see any other PGs from "pure" data pools. Rules are correct. How can I see what is stored in bluestore DATA on some OSD? I can export full RocksDB, but not DATA stored in bluestore.. What kind of DATA can be stored in these OSDs with these pools? We did a lot of offline compactions, 2-5 times for some SSDs in the last 90 days. But i'm not sure if it's related

4 14

Ceph Dashboard TLS
by matthew＠peregrineit.net 24 Sep '24

24 Sep '24

Hi All, I'm running an (experimental) 3-Node Ceph Reef (v18.2.4) Cluster. Each of the 3 nodes runs (amongst other services) the Ceph Dashboard - for fail-over purposes. I can connect to the Ceph Dashboard when not using TLS (ie ceph config set mgr mgr/dashboard/ssl false). I've got a private PKI (Step-CA) and I'd like to use its TLS Certificates for the Ceph Dashboard. The private PKI's CA Cert has been added to my browser, and internal websites, etc, using the internal PKI's Certificates work as expected. I'd like each of the three Ceph Nodes to use their own TLS Certificate. The 3 node's hostnames are ceph01.example.com, ceph02.example.com, and ceph03.example.com. The 3 node's mgr instances are (I think; & this could be where I'm going wrong) are mgr.ceph01, mgr.ceph02, and mgr.ceph03. I've created 3 TLS Certificates with names of cephdash-ceph01.example.com.crt, etc, CNs of cephdash-ceph01.example.com, etc, and SANs of ceph01.example.com, etc I've followed the documentation here: https://docs.ceph.com/en/reef/mgr/dashboard/#ssl-tls-support I have stopped and restarted the Ceph Dashboard service. Commands used (on ceph01.example.com) ceph dashboard set-ssl-certificate ceph01 -i cephdash-ceph01.example.com.crt ceph dashboard set-ssl-certificate-key ceph01 -i cephdash-ceph01.example.com.key ceph mgr module disable dashboard ceph mgr module enable dashboard It's not working (I get an Unable to connect page), and I can't work out why - but I suspect it'll be something bleedingly obvious. :-) My Qs: Can I use EC Certs or must I use RSA Certs? I've tryied both and neither work. Can anyone spot what I'm doing wrong? Thanks in advance Cheers Dulux-Oz

3 5

All new osds are made orphans
by Phil 23 Sep '24

23 Sep '24

Hi, on a healthy cluster, every osd creation made orphans osd. # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 26.38051 root default -7 3.63869 host A 1 hdd 3.63869 osd.1 up 1.00000 1.00000 -11 1.81940 host F 2 hdd 1.81940 osd.2 up 1.00000 1.00000 -5 7.27737 host J 3 hdd 3.63869 osd.3 up 1.00000 1.00000 5 hdd 3.63869 osd.5 up 1.00000 1.00000 -3 7.27737 host K 0 hdd 3.63869 osd.0 up 1.00000 1.00000 8 hdd 3.63869 osd.8 up 1.00000 1.00000 -9 6.36768 host S 6 hdd 3.63869 osd.6 up 1.00000 1.00000 7 hdd 2.72899 osd.7 up 1.00000 1.00000 4 0 osd.4 down 0 1.00000 9 0 osd.9 down 0 1.00000 neither 4 nor 9 osd appear in the crushmap (decompiled) Destroying and recreating osd recreate orphan osd. Any hint ?Thanks for all Best regardsPhilippe

2 3

CLT Meeting Notes 23 September 2024
by Casey Bodley 23 Sep '24

23 Sep '24

notes from today's meeting of the Ceph Leadership Team Squid 19.2.0 release ==================== Laura successfully built/signed prerelease packages LRC upgrade was the last step, but hit iscsi failures again * Ilya to create a tracker issue and tag Dan/Xiubo for details * Ilya to reschedule rbd/iscsi suite against 19.2.0 to determine whether this should block the release * release note needed * in infrastructure call, discuss future plan for LRC (probably nvmeof) Licensing discussion ==================== parent tracker https://tracker.ceph.com/issues/68082 since last week, trackers added for use of GPL in ppc https://tracker.ceph.com/issues/68086 and dashboard https://tracker.ceph.com/issues/68088 Ceph Executive Council ====================== regarding https://docs.ceph.com/en/latest/governance, the Ceph Steering Committee is overdue to elect a replacement member for the Ceph Executive Council we'll dedicate more time in next week's CLT to continue discussing the proposals and voting process Cephalocon ========== one more week until the deadline for early registration. update ceph.io with details about the Dev Summit

1 0

Overlapping Roots - How to Fix?
by Dave Hall 23 Sep '24

23 Sep '24

Helllo, I've reviewed some recent posts in this list and also searched Google for info about autoscale and overlapping roots. In what I have found I do not see anything that I can understand regarding how to fix the issue - probably because I don't deal with Crush on a regular basis. My particulars: I have an up-to-date Reef cluster in production with some EC pools and some replicated pools. My error messages look like: pool default.rgw.buckets.data won't scale due to overlapping roots: {-1, -2} From what I read and looking at 'ceph osd crush rule dump', it looks like the 8 replicated pools have "op": "take", "item": -1, "item_name": "default" whereas the 2 EC pools have "op": "take", "item": -2, "item_name": "default~hdd" To be sure, all of my OSDs are identical - HDD with SSD WAL/DB. Please advise on how to fix this. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

5 12

Radosgw bucket check fix doesn't do anything
by Reid Guyett 23 Sep '24

23 Sep '24

Hello, I recently moved a bucket from 1 cluster to another cluster using rclone. I noticed that the source bucket had around 35k objects and the destination bucket only had around 18k objects after the sync was completed. Source bucket stats showed: > radosgw-admin bucket stats --bucket mimir-prod | jq .usage > { > "rgw.main": { > "size": 4321515978174, > "size_actual": 4321552605184, > "size_utilized": 4321515978174, > "size_kb": 4220230448, > "size_kb_actual": 4220266216, > "size_kb_utilized": 4220230448, > "num_objects": 35470 > }, > "rgw.multimeta": { > "size": 0, > "size_actual": 0, > "size_utilized": 66609, > "size_kb": 0, > "size_kb_actual": 0, > "size_kb_utilized": 66, > "num_objects": 2467 > } > } Destination bucket stats showed: > radosgw-admin bucket stats --bucket mimir-prod | jq .usage > { > "rgw.main": { > "size": 4068176326491, > "size_actual": 4068212576256, > "size_utilized": 4068176326491, > "size_kb": 3972828444, > "size_kb_actual": 3972863844, > "size_kb_utilized": 3972828444, > "num_objects": 18525 > }, > "rgw.multimeta": { > "size": 0, > "size_actual": 0, > "size_utilized": 108, > "size_kb": 0, > "size_kb_actual": 0, > "size_kb_utilized": 1, > "num_objects": 4 > } > } > When I checked the source bucket using aws cli tool it showed around 18k objects. The bucket was actively being used so the 18k is slightly different. > aws --profile mimir-prod --endpoint-url https://my.objectstorage.domain > s3api list-objects --bucket mimir-prod > mimir_objs > cat mimir_objs | grep -c "Key" > 18090 I did a check on the source bucket and it showed a lot of invalid multipart objects. > radosgw-admin bucket check --bucket mimir-prod | head > { > "invalid_multipart_entries": [ > > "_multipart_network/01H9CFRA45MJWBHQRCHRR4JHV4/index.sJRTCoqiZvlge2cjz6gLU7DwuLI468zo.2", > > "_multipart_network/01HMCCRMTC5F4BFCZ56BKHTMWQ/index.6ypGbeMr6Jg3y7xAL8yrLL-v4sbFzjSA.3", > > "_multipart_network/01HMFKR56RRZNX9VT9B4F49MMD/chunks/000001.JIC7fFA_q96nal1yGXsVSPCY8EMe5AU8.2", > > "_multipart_network/01HMFKSND2E5BWF6QVTX8SDRRQ/index.57aSNeXn3j70H4EHfbNCD2RpoOp-P1Bv.2", > > "_multipart_network/01HMFKTDNA3FVSWW7N8KYY2C7N/chunks/000001.2~kRjRbLWWDf1e40P40LUzdU3f_x2P46Q.2", > > "_multipart_network/01HMFTMA8J1DEXYHKMVCXCC0GM/chunks/000001.GVajdCja0gHOLlgyFanF72A4B6ZqUpu5.2", > > "_multipart_network/01HMFTMA8J1DEXYHKMVCXCC0GM/chunks/000001.GYaouEePvEdbQosCb5jLFCAHrSm9VoDh.2", > > "_multipart_network/01HMFTMA8J1DEXYHKMVCXCC0GM/chunks/000001.r4HkP-JK-rBAWDoXBXKJJYEAjk39AswW.1", > ... > So I tried to run `radosgw-admin bucket check --check-objects --bucket mimir-prod --fix` and it showed that it was cleaning things with thousands of lines like > 2024-09-17T12:19:42.212+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.2 > 2024-09-17T12:19:42.212+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.3 > 2024-09-17T12:19:42.212+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.4 > 2024-09-17T12:19:42.212+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.5 > 2024-09-17T12:19:42.213+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.6 but the end result shows nothing has changed. > "check_result": { > "existing_header": { > "usage": { > "rgw.main": { > "size": 4281119287051, > "size_actual": 4281159110656, > "size_utilized": 4281119287051, > "size_kb": 4180780554, > "size_kb_actual": 4180819444, > "size_kb_utilized": 4180780554, > "num_objects": 36429 > }, > "rgw.multimeta": { > "size": 0, > "size_actual": 0, > "size_utilized": 66636, > "size_kb": 0, > "size_kb_actual": 0, > "size_kb_utilized": 66, > "num_objects": 2468 > } > } > }, > "calculated_header": { > "usage": { > "rgw.main": { > "size": 4281119287051, > "size_actual": 4281159110656, > "size_utilized": 4281119287051, > "size_kb": 4180780554, > "size_kb_actual": 4180819444, > "size_kb_utilized": 4180780554, > "num_objects": 36429 > }, > "rgw.multimeta": { > "size": 0, > "size_actual": 0, > "size_utilized": 66636, > "size_kb": 0, > "size_kb_actual": 0, > "size_kb_utilized": 66, > "num_objects": 2468 > } > } > } > } > Does this command do anything? Is it the wrong command for this issue? How does one go about fixing buckets in this state? Thanks! Reid

2 8

Help with cephadm bootstrap and ssh private key location
by Kozakis, Anestis 23 Sep '24

23 Sep '24

Hi All, Very new to Ceph and hoping someone can help me out. We are implementing Ceph in our team's environment, and I have been able to manually set up a test cluster using cephadm bootstrap and answering all the prompts. What we want to do is to automate the setup and maintenance of the production Ceph cluster using SaltStack. This involves pre-configuring the hosts using Salt High States, including installation of packages, firewall configuration, etc etc, and the distribution of ssh keys (root user) and configuring openssh on each host. Whilst I can see the public ssh key in authotrized_keys on all the nodes, I cannot seem to find to find the private key on the initial admin host (the first node). It is not in /root/.ssh and the only file there is the authorized_keys file. Where does cephadm bootstrap store the private key? Also, if we already have ssh keys set up on all the hosts and we use the -skip-ssh option in the cephadm bootstrap command, will using a spec file with the --apply-spec option work to add additional hosts and all the services we need. Thanks in advance for any help/advice on this. Anestis Kozakis Systems Administrator - Multi-Level Security Solutions P: + 61 2 6122 0205 M: +61 4 88 376 339 anestis.kozakis(a)raytheon.com.au<mailto:anestis.kozakis@raytheon.com.au> Raytheon Australia Cybersecurity and Information Assurance 4 Brindabella Cct Brindabella Business Park Canberra Airport, ACT 2609 www.raytheonaustralia.com.au<http://www.raytheonaustralia.com.au/> LinkedIn<https://au.linkedin.com/company/raytheon-australia> | Twitter<https://twitter.com/RaytheonAU> | Facebook<https://www.facebook.com/RaytheonAustralia> | Instagram<https://www.instagram.com/raytheonaustralia/>

2 3

filter @ rados_object_list
by Daniel Biecker 23 Sep '24

23 Sep '24

Hi, I want to search for objects using the rados_object_list function. How do I set the filter? I always get error -22. What is the syntax for the filter value.

1 0

Directory missing in cephfs
by Pavel Kaygorodov 21 Sep '24

21 Sep '24

Hi, All! I'm running a small ceph cluster (16.2.3) with cephfs an number of replicas = 3. Recently, after one of the nodes failure, I have performed a number of copy/move (not a delete ones) operations, while on of the pgs was inactive. There was no hangs, no errors reported, but, some directories was disappeared in the directory, where it was moved. Now, all of pgs are active+clean (some of them are remapped), but missing directories are not returned. I have started journal recovery with the command: cephfs-journal-tool --rank files:all event recover_dentries summary it still working (more then 20 hours till now) without any visible results. I have made snapshots of data and metadata pools, before starting journal recovery, also made a backup of the journal. What is my next steps if the journal recovery will bring no results? Is there a way to manually find a specific directory record directly in the pool and restore it? I can write a program to fetch and parse cephs objects, but, may be, there is a specific tool for that? Thanks in advance, Pavel.

1 0

Multisite sync: is metadata transferred in plain text?
by Mary Zhang 20 Sep '24

20 Sep '24

Hi, We have a multisite Ceph configuration, with http (not https) sync endpoints. Are all sync traffic in plain text? We have concerns about metadata. For example, when syncing a newly created user and its access key and secret key from Master zone to a secondary zone, are the keys also in plain text? If so, what are options to encrypt it? Thank you, Mary

1 0

Cephfs mirroring --
by Devender Singh 20 Sep '24

20 Sep '24

Hello all I tried configuring cephfs mirroring between cluster as per doc https://docs.ceph.com/en/reef/dev/cephfs-mirroring/ using bootstrap. But My replication is not working. Using two mirror daemons on each cluster. [Enabled module/mirroring/snap mirroring/aded directory/allow_new_snaps] ceph fs snapshot mirror peer_bootstrap create backup_fs client.mirror site-remote # on destination cluster ceph fs snapshot mirror peer_bootstrap import sourcefs <token> # token from above. I see the details on source cluster using “ceph fs snapshot mirror daemon status | jq” but it’s not copying nay data. I am using /repo directory under source filesystem with snapshot and schedule. “ceph fs snapshot mirror add $SOURCEFS /repo" When snap schedule starts, it showing one or two dir on destination but doesn’t grow more than few kb or mb. would be agree help if someone has done that. Regards Dev

1 0

scrubing
by Albert Shih 20 Sep '24

20 Sep '24

Hi everyone. Few time ago I add a new node to my cluster with some HDD. Currently the cluster does the remapping and backfill. I now got a warning about HEALTH_WARN 1 pgs not deep-scrubbed in time So I check and find something a litle weird. root@cthulhu1:~# ceph config get osd osd_deep_scrub_interval 604800.000000 so that's one week. If I check the LAST DEEP SCRUB TIMESTAMP I got root@cthulhu1:~# ceph pg dump pgs | awk '{print $1" "$24}' | grep -v 2024-09-[1-2][0-9] dumped pgs PG_STAT DEEP_SCRUB_STAMP 4.63 2024-09-09T19:00:57.739975+0000 4.5a 2024-09-09T08:17:15.124704+0000 4.56 2024-09-09T21:51:07.478651+0000 4.51 2024-09-08T00:10:30.552347+0000 4.4c 2024-09-09T10:35:02.048445+0000 4.4b 2024-09-09T19:53:19.839341+0000 4.14 2024-09-08T18:36:12.025455+0000 4.c 2024-09-09T16:00:59.047968+0000 4.4 2024-09-09T00:19:07.554153+0000 4.8 2024-09-09T22:19:15.280310+0000 4.25 2024-09-09T06:45:37.258306+0000 4.30 2024-09-09T16:56:21.472410+0000 4.82 2024-09-09T21:14:09.802303+0000 4.c9 2024-09-08T17:10:56.133363+0000 4.f7 2024-09-09T08:25:40.011924+0000 If I check the status of those PG it's or active+clean+scrubbing+deep and deep scrubbing for Xs or queued for deep scrub So my questions are : Why ceph tell me «1» pg not been scrub when I see 15 ? Is they are any way to find which pg ceph status are talking about. Is they are any way to see the progress or scrubbing/remapping/backfill ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 20 sept. 2024 09:35:43 CEST

2 4

CPU requirements
by Laszlo Budai 20 Sep '24

20 Sep '24

Hello everyone, I'm trying to understand the CPU requirements for recent versions of CEPH. Reading the documentation (https://docs.ceph.com/en/latest/start/hardware-recommendations/) I cannot get any conclusion about how to plan CPUs for ceph. There is the following statement: "With earlier releases of Ceph, we would make hardware recommendations based on the number of cores per OSD, but this cores-per-osd metric is no longer as useful a metric as the number of cycles per IOP and the number of IOPS per OSD. For example, with NVMe OSD drives, Ceph can easily utilize five or six cores on real clusters and up to about fourteen cores on single OSDs in isolation. So cores per OSD are no longer as pressing a concern as they were. When selecting hardware, select for IOPS per core." How should I understand this? On one side it's saying "with NVMe OSD drives, Ceph can easily utilize five or six cores on real clusters" and then continues: "and up to about fourteen cores on single OSDs in isolation." so should I cont 5/6 cores per NVMe OSD or 14cores? And after all that there's the next sentence that says: " So cores per OSD are no longer as pressing a concern as they were.", But if a single OSD can consume 5/6 cpu cures (that could go up to 14) then I would assume that cores per OSD is still an important concern. Can anyone explain these CPU requirements? Or point me to some other documents that describes in more details the resources required for Ceph? Thank you, Laszlo

4 5

mclock scheduler kills clients IOs
by Denis Polom 20 Sep '24

20 Sep '24

Hi guys, we have Ceph cluster with Quincy 17.2.7 where we are draining 3 hosts (each one is from one failure domain). Each host has 13 HDDs and there are another 38 hosts with same size in each failure domain. There is a lot of free space. I've set up primary-affinity on drained OSDs to 0 and set OSDs out mclock profile is: osd_mclock_profile = high_client_ops but soon after that I got slow ops on some of the OSDs and clients getting high iowaits until I stop recovering. It's strange to me that client is still trying to use OSDs where primary-affinity is set to 0 Here is the case of osd.15 with primary-affinity 0: REQUESTS 4 homeless 0 8337634 osd15 16.35734e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5278.00000004 0x400024 1 write 8358040 osd15 16.ace50e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5a4e.00000007 0x400024 1 write 8438861 osd15 16.a7b80e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7c6a.0000001f 0x400024 1 write 8443242 osd15 16.de504e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7d76.00000014 0x400024 1 write LINGER REQUESTS BACKOFFS Recovery speed is also not high (about 4GiB) and network is not saturated. I've tried also following custom profile with other custom settings: osd advanced osd_mclock_override_recovery_settings true osd advanced osd_async_recovery_min_cost 100 osd advanced osd_max_backfills 1 osd advanced osd_mclock_profile custom osd advanced osd_mclock_scheduler_background_recovery_res 0.100000 osd advanced osd_mclock_scheduler_client_res 0.900000 osd advanced osd_mclock_scheduler_client_wgt 6 osd advanced osd_recovery_max_active 1 osd advanced osd_recovery_max_chunk 8388608 osd advanced osd_recovery_op_priority 1 but result is same. Immediately clients get high iowaits and can't do any IOs. This didn't happen on this cluster before when we were using wpq scheduler. My question is - are there some other options I need to set? With wpq I was able to set osd_recovery_sleep but this option looks like not possble to change with mclock. Any ideas, please?

5 6

VFS: Busy inodes after unmount of ceph lead to kernel panic (maybe?)
by Christian Kugler 19 Sep '24

19 Sep '24

Hi, we had a problem with a Ceph OSD stuck in snaptrim. We set the Ceph OSD down temporarily to potentially solve the issue. This ended up kernel panicking two of our Kubernetes workers that use this Ceph cluster for RBD and CephFS directly and via CSI driver. The more notable lines were these after "osd25 down" and "osd25 up". Especially VFS busy inodes and the mention of generic_shutdown_super which seems to be dealing with inodes seem suspicious to be to be the culprit. 2024-09-16T16:18:37.687175+02:00 HOSTNAME kernel: [6913636.547576] VFS: Busy inodes after unmount of ceph (ceph) 2024-09-16T16:18:37.687191+02:00 HOSTNAME kernel: [6913636.547588] ------------[ cut here ]------------ 2024-09-16T16:18:37.687196+02:00 HOSTNAME kernel: [6913636.548482] kernel BUG at fs/super.c:503! […snip…] 2024-09-16T16:18:37.731920+02:00 HOSTNAME kernel: [6913636.562214] ? generic_shutdown_super.cold+0x1a/0x1c Log from "libceph (…): osd25 down" to reboot as attachment including some null bytes that snuck in somehow. Has anybody run into this before or has an idea how to prevent this in the future? We are currently running Reef (18.2.2, updating soon). Best, Christian

1 0

Deploy rgw different version using cephadm
by Mahdi Noorbala 19 Sep '24

19 Sep '24

Hello Recently I deployed a ceph cluster (version: reef) in my lab and after that, I deployed RGW using this manifest: service_type: rgw service_id: lab-object-storage placement: label: rgw count_per_host: 1 spec: rgw_frontend_port: 8080 Now I have a rgw container. the docker image is: quay.io/ceph/ceph (latest version) My question is: Is there any way to change rgw image version? How can I downgrade rgw version using cephadm? Thanks.

2 1

Ceph Dashboard TLS
by matthew＠peregrineit.net 19 Sep '24

19 Sep '24

Hi All, I'm running an (experimental) 3-Node Ceph Reef (v18.2.4) Cluster. Each of the 3 nodes runs (amongst other services) the Ceph Dashboard - for fail-over purposes. I can connect to the Ceph Dashboard when not using TLS (ie ceph config set mgr mgr/dashboard/ssl false). I've got a private PKI (Step-CA) and I'd like to use its TLS Certificates for the Ceph Dashboard. The private PKI's CA Cert has been added to my browser, and internal websites, etc, using the internal PKI's Certificates work as expected. I'd like each of the three Ceph Nodes to use their own TLS Certificate. The 3 node's hostnames are ceph01.example.com, ceph02.example.com, and ceph03.example.com. The 3 node's mgr instances are (I think; & this could be where I'm going wrong) are mgr.ceph01, mgr.ceph02, and mgr.ceph03. I've created 3 TLS Certificates with names of cephdash-ceph01.example.com.crt, etc, CNs of cephdash-ceph01.example.com, etc, and SANs of ceph01.example.com, etc I've followed the documentation here: https://docs.ceph.com/en/reef/mgr/dashboard/#ssl-tls-support I have stopped and restarted the Ceph Dashboard service. Commands used (on ceph01.example.com) ceph dashboard set-ssl-certificate ceph01 -i cephdash-ceph01.example.com.crt ceph dashboard set-ssl-certificate-key ceph01 -i cephdash-ceph01.example.com.key ceph mgr module disable dashboard ceph mgr module enable dashboard It's not working (I get an Unable to connect page), and I can't work out why - but I suspect it'll be something bleedingly obvious. :-) My Qs: Can I use EC Certs or must I use RSA Certs? I've tryied both and neither work. Can anyone spot what I'm doing wrong? Thanks in advance Cheers Dulux-Oz

1 0

Re: [EXT] mclock scheduler kills clients IOs
by Kai Stian Olstad 18 Sep '24

18 Sep '24

On Tue, Sep 17, 2024 at 08:48:11PM -0400, Anthony D'Atri wrote: >Were all three in the same failure domain? No they were all in different failure domain. -- Kai Stian Olstad

1 0

Re: High usage (DATA column) on dedicated for OMAP only OSDs
by Александр Руденко 18 Sep '24

18 Sep '24

Hi, Anthony! Thank you, but I checked it twice and carefully. No one other PGs on these OSDs. I did it using ceph pg ls-by-osd XX ср, 18 сент. 2024 г. в 14:47, Anthony D'Atri <anthony.datri(a)gmail.com>: > Dump your CRUSH rules and compare with `ceph osd dump | grep pool` > > I suspect you have a data pool using those SSDs that you don’t realize. > > > On Sep 18, 2024, at 7:32 AM, Александр Руденко <a.rudikk(a)gmail.com> > wrote: > > > > Hi! > > > > We have S3/CephFS cluster with dedicated SSDs for bucket indexes, CephFS > > metadata and for a few small rgw metadata pools. > > We have special crush rules for these pools. > > All of our pools (pool naming is old because it's very old cluster) that > > are placed on SSDs: > > POOL ID PGS STORED OBJECTS USED %USED MAX > > AVAIL > > .rgw.root 1 32 1.3 KiB 5 108 KiB 0 > 690 > > GiB > > .rgw.control 2 32 0 B 8 0 B 0 > 690 > > GiB > > .rgw 3 32 2.0 MiB 8.02k 328 MiB 0.02 > 690 > > GiB > > .rgw.gc 4 64 1.6 GiB 64 4.9 GiB 0.24 > 690 > > GiB > > .users.uid 5 32 1.8 MiB 2.63k 67 MiB 0 > 690 > > GiB > > .users 6 32 275 KiB 7.02k 379 MiB 0.02 > 690 > > GiB > > .usage 7 32 30 MiB 64 30 MiB 0 > 690 > > GiB > > .intent-log 8 64 0 B 0 0 B 0 > 690 > > GiB > > .log 9 64 6.6 GiB 34.29k 21 GiB 1.01 > 690 > > GiB > > .rgw.buckets.index 12 4096 1.9 TiB 248.96k 1.9 TiB 48.02 > 690 > > GiB > > .users.email 13 32 64 KiB 1.61k 72 MiB 0 > 690 > > GiB > > fs1_meta 14 64 467 MiB 35.39k 829 MiB 0.04 > 690 > > GiB > > > > But on all our SSDs we can see high DATA usage, for example: > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META > > AVAIL %USE VAR PGS STATUS TYPE NAME > > 57 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 96 GiB > 1.4 > > GiB 40 GiB 90.96 1.37 138 up osd.57 > > 19 ssd 0.00005 1.00000 447 GiB 402 GiB 309 GiB 92 GiB > 1.4 > > GiB 44 GiB 90.09 1.35 145 up osd.19 > > 10 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 97 GiB > 1.1 > > GiB 40 GiB 91.01 1.37 137 up osd.10 > > 12 ssd 0.00005 1.00000 447 GiB 401 GiB 309 GiB 91 GiB > 1.3 > > GiB 46 GiB 89.77 1.35 134 up osd.12 > > > > All SSDs are bluestore. > > Ceph 16.3.11 and many SSDs were deployed on 16.x. > > As I understand, the majority of these pools contain only "OMAP" which is > > stored in RocksDB and tracked in stats as *OMAP*. > > > > And I don't understand why we have such high *DATA* and we can see that > > DATA usage grows by 1-2 GB/day for the last 90 days (we don't have more > > monitoring data)! > > > > I have checked crush rules and all SSDs for PGs from other pools and > can't > > see any other PGs from "pure" data pools. Rules are correct. > > > > How can I see what is stored in bluestore DATA on some OSD? I can export > > full RocksDB, but not DATA stored in bluestore.. > > What kind of DATA can be stored in these OSDs with these pools? > > We did a lot of offline compactions, 2-5 times for some SSDs in the last > 90 > > days. But i'm not sure if it's related > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

For 750 osd 5 monitor in redhat doc
by Szabo, Istvan (Agoda) 18 Sep '24

18 Sep '24

Hi, I went through some Redhat doc and saw this sentence which I didn't see in Ceph docs: "The storage cluster can run with only one Ceph Monitor; however, to ensure high availability in a production storage cluster, Red Hat will only support deployments with at least three Ceph Monitor nodes. Red Hat recommends deploying a total of 5 Ceph Monitors for storage clusters exceeding 750 Ceph OSDs." here: https://docs.redhat.com/en/documentation/red_hat_ceph_storage/4/html/instal… We will reach this number so wonder do we really need 5 mon? Currently we have 3 collocated mgr/mon on baremetal server with some multiste gw traffic, if needed I'd plan to simply add some mon on vm only. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 2

lifecycle for versioned bucket
by Lukasz Borek 17 Sep '24

17 Sep '24

Hi, I'm having issue with lifecycle jobs for 18.2.4 cluster with versioning enabled bucket. /# radosgw-admin lc list [ { "bucket": ":mongobackup-prod:c3e0a369-71df-40f5-a5c0-51e859efe0e0.96754.1", "shard": "lc.0", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" } ] During RGW startup : 2024-09-17T16:51:06.148+0000 7f944d02d640 2 garbage collection: garbage collection: start 2024-09-17T16:51:06.152+0000 7f944501d640 2 lifecycle: life cycle: start 2024-09-17T16:51:06.152+0000 7f944501d640 5 lifecycle: RGWLC::process(): ENTER: index: 26 worker ix: 0 2024-09-17T16:51:06.152+0000 7f943f812640 2 lifecycle: life cycle: start 2024-09-17T16:51:06.152+0000 7f943f812640 5 lifecycle: RGWLC::process(): ENTER: index: 18 worker ix: 1 2024-09-17T16:51:06.152+0000 7f944501d640 5 lifecycle: RGWLC::process() process shard rollover lc_shard=lc.26 head.marker= head.shard_rollover_date=0 2024-09-17T16:51:06.152+0000 7f944501d640 5 lifecycle: RGWLC::process() entry.bucket.empty() == true at START 1 (this is possible mainly before any lc policy has been stored or after removal of an lc_shard object) 2024-09-17T16:51:06.152+0000 7f943a007640 2 lifecycle: life cycle: start 2024-09-17T16:51:06.152+0000 7f943a007640 5 lifecycle: RGWLC::process(): ENTER: index: 17 worker ix: 2 2024-09-17T16:51:06.156+0000 7f943f812640 5 lifecycle: RGWLC::process() process shard rollover lc_shard=lc.18 head.marker= head.shard_rollover_date=0 2024-09-17T16:51:06.156+0000 7f943f812640 5 lifecycle: RGWLC::process() entry.bucket.empty() == true at START 1 (this is possible mainly before any lc policy has been stored or after removal of an lc_shard object) 2024-09-17T16:51:06.156+0000 7f944501d640 5 lifecycle: RGWLC::process(): ENTER: index: 9 worker ix: 0 [...] 2024-09-17T16:51:06.568+0000 7f94137ba640 2 lifecycle: life cycle: stop 2024-09-17T16:51:06.572+0000 7f94137ba640 5 lifecycle: schedule life cycle next start time: Wed Sep 18 01:00:00 2024 Scheduled job ends with: 2024-09-17T00:00:01.379+0000 7fd17f843700 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.31 head last stored at Mon Sep 16 06:34:39 2024 2024-09-17T00:00:01.379+0000 7fd17f843700 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 Anyone can explain the logic behind that? Thanks. -- Łukasz Borek lukasz(a)borek.org.pl

1 0

Strange CephFS Permission error
by Carsten Feuls 17 Sep '24

17 Sep '24

Hello, a cluster from my friend has an strage issue with CephFS Permission. Ceph Version is 17.2.7 in cephadm. The Cluster was much older and was upgraded over years from 12 and moved to new hosts. When I create a cephfs user for my private cluster with "ceph fs authorize" I get permission like this. client.test2 key: <placeholder caps: [mds] allow r fsname=cephfs, allow rw fsname=cephfs path=/all caps: [mon] allow r fsname=cephfs caps: [osd] allow rw tag cephfs data=cephfs For my cluster this is working fine. But for the cluster from my friend I get only metadata access no file access. I figured it out that when I modifie the caps like this I get it working again. client.test2 key: <placeholder caps: [mds] allow r fsname=cephfs, allow rw fsname=cephfs path=/all caps: [mon] allow r fsname=cephfs caps: [osd] allow rw But this is a solution I do not wan't to keep and every auth createtd with "ceph fs authorize" is not working. ceph osd get-require-min-compat-client -> nautilus Any Ideas how to solve this? Cheers Carsten Feuls

1 0

Ceph RBD w/erasure coding
by andre＠drenet.net 17 Sep '24

17 Sep '24

I have a Ceph instance I'm wherein I'm trying to use erasure coding with RBD (and libvirt). I've followed https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-codi… and enabled overwrites so that RBD can be used. In doing so, I've set the data_pool to the erasure coded pool, with set to the replicated pool. I have the following in ceph.conf: rbd_default_data_pool = libvirt-pool-ec Here's an rbd info on the image I've created (notice the "data_pool" config): rbd image 'xxxxxxx': size 500 GiB in 128000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 7e37e442592b data_pool: libvirt-pool-ec block_name_prefix: rbd_data.6.7e37e442592b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Fri Sep 13 16:34:06 2024 access_timestamp: Fri Sep 13 16:34:06 2024 modify_timestamp: Fri Sep 13 16:34:06 2024 The problem, when I attach this rbd image to a VM, I cannot write to it at all, I get an I/O error. But, when I create and rbd image without the "rbd_default_data_pool = libvirt-pool-ec" setting, I can write fine. Wanted to see if anyone has any ideas, maybe I'm missing something simple. Thanks in advance!

6 6

Re: Ceph RBD w/erasure coding
by andre＠drenet.net 17 Sep '24

17 Sep '24

My VM was running Reef, actually. But I determined the issue to be write perms on the erasure coded pool. The resolution in my case was editing my client.libvirt user adding to the mon caps, and allowing the erasure coded pool on the osd caps: ceph auth caps client.libvirt mon 'allow r, allow command "osd blacklist", allow command "osd blocklist", allow command "blacklistop", allow command "blocklistop"' osd 'allow class-read object_prefix rbd_children, allow rwx pool=libvirt-pool, allow rwx pool=images, allow rwx pool=libvirt-pool-ec' :) On 2024-09-13 14:58, Anthony D'Atri wrote: > Is your VM running something older than Luminous? Depending on how > your attachment is being done, you may need to specify both pools on > the client side too. > >> On Sep 13, 2024, at 1:45 PM, andre(a)drenet.net wrote: >> >> I have a Ceph instance I'm wherein I'm trying to use erasure coding >> with RBD (and libvirt). I've followed >> https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-codi… >> and enabled overwrites so that RBD can be used. In doing so, I've set >> the data_pool to the erasure coded pool, with set to the replicated >> pool. >> >> I have the following in ceph.conf: >> >> rbd_default_data_pool = libvirt-pool-ec >> >> Here's an rbd info on the image I've created (notice the "data_pool" >> config): >> >> rbd image 'xxxxxxx': >> size 500 GiB in 128000 objects >> order 22 (4 MiB objects) >> snapshot_count: 0 >> id: 7e37e442592b >> data_pool: libvirt-pool-ec >> block_name_prefix: rbd_data.6.7e37e442592b >> format: 2 >> features: layering, exclusive-lock, object-map, fast-diff, >> deep-flatten, data-pool >> op_features: >> flags: >> create_timestamp: Fri Sep 13 16:34:06 2024 >> access_timestamp: Fri Sep 13 16:34:06 2024 >> modify_timestamp: Fri Sep 13 16:34:06 2024 >> The problem, when I attach this rbd image to a VM, I cannot write to >> it at all, I get an I/O error. But, when I create and rbd image >> without the "rbd_default_data_pool = libvirt-pool-ec" setting, I can >> write fine. >> >> Wanted to see if anyone has any ideas, maybe I'm missing something >> simple. Thanks in advance! >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Blocking/Stuck file
by dominik.baack 17 Sep '24

17 Sep '24

Hi, we started upgrading our Ceph Cluster consisting of 7 Nodes from quincy to reef two days ago. This included the upgrade of the underlying OS and several other small changes. After hitting the osd_remove_queue bug, we could recover mostly, but are still in a non-healthy state because of changes on our network. (See attached image) But overall we can mount the filesystem on all nodes, read and write. The Problem now is that at least one file created by slurmctld exist which seems to be somehow compromised during this. It cannot be read or removed on any of the storage or working nodes. All operations (cat, rm, less, ...) are stuck until they are forcefully terminated. I checked all nodes manually and could not identify any process with an open handle to the file. If possible we would like to unblock the file, but removing would also be a possiblity. Cheers Dominik

2 2

Grand Piano 9/17
by Michelle Fangman 17 Sep '24

17 Sep '24

Hello, I'm offering my late husband's Yamaha Piano to any music enthusiast who may appreciate it. If you or someone you know might be interested in receiving this instrument for free, please feel free to reach out to me. Warm regards, Michelle

1 0

Ceph octopus version cluster not starting
by Amudhan P 17 Sep '24

17 Sep '24

Hi, Recently added one disk in Ceph cluster using "ceph-volume lvm create --data /dev/sdX" but the new OSD didn't start. After some rest of the other nodes OSD service also stopped. So, I restarted all nodes in the cluster now after restart. MON, MDS, MGR and OSD services are not starting. Could find any new logs also after restart it is totally silent in all nodes. Could find some logs in Ceph-volume service. Error in Ceph-volume logs :- [2024-09-15 23:38:15,080][ceph_volume.process][INFO ] stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-5 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-33cd42cd-8570-47de-8703-d7cab1acf2ae/osd-block-21968433-bb53-4415-b9e2-fdc36bc4a28e --path /var/lib/ceph/osd/ceph-5 --no-mon-config stderr: failed to read label for /dev/ceph-33cd42cd-8570-47de-8703-d7cab1acf2ae/osd-block-21968433-bb53-4415-b9e2-fdc36bc4a28e: (2) No such file or directory 2024-09-15T23:38:15.059+0530 7fe7767c8100 -1 bluestore(/dev/ceph-33cd42cd-8570-47de-8703-d7cab1acf2ae/osd-block-21968433-bb53-4415-b9e2-fdc36bc4a28e) _read_bdev_label failed to open /dev/ceph-33cd42cd-8570-47de-8703-d7cab1acf2ae/osd-block-21968433-bb53-4415-b9e2-fdc36bc4a28e: (2) No such file or directory --> RuntimeError: command returned non-zero exit status: 1 [2024-09-15 23:38:15,084][ceph_volume.process][INFO ] stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-9a9b8328-66ad-4997-8b9f-5216b56b73e8/osd-block-ac2ae41d-3b77-4bfd-ba5c-737e4266e988 --path /var/lib/ceph/osd/ceph-2 --no-mon-config stderr: failed to read label for /dev/ceph-9a9b8328-66ad-4997-8b9f-5216b56b73e8/osd-block-ac2ae41d-3b77-4bfd-ba5c-737e4266e988: (2) No such file or directory But I could find " /dev/ceph-9a9b8328-66ad-4997-8b9f-5216b56b73e8/osd-block-ac2ae41d-3b77-4bfd-ba5c-737e4266e988" the path valid and listing folder. Not sure how to proceed or where to start any idea or suggestion ?

3 11

Bucket Notifications v2 & Multisite Redundancy
by Alex Hussein-Kershaw (HE/HIM) 16 Sep '24

16 Sep '24

Hi Folks, I see in the pending release notes for Squid a description of the "notification_v2" feature. ceph/PendingReleaseNotes at main · ceph/ceph (github.com)<https://github.com/ceph/ceph/blob/main/PendingReleaseNotes#L43> I have some concerns about the multisite nature of this feature, I am using Ceph multisite to provide geographic redundancy for my application data. That seems to be a mainline use case. My application is designed to cope if an entire zone fails. My application currently relies on pubsub (and soon to be replaced by bucket notifications) to provide service. More specifically I rely on the ability to semi-regularly automatically reconfigure topics to adjust the HTTP endpoint. I believe with the "notification_v2" feature all adjustments of the topic are routed through the master for metadata site (much like creating an S3 user). Fundamentally I think this breaks the combination of Ceph multisite, notifications_v2 and reconfiguring a topic; you cannot reconfigure a topic unless both sites are up. I tested this, by setting up a deployment, shutting down a site and trying to reconfigure a topic. I get a 500 HTTP response to my createTopic, and can see this in the RGW logs: debug 2024-09-03T08:24:39.997+0000 7f53a3372640 20 ERROR: curl error: Couldn't connect to server req_data->error_buf=Failed to connect to 10.225.20.19 port 7480: No route to host debug 2024-09-03T08:24:39.997+0000 7f5329a7f640 4 req 2968544598970267460 0.686004043s sns:pubsub_topic_create CreateTopic forward_request_to_master returned ret = -2200 Have the implications of this been considered? For now I may be able to disable the notifications_v2 feature, however in the release note the predecessor is deprecated in Squid. Hopefully I'm missing something obvious. Kind regards, Alex

2 8

Metric or any information about disk (block) fragmentation
by Szabo, Istvan (Agoda) 16 Sep '24

16 Sep '24

Hi, Is there any way to pull out the fragmentation value from the disk somehow without shutdown the osd and run the ceph-objectstore-tool command? We had a total outage last week which is based on multiple bugs but most of them are down to disk fragmentation which is super high in our case. I'm trying to find in the logs and in ceph metrics some hint but I couldn't find anything how I can monitor or at least check this value: ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-218/ --command free-score block: { "fragmentation_rating": 0.90989583876508828 } bluefs-db: { "fragmentation_rating": 0.41012391001249843 } failure querying 'bluefs-wal' 2024-09-15T12:19:31.814+0700 7f90c66d00c0 -1 asok(0x55c637e7c000) AdminSocket: request '[{"prefix": "bluestore allocator score bluefs-wal"}]' not defined Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

telemetry
by Marc 16 Sep '24

16 Sep '24

maybe add filter for types as well? https://telemetry-public.ceph.com/d/x1_ISxiMz_01/models-per-vendor?orgId=1

1 0

no mds services
by Ex Calibur 15 Sep '24

15 Sep '24

Hello, I'm following this guide to upgrade our cephs: https://ainoniwa.net/pelican/2021-08-11a.html (Proxmox VE 6.4 Ceph upgrade Nautilus to Octopus) It's a requirement to upgrade our ProxMox environnement. Now I've reached the point at that guide where i have to "Upgrade all CephFS MDS daemons" But before I started this piece, I checked the status. root@pmnode1:~# ceph status cluster: id: xxxxxxxxxxxxxxx health: HEALTH_ERR noout flag(s) set 1 scrub errors Possible data damage: 1 pg inconsistent 2 pools have too many placement groups services: mon: 3 daemons, quorum pmnode1,pmnode2,pmnode3 (age 19h) mgr: pmnode2(active, since 19h), standbys: pmnode1 osd: 15 osds: 12 up (since 12h), 12 in (since 19h) flags noout data: pools: 3 pools, 513 pgs objects: 398.46k objects, 1.5 TiB usage: 4.5 TiB used, 83 TiB / 87 TiB avail pgs: 512 active+clean 1 active+clean+inconsistent root@pmnode1:~# ceph mds metadata [] as you can see there is no mds service running. What can be wrong and how to solve this? Thank you in advance.

3 2

Numa pinning best practices
by Szabo, Istvan (Agoda) 15 Sep '24

15 Sep '24

Hi, Haven't really found a proper descripton in case of 2 socket how to pin osds to numa node, only this: https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment… Tuning for All Flash Deployments - Ceph - Ceph <https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment…> Redmine tracker.ceph.com Is there anybody have some good how to on this topic? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 5

[RGW][cephadm] How to configure RGW as code and independantely of daemon names ?
by Gilles Mocellin 14 Sep '24

14 Sep '24

Hello Cephers, In our journey to migrate to cephadm, I face a problem with RGW: We use Ceph with OpenStack, so we have to configure keystone authentification, with many rgw_keystone_* variables in /etc/ceph/ceph.conf. It's easy with ceph-ansible, and the section in ceph.conf has always the same name, constructed from the inventory. With cephadm, daemon names for RGW contain random characters, and change at every redeploy. As we dont't have ceph.conf anymore, the configuration should be in ceph config. But how can we have it set once, and preserved between name changes ? I wish I could set thoses config vars in the RGW spec file, having a good "as code" approach, but no, extra params are only for the radosgw binary. It would also be fine if we could have defined ceph confg keys with wildcards, but no. How do you handle rados GW config, with cephadm and ceph config ? -- Gilles

5 6

Re: ceph-mgr perf throttle-msgr - what is caused fails?
by Konstantin Shalygin 14 Sep '24

14 Sep '24

As I said before, currently Prometheus module performance degradation is only one _visible_ issues. I named things like is as indicator (of feature problem's) k Sent from my iPhone > On 12 Sep 2024, at 23:18, Eugen Block <eblock(a)nde.ag> wrote: > > But did you notice any actual issues or did you just see that value being that high without any connection to an incident?

2 2

bluefs _allocate unable to allocate on bdev 2
by Szabo, Istvan (Agoda) 13 Sep '24

13 Sep '24

Hi, Since yesterday on ceph octopus it started to crash multiple osds in the cluster and I can see this error in most of the logs: 2024-09-12T06:13:35.805+0700 7f98b8b27700 1 bluefs _allocate failed to allocate 0xf0732 on bdev 1, free 0x40000; fallback to bdev 2 2024-09-12T06:13:35.805+0700 7f98b8b27700 1 bluefs _allocate unable to allocate 0xf0732 on bdev 2, free 0xffffffffffffffff; fallback to slow device expander The osds are 57% full so I think space issue shouldn't happen. I'm using ssds so I don't have separated wal/rocksdb. Running some compaction now but I don't think it will happen on a long run. What could be this issue and how to fix? Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 7

squid 19.2.0 QE validation status
by Yuri Weinstein 12 Sep '24

12 Sep '24

Details of this release are summarized here: https://tracker.ceph.com/issues/67779#note-1 Release Notes - TBD Gibba upgrade -TBD LRC upgrade - TBD It was decided and agreed upon that there would be limited testing for this release, given it is based on 19.1.1 rather than a full rebase. Seeking approvals/reviews for: (some reruns are still in progress) rgw - Eric, Adam E fs - Venky orch - Adam King quincy-x - Laura, Neha reef-x - Laura, Neha Pls let me know if any tests were missed from this list.

7 18

lifecycle policy on non-replicated buckets
by Christopher Durham 12 Sep '24

12 Sep '24

Hi, I have a multisite system with two sites on 18.2.2, on Rocky 8. I have set up a sync policy to allow replication between sites. I have also createda policy for a given bucket that prevents replication on that given bucket. This allworks just fine, and objects I create in that bucket on side A do not get replicatedto side B, and objects I create in that bucket on side B do not get replicated to side A, butreplication for all other buckets works fine. This is great. But I still want to have a lifecycle policy that deletes objects after say, 4 days. If I create and install the json for this policy, via s3api put-bucket-lifecycle-configuration then only on the master side, side A, do objects get deleted after 4 days. Any objects on side B never get deleted. What am I doing wrong? Thanks -Chris

2 4

ceph-mgr perf throttle-msgr - what is caused fails?
by Konstantin Shalygin 12 Sep '24

12 Sep '24

Hi, seems something in mgr is throttle due val > max. I'm right? root@mon1# ceph daemon /var/run/ceph/ceph-mgr.mon1.asok perf dump | jq '."throttle-msgr_dispatch_throttler-mgr-0x55930f4aed20"' { "val": 104856554, "max": 104857600, "get_started": 0, "get": 9700833, "get_sum": 654452218418, "get_or_fail_fail": 1323887918, "get_or_fail_success": 9700833, "take": 0, "take_sum": 0, "put": 9698716, "put_sum": 654347361864, "wait": { "avgcount": 0, "sum": 0, "avgtime": 0 } } The question is - how-to determine what exactly? Another fail_fail in perf counters is zero. mgr is not in container, and have resources to work Thanks, k

2 3

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

RGW sync gets stuck every day
by Olaf Seibert 12 Sep '24

12 Sep '24

Hi all, we have some Ceph clusters with RGW replication between them. It seems that in the last month at least, it gets stuck at around the same time ~every day. Not 100% the same time, and also not 100% of the days, but in the more recent days seem to happen more, and for longer. With "stuck" I mean that the "oldest incremental change not applied" is getting 5 or more minutes old, and not changing. In the past this seemed to resolve itself in a short time, but recently it didn't. It remained stuck at the same place for several hours. Also, on several different occasions I noticed that the shard number in question was the same. We are using Ceph 18.2.2, image id 719d4c40e096. The output on one end looks like this (I redacted out some of the data because I don't know how much of the naming would be sensitive information): root@zone2:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) current time 2024-08-04T10:22:00Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [30,90,95] oldest incremental change not applied: 2024-08-04T10:05:54.015403+0000 [30] while on the other side it looks ok (not more than half a minute behind): root@zone1:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) current time 2024-08-04T10:23:05Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 4 shards behind shards: [89,92,95,98] oldest incremental change not applied: 2024-08-04T10:22:53.175975+0000 [95] With some experimenting, we found that redeploying the RGWs on this side resolves the situation: "ceph orch redeploy rgw.zone1-backup". The shards go into "Recovering" state and after a short time it is "caught up with source" as well. Redeploying stuff seems like a much too big hammer to get things going again. Surely there must be something more reasonable? Also, any ideas about how we can find out what is causing this? It may be that some customer has some job running every 24 hours, but that shouldn't cause the replication to get stuck. Thanks in advance, -- Olaf Seibert Site Reliability Engineer SysEleven GmbH Boxhagener Straße 80 10245 Berlin T +49 30 233 2012 0 F +49 30 616 7555 0 https://www.syseleven.de https://www.linkedin.com/company/syseleven-gmbh/ Current system status always at: https://www.syseleven-status.net/ Company headquarters: Berlin Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens Plogsties

3 4

Seeking a (paid) consultant - Ceph expert.
by ceph＠denovogroup.org 11 Sep '24

11 Sep '24

We are a small non-profit company that develops technologies for affordable delivery of broadband Internet access to rural communities in developing regions. We are experimenting with a small Ceph cluster that is housed in a street-side cabinet. Our goals are to maintain availability, and avoid data-loss, even under extreme and hardware failures. We are NOT sensitive to performance, scalability or optimization of storage size --- rather high degree of replication and focus on redundancy. Our current experiment consists of 3 rack-mount chassis by Dell - PowerEdge C6220. Each chassis holds 4 server blades. Total of 12 server blades. Each blade has two 10Gig network cards, used in an MLAG (Multi-chassis Link Aggregation Group) configuration to two redundant network switches. Each blade has 4 SAS drives (two SSDs and two HDDs). Each blade has 192Gb of RAM and 24 cores of Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (2 Sockets). We are using Proxmox (we're no experts, but got it working pretty nicely, migration of workloads and all). We are using Ceph version 16.2.9 and manage its basics via the Proxmox integration/GUI with nearly default settings, and without understanding much of what we're doing. (We made minor edits to the Crash-maps - but I'll admit that I don't really understand Ceph's building blocks and nuances). We need hand-holding in re-designing the system with our specific (extreme) availability goals -- we don't need step-by-step spoon-feeding, but rather big-picture design guidelines. Someone who understands deeply the specifics and nuances of Ceph, but also can maintain a big-picture operational view of our needs. Ideally reply directly to me via email - although a discussion on the list could probably be useful as well. (I don't even know what questions to ask). Thanks! Yahel. Yahel Ben-David, Ph.D. De Novo Group - Executive Director Bridging the gap between research and impact.

1 0

Re: [RGW][CEPHADM] Multisite configuration and Ingress
by Gilles Mocellin 11 Sep '24

11 Sep '24

Yes, I've read some stories about that in this mailing list, but, my question was not clear enough, I want to know what's possible to do with cephadm : - Is it possible to create several Ingress with cephadm ? (I can't reach my test env to test until tomorrow !) - If we do, how can we configure differently thoses groups of RGWs daemons ? (related to my previous question on ceph config / ceph.conf) Le mercredi 11 septembre 2024, 22:23:14 CEST Anthony D'Atri a écrit : > I've seen deployments with more than one cohort of RGWs. E.g. > > * one group of RGWs that serve internal requests with an LB > * one group of RGWs that serve external requests with a different LB > * one group of RGWs that only process LC / GC events > > > On Sep 11, 2024, at 4:01 PM, Gilles Mocellin > > <gilles.mocellin(a)nuagelibre.org> wrote: > > > > Hi again, > > > > I wonder if there is a better way of configuring multisite configuration, > > especially zone/zonegroup endpoints : > > > > - many endpoints, pointing directly at RGW daemons > > - one endpoint, pointing at the load balancer VIP of Ingress > > > > If we want to separate client access from replication, is it possible to > > have two Ingress ? It can be great to have them in two different > > networks. > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

[RGW][CEPHADM] Multisite configuration and Ingress
by Gilles Mocellin 11 Sep '24

11 Sep '24

Hi again, I wonder if there is a better way of configuring multisite configuration, especially zone/zonegroup endpoints : - many endpoints, pointing directly at RGW daemons - one endpoint, pointing at the load balancer VIP of Ingress If we want to separate client access from replication, is it possible to have two Ingress ? It can be great to have them in two different networks.

1 0

Re: bilog trim fails with "No such file or directory"
by Florian Schwab 11 Sep '24

11 Sep '24

Hi, sorry forgot to add this. Ceph release: 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) The cluster consists out of 360 OSDs and the index pool has 128 PGs and auto-resharding is enabled. Cheers, Florian > On 11. Sep 2024, at 18:18, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > > Which Ceph release? How many OSDs and how many PBs in the index pool? Auto-resharding enabled? > >> On Sep 11, 2024, at 10:24 AM, Florian Schwab <fschwab(a)impossiblecloud.com> wrote: >> >> Hi everyone, >> >> I hope maybe someone here has an idea what is happening here or can give some pointers how to debug it further. >> >> We currently have a bucket which has large omap objects. Following this guide (https://access.redhat.com/solutions/6450561) we are able to identify the bucket etc. >> >> $ ceph health detail >> HEALTH_WARN 25 large omap objects >> [WRN] LARGE_OMAP_OBJECTS: 25 large omap objects >> 25 large objects found in pool 'ceph-objectstore.rgw.buckets.index' >> Search the cluster log for 'Large omap object found' for more details. >> >> $ radosgw-admin metadata list --metadata-key bucket.instance | grep -i XXX >> “YYY:XXX”, >> >> $ radosgw-admin bilog list --bucket-id=“XXX" --bucket=“YYY" --max-entries=600000 | grep -c op_id >> 600000 >> >> So far so good! But when try to trim the bilog we get the following error: >> >> $ radosgw-admin bilog trim --bucket-id=“XXX" --bucket=“YYY" >> ERROR: trim_bi_log_entries(): (2) No such file or directory >> >> >> The bucket itself doesn’t show any issues - all S3 operations are working. >> >> >> Thanks for any input! >> >> >> Cheers, >> Florian >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

bilog trim fails with "No such file or directory"
by Florian Schwab 11 Sep '24

11 Sep '24

Hi everyone, I hope maybe someone here has an idea what is happening here or can give some pointers how to debug it further. We currently have a bucket which has large omap objects. Following this guide (https://access.redhat.com/solutions/6450561) we are able to identify the bucket etc. $ ceph health detail HEALTH_WARN 25 large omap objects [WRN] LARGE_OMAP_OBJECTS: 25 large omap objects 25 large objects found in pool 'ceph-objectstore.rgw.buckets.index' Search the cluster log for 'Large omap object found' for more details. $ radosgw-admin metadata list --metadata-key bucket.instance | grep -i XXX “YYY:XXX”, $ radosgw-admin bilog list --bucket-id=“XXX" --bucket=“YYY" --max-entries=600000 | grep -c op_id 600000 So far so good! But when try to trim the bilog we get the following error: $ radosgw-admin bilog trim --bucket-id=“XXX" --bucket=“YYY" ERROR: trim_bi_log_entries(): (2) No such file or directory The bucket itself doesn’t show any issues - all S3 operations are working. Thanks for any input! Cheers, Florian

1 0

Blocking/Stuck/Corrupted files
by dominik.baack 11 Sep '24

11 Sep '24

Hi, we started upgrading our Ceph Cluster consisting of 7 Nodes from quincy to reef two days ago. This included the upgrade of the underlying OS and several other small changes. After hitting the osd_remove_queue bug, we could recover mostly, but are still in a non-healthy state because of changes on our network. (See attached image) But overall we can mount the filesystem on all nodes, read and write. The Problem now is that at least one file created by slurmctld exist which seems to be somehow compromised during this. In addition there a several more files which are currently unidentified but noticable through stuck container executions. Each file cannot be read or removed on any of the storage or working nodes. All operations (cat, rm, less, ...) are stuck until they are forcefully terminated. I checked all nodes manually and could not identify any process with an open handle to the file. A manual deep scrub of all pgs and normal scrub does not show any further problems If possible we would like to identify and unblock or remove the compromised files. Cheers Dominik

1 0

Re: Somehow throotle recovery even further than basic options?
by Szabo, Istvan (Agoda) 11 Sep '24

11 Sep '24

Hi, I've played around a bit with these scripts, finally I went with the pgremapper one because it's documented better and no need multiple run but I'd like to get some confirmation about the functionality from someone who used it because it was too fast. What I've tested is the cancel-backfill (pgremapper cancel-backfill --yes) option as written in the documentation: https://github.com/digitalocean/pgremapper which is based on the upmap-remapped.py It made me 2 questions, hopefully someone can answer: Q1 regarding osd removal: What I've done in test env: 1. Set the flags: ceph osd set nobackfill;ceph osd set norebalance 2. Reweight all the osds that I want to remove: for i in {36..43};do ceph osd reweight $i 0.0;done 3. Run the tool and wait couple of minutes to get back the prompt: pgremapper cancel-backfill --yes 4. Unset flags: ceph osd unset norebalance;ceph osd unset nobackfill 5. When cluster health ok, I removed the osds: for num in {36..43}; do ceph osd out osd.$num;systemctl disable ceph-osd@$num;systemctl purge $num --yes-i-really-mean-it;umount /var/lib/ceph/osd/ceph-$num;done 1. In theory here I'm missing a lot of chunks so how I know that all the data actually recovered/regenerated somewhere else in the cluster? Because it was too fast. If the ceph balancer status says no more optimization that is considered to be the done signal? Q2 regarding the process: Could this tool be used for removal actually or only extension? Reweight can be up and down however they say augment which for me extension. "Example - Cancel all backfill in the system as a part of an augment This is useful during augment scenarios, if you want to control PG movement to the new nodes via the upmap balancer (a technique based on this CERN talk<https://www.slideshare.net/Inktank_Ceph/ceph-day-berlin-mastering-ceph-oper…>." Thank you ________________________________ From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Sent: Monday, September 9, 2024 2:07 PM To: Janne Johansson <icepic.dz(a)gmail.com> Subject: Re: [ceph-users] Re: Somehow throotle recovery even further than basic options? Let me try to understand, so this actually move pg by pg to the new osds which means will not push the osds on the same hosts too hard due to 1 pg at a time. Am I correct? Also if something goes wrong, is it possible to stop this script and let it go as is or if I want it faster, can I? As I see this can be used during rebalance also, just need to stop rebalance, run the script and unset rebalance. ________________________________ From: Janne Johansson <icepic.dz(a)gmail.com> Sent: Saturday, September 7, 2024 2:00 PM To: Eugen Block <eblock(a)nde.ag> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Somehow throotle recovery even further than basic options? Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ The pgremapper (and the python one) will allow you to mark all the PGs that a new disk gets as an empty-misplaced PG to be correct where they currently are. This means that after you run one of the remappers, the upmap will tell the cluster to stay as it is even though new empty OSDs have arrived with correct crush weights and all. So you set norebalance, add N+1 new OSDs, the cluster shudders for a short while when the new-empty PGs are created on the new drives, then you have lots and lots of misplaced PGs which norebalance prevents from starting backfills on. Then you run the remapper and "fix" the upmap so all the current placements are considered "correct", and hence the PGs that were supposed to move stop being misplaced. After this, you remove "norebalance" to allow moves to start happening. By now, no movement (or at least very few PGs) should occur. What happens next is that the balancer notices there actually is more space, and figures out the optimal result is more or less the same as above where lots of PGs should go to the new OSDs, but it does this with the max-misplaced-ratio in mind, and it will "move" the PGs by just unsetting their upmap entry that forced them to want to stay in place, so as time passes, it moves a few PGs at a time, and moves them by removing the upmaps from them so that most of the time your other OSDs will look perfectly healthy and will continue to do all the scrubs and things a healthy OSD should do and which it will not do if it has a long queue of backfills waiting to eat up all the slots for non-client IO. While you can do soft additions with increasing crush weights, this potentially causes lots of more movements since an OSD host with OSDs going from 0.1 to 0.2 weight might not place all PGs in the same spot in those two cases, so you could have movement within the host and so on from the recalculated pseudorandom placements on every increase. Den lör 7 sep. 2024 kl 00:15 skrev Eugen Block <eblock(a)nde.ag>: > > I can’t say anything about the pgremapper, but have you tried > increasing the crush weight gradually? Add new OSDs with crush initial > weight 0 and then increase it in small steps. I haven’t used that > approach for years, but maybe that can help here. Or are all OSDs > already up and in? Or you could reduce the max misplaced ratio to 1% > or even lower (default is 5%)? > > Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo(a)agoda.com>: > > > Forgot to paste, somehow I want to reduce this recovery operation: > > recovery: 0 B/s, 941.90k keys/s, 188 objects/s > > To 2-300Keys/sec > > > > > > > > ________________________________ > > From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> > > Sent: Friday, September 6, 2024 11:18 PM > > To: Ceph Users <ceph-users(a)ceph.io> > > Subject: [ceph-users] Somehow throotle recovery even further than > > basic options? > > > > Hi, > > > > 4 years ago we've created our cluster with all disks 4osds (ssds and > > nvme disks) on octopus. > > The 15TB SSDs still working properly with 4 osds but the small 1.8T > > nvmes with the index pool not. > > Each new nvme osd adding to the existing nodes generates slow ops > > with scrub off, recovery_op_priority 1, backfill and recovery 1-1. > > I even turned off all index pool heavy sync mechanism but the read > > latency still high which means recovery op pushes it even higher. > > > > I'm trying to somehow add resource to the cluster to spread the 2048 > > index pool pg (in replica 3 means 6144pg index pool) but can't make > > it more gentle. > > > > The balancer is working in upmap with max deviation 1. > > > > Have this script from digitalocean > > https://github.com/digitalocean/pgremapper, is there anybody tried > > it before how is it or could this help actually? > > > > Thank you the ideas. > > > > ________________________________ > > This message is confidential and is for the sole use of the intended > > recipient(s). It may also be privileged or otherwise protected by > > copyright or other legal rules. If you have received it by mistake > > please let us know by reply email and delete it from your system. It > > is prohibited to copy this message or disclose its content to > > anyone. Any confidentiality or privilege is not waived or lost by > > any mistaken delivery or unauthorized disclosure of the message. All > > messages sent to and from Agoda may be monitored to ensure > > compliance with company policies, to protect the company's interests > > and to remove potential malware. Electronic messages may be > > intercepted, amended, lost or deleted, or contain viruses. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

RFC: cephfs fallocate
by Milind Changire 11 Sep '24

11 Sep '24

Problem: CephFS fallocate implementation does not actually reserve data blocks when mode is 0. It only truncates the file to the given size by setting the file size in the inode. So, there is no guarantee that writes to the file will succeed Solution: Since an immediate remediation of this problem is not possible, I request users to vote on the most suitable approach to avoid breaking dependent software: 1. report EOPNOTSUPP (operation not supported) when mode is 0 2. continue with existing implementation of file size truncation when mode is 0 -- Milind

2 2

User + Dev Monthly Meetup coming up on Sept. 25th!
by Laura Flores 10 Sep '24

10 Sep '24

Hi all, The User + Dev Monthly Meetup is coming up on Sept. 25th! This month, we will have a discussion on incorporating the upmap-remapped [1] & pgremapper [2] tools with the balancer mgr module. There has been interest in the community for a while about incorporating the logic from these external tools with the existing Ceph code. There will be a presentation on this high-level design doc: https://tracker.ceph.com/issues/67418 See you all there! RSVP here: https://www.meetup.com/ceph-user-group/events/303358179/?utm_medium=referra… 1. https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-rema… 2. https://github.com/digitalocean/pgremapper -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

CLT meeting notes: Sep 09, 2024
by David Orman 10 Sep '24

10 Sep '24

CLT discussion on Sep 09, 2024 19.2.0 release: * Cherry picked patch: https://github.com/ceph/ceph/pull/59492 * Approvals requested for re-runs CentOS Stream/distribution discussions ongoing * Significant implications in infrastructure for building/testing requiring ongoing discussions/work to determine best course of action/path forward Vstart documentation cleanup * Zac Dover working on this documentation, open to any suggestions/ideas (commands that people new to Ceph would find useful — idea like "test-driving a vehicle"

1 0

Multisite replication design
by Nathan MALO 10 Sep '24

10 Sep '24

Hi, I am currently setting up a multi-site replication with rook-ceph. My setup is the following: - clusterA in regionA containing local buckets (let's call them bucketsA) - clusterB in regionB containing local buckets (let's call them bucketsB) - clusterC in regionC containing local buckets (let's call them bucketsC) I want to replicate bucketsA on clusterB, bucketsB on clusterC and bucketsC on clusterA. I am designing my setup and wonder what to do, realms wise: - have one global realm initially created on clusterA and pulled on clusterB and C, one zonegroup named "country" and 3 zones in it, one for each region. And then setup sync policies to replicate buckets according to my needs - have a realm per cluster, that is pulled by the cluster in charge to replicate the data (e.g: clusterB pulls the realm of clusterA), one zonegroup named "country" and two zones in it: one with the local data and the other one "remote" replicating the data, and using sync policies to replicate buckets according to my needs If you have any advice on which setup is best that would be great. I wonder, for the first option, what does happen if clusterA (who originally created the realm) happened to die. Can another cluster becomes the "realm master" (since they previously pulled the realm), or does the realm is completely lost ? If the realm is recoverable, does rook handles this procedure ? Best regards, Nate

1 0

Grafana dashboards is missing data
by Sake Ceph 10 Sep '24

10 Sep '24

After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages. I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus. Kind regards, Sake

6 16

CEPH monitor slow ops
by Jan Marek 10 Sep '24

10 Sep '24

Hello, we have CEPH cluster with 144 NVMe "disks", "background" network is RoCE. CEPH cluster is version 18.2.2 and is installed via CEPH orchestrator cephadm, container daemon is podman. OS is Debian bookworm, podman is in the version 4.3.1+ds1-8+b1, now we are installed version 4.3.1+ds1-8+deb12u1 (standard Debian bookworm package) We have sometimes problem with the laggy OSD. This Sunday we had problem with osd.25. It was laggy, some other OSDs reported it to MON, MON dismissed it from crush map, but osd.25 complaint about it, MON tried to add it again to the crush map - but this behavior caused, that whole CEPH cluster was laggy and had slow ops, which was a "kiss of death" to some ProxMox VMs :-(. When I restarted osd.25 daemon, after a couple minutes cluster goes back to HEALTHY state. I have prepared logs from 3 machines: c2-dc1-elected-mon-osd.25.txt - machine c2-dc1 had elected MON and 12 OSDs, same of them complait's about osd.25. c1-dc1-mon-osd.25.txt - machine with another MON and 12 OSDs c4-dc2-osd.25.txt - machine with osd.25 daemon (and other 11 OSDs) Problem was, that the first laggy daemon occured around 6 a.m., but problem insisted up to cca 4:20 p.m., when I've restarted osd.25. There is a question, if will be better to restart laggy OSD daemon by ceph orchestrator? For example: If OSD daemons from 3 host (now it is from 2 hosts by default) and after 20s reported some OSD as a dead, MON give a command to mgr to restart this OSD? Somethink like STONITH in corosync/pcs cluster ( https://en.wikipedia.org/wiki/STONITH )? Logs are very big (around 23MB tar.bz2), I've prepared it here: https://hazard.jcu.cz/osd.25.tar.bz2 We thinking about some problem with network, but on the same machine, there is another 11 OSDs and they parformed well. This is not the first occurence of this problem, when it happened the first time, we tried to reload whole server, but then we found, that reload of OSD container is enough... Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

1 0

Mon is unable to build mgr service
by Jorge Ventura 09 Sep '24

09 Sep '24

I am trying to configure ceph for the first time manually. I followed the instructions and at this point I have only ceph-mon and ceph-mgr installed. Here is my ceph status: root@ceph-n1:~# ceph status > cluster: > id: a93114e4-b0af-4b56-b019-0900310a14f8 > health: HEALTH_WARN > OSD count 0 < osd_pool_default_size 3 > > services: > mon: 1 daemons, quorum mon-n1 (age 10m) > > * mgr: mgr-n1(active, since 9m)* osd: 0 osds: 0 up, 0 in > > data: > pools: 0 pools, 0 pgs > objects: 0 objects, 0 B > usage: 0 B used, 0 B / 0 B avail > pgs: For a reason that I do not understand I am getting this message in syslog every second: *Sep 05 21:05:18 ceph-n1 ceph-mon[1089]: 2024-09-05T21:05:18.636+0000 > 7ffae4fa6640 -1 mon.mon-n1@0(leader) e2 get_authorizer failed to build mgr > service session_auth_info (22) Invalid argument* I don't know if this is something about caps and at this point I have no idea what this problem is. -- Ventura

2 1

Somehow throotle recovery even further than basic options?
by Szabo, Istvan (Agoda) 09 Sep '24

09 Sep '24

Hi, 4 years ago we've created our cluster with all disks 4osds (ssds and nvme disks) on octopus. The 15TB SSDs still working properly with 4 osds but the small 1.8T nvmes with the index pool not. Each new nvme osd adding to the existing nodes generates slow ops with scrub off, recovery_op_priority 1, backfill and recovery 1-1. I even turned off all index pool heavy sync mechanism but the read latency still high which means recovery op pushes it even higher. I'm trying to somehow add resource to the cluster to spread the 2048 index pool pg (in replica 3 means 6144pg index pool) but can't make it more gentle. The balancer is working in upmap with max deviation 1. Have this script from digitalocean https://github.com/digitalocean/pgremapper, is there anybody tried it before how is it or could this help actually? Thank you the ideas. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 7