December 2024 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

constant increase in osdmap epoch
by Frank Schilder 15 Apr '25

15 Apr '25

Hi all, we observe a problem that has been reported before, but I can't find a resolution. This is related to an earlier thread "failed to load OSD map for epoch 2898146, got 0 bytes" (https://www.spinics.net/lists/ceph-users/msg84485.html) We run an octopus latest cluster and observe a constant increase in osdmap epoch every few seconds. There is no change in the contents between two successive epochs: # diff map.3075085.txt map.3075086.txt 1c1 < epoch 3075085 --- > epoch 3075086 4c4 < modified 2024-11-18T15:38:45.512100+0100 --- > modified 2024-11-18T15:38:47.858092+0100 This is exactly what others reported too, for example, "steady increasing of osd map epoch since octopus" (https://www.spinics.net/lists/ceph-users/msg69443.html) Its a real problem since it dramatically shortens the time window an OSD can be down before its latest OSD map is purged from the cluster. This, in turn, leads to serious follow-up problems with OSD restart as reported in the thread I'm referring to at the beginning. Related to that I also see the mgrs increasing the pgmap version constantly every 2 seconds. However, I believe this is intentional. I don't see this redundant pgp_num_actual setting by the mgrs reported here: https://tracker.ceph.com/issues/51433 . I can't find a resolution anywhere. Any help would be very much appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 7

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

RGW multisite metadata sync issue
by Vahideh Alinouri 11 Mar '25

11 Mar '25

Hi guys, My Ceph release is Quincy 17.2.5. I need to change the master zone to decommission the old one and upgrade all zones. I have separated the client traffic and sync traffic in RGWs, meaning there are separate RGW daemons handling the sync process. I encountered an issue when trying to sync one of the zones in the zonegroup. The data sync is proceeding fine, but I have an issue with the metadata sync. It gets stuck behind on a shard. Here is the output from radosgw-admin sync status: metadata sync syncing full sync: 1/64 shards full sync: 135 entries to sync incremental sync: 63/64 shards metadata is behind on 1 shard behind shards: [0] In the RGW log, I see this error: 2024-12-15T21:30:59.641+0000 7f6dff472700 1 beast: 0x7f6d2f1cf730: 172.19.66.112 - s3-cdn-user [15/Dec/2024:21:30:59.641 +0000] "GET /admin/log/?type=data&id=56&marker=00000000000000000000%3A00000000000000204086&extra-info=true&rgwx-zonegroup=7c01d60f-88c6-4192-baf7-d725260bf05d HTTP/1.1" 200 44 - - - latency=0.000000000s 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]: ERROR: failed to list omap keys, status=-2 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2 I’ve tried the following steps: - Changed the PG number of the metadata pool to force a rebalance, but everything was fine. - Ran metadata sync init and tried to run it again. - Restarted RGW services in both the zone and the master zone. - Created a user in the master zone to ensure metadata sync works, which was successful. - Checked OSD logs but didn’t see any specific errors. - Attempted to list metadata in the pool using rados ls -p s3-cdn-dc07.rgw.meta, but got an empty result. - Compared the code for listing OMAP keys between Quincy and Squid versions; there were no specific changes. I’m looking for any advice or suggestions to resolve this issue.

3 6

Move block.db to new ssd
by Roland Giesler 11 Mar '25

11 Mar '25

I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions. The new 4TB SSD's I have split into volumes with: # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 & osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this: # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami I then take osd4 out of the cluster and stop it. Then I do this: # ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami Thereafter I do this to load the changes: # ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 --path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami After that the configuration change has reverted to what it was before. I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 What am I missing to get these changes to be permanent? regards Roland

6 13

When 18.2.5 will be released?
by wojiaowugen 04 Mar '25

04 Mar '25

Hi, everyone, It's an honor to ask questions here. can I ask when 18.2.5 will be released?

2 1

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

ceph-14.2.22 OSD crashing - PrimaryLogPG::hit_set_trim on unfound object
by Nikola Ciprich 29 Jan '25

29 Jan '25

Hello dear ceph users and developers, today, we've hit issue on one of our legacy clusters running 14.2.22. we've apparently lost two objects containing cache tier hit_set history. when hit_set_trim hits missing object, it causes OSD to crash: ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x141) [0x55714332faad] 2: (()+0x4e8ca8) [0x55714332fca8] 3: (PrimaryLogPG::hit_set_trim(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >&, unsigned int)+0xcd6) [0x557143626136] 4: (PrimaryLogPG::hit_set_remove_all()+0x2d5) [0x5571436264b5] 5: (PrimaryLogPG::on_activate()+0x55a) [0x5571436360aa] 6: (PG::RecoveryState::Active::react(PG::AllReplicasActivated const&)+0x130) [0x557143512b50] 7: (boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::react_impl(boost::statechart:: 8: (boost::statechart::simple_state<PG::RecoveryState::Activating, PG::RecoveryState::Active, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na 9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x15d) [0x55714353c7ed] 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x211) [0x5571434555d1] 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x52) [0x557143723352] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x582) [0x557143443122] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3eb) [0x557143ac2f4b] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x557143ac5d30] 15: (()+0x7ea5) [0x7f22383e5ea5] 16: (clone()+0x6d) [0x7f22372a8b0d] I've found old report of similar problem here: https://www.spinics.net/lists/ceph-users/msg48660.html however using the same patch: diff -Naur ceph-14.2.22/src/osd/PrimaryLogPG.cc ceph-14.2.22-fix-hit_set_trim/src/osd/PrimaryLogPG.cc --- ceph-14.2.22/src/osd/PrimaryLogPG.cc 2024-12-29 12:34:17.000000000 +0100 +++ ceph-14.2.22-fix-hit_set_trim/src/osd/PrimaryLogPG.cc 2024-12-29 19:42:31.527632776 +0100 @@ -13932,11 +13932,13 @@ updated_hit_set_hist.history.pop_front(); ObjectContextRef obc = get_object_context(oid, false); - ceph_assert(obc); - --ctx->delta_stats.num_objects; - --ctx->delta_stats.num_objects_hit_set_archive; - ctx->delta_stats.num_bytes -= obc->obs.oi.size; - ctx->delta_stats.num_bytes_hit_set_archive -= obc->obs.oi.size; + //ceph_assert(obc); + if(obc){ + --ctx->delta_stats.num_objects; + --ctx->delta_stats.num_objects_hit_set_archive; + ctx->delta_stats.num_bytes -= obc->obs.oi.size; + ctx->delta_stats.num_bytes_hit_set_archive -= obc->obs.oi.size; + }; } } causes OSD to crash later: /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: In function 'virtual void PrimaryLogPG::op_applied(const eversion_t&)' thread 7f91e146d700 time 2024-12-29 19:58:13.880029 /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: 10457: FAILED ceph_assert(applied_version <= info.last_update) /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: In function 'virtual void PrimaryLogPG::op_applied(const eversion_t&)' thread 7f91df469700 time 2024-12-29 19:58:13.882629 /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: 10457: FAILED ceph_assert(applied_version <= info.last_update) ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x122) [0x55e07e95b815] 2: (()+0x49899a) [0x55e07e95b99a] 3: (PrimaryLogPG::op_applied(eversion_t const&)+0x1f2) [0x55e07eb9fef2] 4: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x720) [0x55e07ed411b0] 5: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xdc5) [0x55e07eba0cd5] 6: (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >)+0x9b) [0x55e07eba2acb] 7: (PrimaryLogPG::hit_set_remove_all()+0x2e5) [0x55e07ebdcbf5] 8: (PrimaryLogPG::on_pool_change()+0xeb) [0x55e07ebddc7b] 9: (PG::handle_advance_map(std::shared_ptr<OSDMap const>, std::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, int, std::vector<int, std::allocator<int> >&, int, PG::RecoveryCtx*)+0x34f) [0x55e07eaf8c4f] 10: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x2f6) [0x55e07ea482e6] 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x55e07ea50a24] 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x4e) [0x55e07ecde60e] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xfea) [0x55e07ea5527a] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x415) [0x55e07f032345] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55e07f034740] 16: (()+0x7ea5) [0x7f9203024ea5] 17: (clone()+0x6d) [0x7f9201ee7b0d] unfortunately here I don't have any clue how to proceed further. I've increased hit_set_count to 32 and hit_set_period to 36000 thus hopefully gaining some time (now OSD seem to be running). Any ideas on how to safely get from this mess? I can't easily get rid of cache tier now since it's used by running VMs and what is worse, I'm not sure I won't hit same problem when deleting cache pool anyways - OSDs are shared by cache pool and NVME data pool so it's pretty uncomfortable situation :-( I'm using bluestore for all OSDs, so can't even try copying other hit_set within filestore.. I'll be very grateful for any help with best regards nikola ciprich

2 1

squid 19.2.1 RC QE validation status
by Yuri Weinstein 17 Jan '25

17 Jan '25

Details of this release are summarized here: https://tracker.ceph.com/issues/69234#note-1 Release Notes - TBD LRC upgrade - TBD Gibba upgrade -TBD Please provide tracks for failures so we avoid duplicates. Seeking approvals/reviews for: rados - Radek, Laura rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume

14 33

Modify or override ceph_default_alerts.yml
by Devin A. Bougie 14 Jan '25

14 Jan '25

Hi, All. We are using cephadm to manage a 19.2.0 cluster on fully-updated AlmaLinux 9 hosts, and would greatly appreciate help modifying or overriding the alert rules in ceph_default_alerts.yml. Is the best option to simply update the /var/lib/ceph/<cluster_id>/home/ceph_default_alerts.yml file? In particular, we’d like to either disable the CephPGImbalance alert or change it to calculate averages per-pool or per-crush_rule instead of globally as in [1]. We currently have PG autoscaling enabled, and have two separate crush_rules (one with large spinning disks, one with much smaller nvme drives). Although I don’t believe it causes any technical issues with our configuration, our dashboard is full of CephPGImbalance alerts that would be nice to clean up without having to create periodic silences. Any help or suggestions would be greatly appreciated. Many thanks, Devin [1] https://github.com/rook/rook/discussions/13126#discussioncomment-10043490

3 14

Snaptriming speed degrade with pg increase
by Szabo, Istvan (Agoda) 14 Jan '25

14 Jan '25

Hi, When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04) -osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2 We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours. Is there any good solution that we are missing to fix this? On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still. Thank you

4 9

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Experimental upgrade of a Cephadm-managed Squid cluster to Ubuntu Noble (walk-through and RFC)
by Florian Haas 09 Jan '25

09 Jan '25

Hi everyone, as part of keeping our Ceph course updated, we recently went through the *experimental* process of upgrading a Cephadm-managed cluster from Ubuntu Jammy to Noble. Note that at this point there are no community-built Ceph packages that are available for Noble, though there *are* packages available directly from the Ubuntu mirrors. We figured that this shouldn't make too big of a difference, considering the Ceph deployment is all containerised anyway. We did eventually manage to successfully upgrade the cluster, with minimal disruption. That said, DO NOT consider the rest of this message to be an instruction for how to do this in production. Rather, it's meant to highlight some quirks that — I think — still need ironing out. Again: for the time being, DO NOT RUN THIS IN PRODUCTION, please and thank you. :) What's included here is a walkthrough of the process, and I would ask for feedback on this walkthrough. Specifically, if someone notices an error or omission in this process, please point them out. Each one of these steps needs to be repeated for every node in the cluster. It assumes a working Cephadm-managed cluster on Ubuntu Jammy with Podman, running Ceph Squid. All commands should be run as root. == 0. If the host you're about to upgrade is a MON, ensure that your orchestrator placement policy allows for the mon service to move to a different host. (This is to ensure that throughout your OS upgrade, you always have 3 MONs available.) 1. Disable orchestrator scheduling for the affected node: "ceph orch host label add <host> _no_schedule". 2. Wait for services (except OSDs) to be redeployed. 3. Check with "ceph orch host ok-to-stop <host>" to see whether the host reports "presumed safe to stop". (May require failing a mgr instance.) 4. Set the noout flag for that host: "ceph osd set-group noout <host>". 5. Check with "ceph health detail" to see if it says "host <host> has flags noout". 6. Shell into the host. 7. Make sure the host is fully updated on Jammy: "apt update && apt upgrade -y". Reboot the host if necessary. 8. Stop all remaining Ceph services (at this point, this should only be OSDs) with "systemctl stop ceph.target". 9. If your host has the cephadm package installed from a Ceph community repo, manually create the "/var/lib/cephadm" directory. This is required by the Ubuntu cephadm package, which during the OS upgrade will be cross-upgraded from a different package source. Failure to create this directory will cause this package upgrade to fail. I presume that this issue will go away whenever the Ceph community repos start including packages for Noble. 10. Run "do-release-upgrade" confirming all steps *except* the automated reboot at the end (opt out of that). 11. Remove the MongoDB_Compass profile from the /etc/apparmor.d directory. (If left in place, it will cause CEPHADM_REFRESH_FAILED errors.) 12. Remove any legacy ceph-osd packages, if installed on the host: "apt remove ceph-osd". (Failure to do so will keep your OSD containers from starting.) 13. Reboot the host. 14. Re-enable orchestrator scheduling with "ceph orch host label rm <host> _no_schedule". 15. Unset the noout flag for the host: "ceph osd unset-group noout <host>". 16. Check overall cluster health with "ceph -s" and "ceph health detail" before proceeding to the next node. == Once carried out for each Cephadm-managed Ubuntu node, you should have a reasonable expection of having a working Ceph/Noble test cluster. Again: do not run this in production, however, feedback on the above steps is much appreciated. Please let us know your thoughts. Cheers, Florian

3 7

Random ephemeral pinning, what happens to sub-tree under pin root dir
by Frank Schilder 09 Jan '25

09 Jan '25

Dear all, I have a question about random ephemeral pinning that I can't find an answer in the docs to. Question first and some background later. Docs checked for any version from octopus up to latest. Our version for applying random ephemeral pinning is pacific. What I would like to configure on a subtree is this: Enable random ephemeral pinning at the root of the tree, say, /cephfs/root: setfattr -n ceph.dir.pin.random -v 0.0001 /cephfs/root Will this have the following effects: A) The root of the tree /cephfs/root is ephemerally pinned to a rank according to a consistent hash of its inode number. B) Any descendant sub-directory may be ephemerally pinned 0.01 percent of the time to a rank according to a consistent hash of its inode number. The important difference to the docs is point A. I don't want to have *any* subdir under the root /cephfs/root *not pinned* to an MDS. The docs only talk about descendant sub-dirs, but the root is here important too because if it is not pinned it will create a large number of unpinned dirfrags that float around with expensive exportdir operations that pinning is there to avoid in the first place. My questions are: 1) What does random ephemeral pinning do to the sub-tree root? Is it pinned or not? 2) If it doesn't pin the root, does this work as intended or will it pin everything to rank 1: setfattr -n ceph.dir.pin.random -v 0.001 /cephfs/root setfattr -n ceph.dir.pin -v 1 /cephfs/root Background: We use cephfs as a home file system for an HPC cluster and are in exactly in the situation of the example for distributed ephemeral pinning (https://docs.ceph.com/en/latest/cephfs/multimds/?highlight=ephemeral+pin#se…) *except* that the home-dirs of users differ dramatically in size. We were not pinning at first and this lead to our MDSes go crazy due to the load balancer moving dirfrags around all the time. This "load balancing" was itself responsible for 90-95% (!!!) of the total MDS load. After moving to octopus we simulated distributed ephemeral pinning with a cron job that assigned home-dirs in a round robin fashion to MDS ranks that had the least pins. This immediately calmed down our entire MDS cluster (8 active and 4 stand-by) and user experience improved dramatically. MDS load dropped from 125-150% (idle load!!) to about 20-25% per MDS and memory usage stabilized as well. The easy way forward would be to replace our manual distribution with distributed ephemeral pinning of /home (in octopus this was experimental, after our recent upgrade to pacific we can use the built-in distribution). However, as stated above, the size of home-dirs differs to a degree that chunking up the file system into equally-sized sub-dir trees would be better than distributing entire home dir trees over ranks. Users with very large sub-trees might get spread out over more than one rank. This is what random ephemeral pinning seems to be there for and I would like to chunk our entire filesystem up into sub-trees of size 10000-100000 directory fragments and distribute these over the MDSes. However, this only works if the root and with it the first sub-tree is also pinned. Note that this is not a problem with distributed ephemeral pinning, because this policy pins *all* *immediate* children of the pin root and, therefore, does not create free-floating directory fragments. I would be grateful if someone could shed light on the question whether or not the pin root of random ephemeral pinning is itself pinned or not. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

How to configure prometheus password in ceph dashboard.
by s.dhivagar.cse＠gmail.com 07 Jan '25

07 Jan '25

We have configured basic authentication in Prometheus, so how to set username and password in ceph dashboard.

3 2

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

recovery a downed/inaccessible pg
by Nick Anderson 02 Jan '25

02 Jan '25

Hello ceph user-community, We have a reef (18.2.2) cluster with an erasure coded pool of k=8, m=4 that has a pg that is currently down with the following: ``` WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg incomplete pg 2.699 is remapped+incomplete, acting [NONE,NONE,NONE,NONE,NONE,353,488,333,408,282,167,145] (reducing pool i1-sea-rbd-01-data min_size from 9 may help; search ceph.com/docs for 'incomplete') ``` Timeline leading up to our current state noted above. We had an issue with a host that had a flappy network and during our efforts to restore the network some prolonged OSD heartbeats not reachable seems to have caused two OSDs to go down/out. The OSD containers were stuck in a restart loop and eventually were auto-out'd from the cluster. Restarting the affected OSD containers had no effect and the pg was stuck in a "down+remapped state." ``` [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg down pg 2.699 is down+remapped, acting [NONE,NONE,NONE,NONE,613,353,488,333,408,282,167,145] ``` Looking over the `pg query 2.699` we could the following: ``` "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 38, 120 ], "peering_blocked_by": [ { "osd": 38, "current_lost_at": 90605, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 120, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, ``` With that output and reading ceph documentation[1], we decided to mark osd.38 as lost (`ceph osd lost 38`). Within a couple minutes, a 3rd OSD went down in osd.613 and exhibited the same restart loop behavior. Leaving us with the following state from a ` ceph pg query 2.699` ``` "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 38, 120, 613 ], "peering_blocked_by": [ { "osd": 38, "current_lost_at": 90605, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 120, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 613, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] }, { "name": "Started", "enter_time": "2024-12-21T17:13:18.435769+0000" } ], "agent_state": {} ``` After sleeping on it and letting the cluster recover from another failed OSD, we decided to mark all three as lost (osd.38, osd.120, and osd.613). Unfortunately, this caused two additional OSDs (osd.226 and osd.339) to go down. This changed the pg state from `down+remapped` to`remapped+incomplete`. We now have 5 down/out OSDs and are wondering if there is anyone to recover this PG? This one PG has caused our 1.6 PiB to be inaccessible.. We've gleaned some documentation[2] involving exporting the affected pg data with ceph-objectstore-tool and importing into a new OSD. Is this our only route to possibly recover this pg and its data? The strange thing is the OSDs underlying disk seems fine in that the ceph volume-group/LVM are intact. There are 0 issues with the disks from what I can glean or any indication of disk issues (the last 3 disks that were outed were part of a successful recovery/backfill with no issues). I tested the two original outed OSDs and the underlying disks all pass smartctl short tests and have seen no indication of hardware issues with these spinning disks. Looking over the ceph documentation, it looks like there is a newer tool that is cephadm/bluestore friendly in ceph-bluestore-tool[3] that does the same job as ceph-objectstore-tool? We are hoping there is a way to recover this PG and appreciate any advice. Currently our production cluster is down and are looking for recovery avenues. - Nick Links: [1] https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#pla… [2] https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor… [3] https://docs.ceph.com/en/reef/man/8/ceph-bluestore-tool/

3 2

disregard Re: Missing Release file? (cephadm add-repo —release squid fails on Ubuntu 24.04.1 LTS)
by Christian Kuhtz 30 Dec '24

30 Dec '24

Ooops, I can’t read. 24.04.1 LTS isn’t on the OS list, only 22.04. :-( On Dec 30, 2024 at 13:29:39, Christian Kuhtz <christian(a)kuhtz.com> wrote: > Hello, > > Host is an Ubuntu 24.04.1 LTS server image. Following instructions at > https://docs.ceph.com/en/squid/cephadm/install/ for cephadm install with > curl. The command > > ./cephadm add-repo --release squid > > fails because it can’t find the Release file and is getting a 404 from the > repo. Full output is below. It does install /etc/apt/sources.d/ceph.list > with the line below > > deb https://download.ceph.com/debian-squid/ noble main > > before failing. If I modify ceph.list with bookworm instead of noble, apt > update succeeds. If this is tested on Ubuntu 24.01.4 LTS as > https://docs.ceph.com/en/squid/start/os-recommendations/ suggests, > shouldn’t this work out of the box? > > Thanks, > Christian > > # lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 24.04.1 LTS > Release: 24.04 > Codename: noble > # /cephadm add-repo --release squid > Installing repo GPG key from https://download.ceph.com/keys/release.gpg... > Installing repo file at /etc/apt/sources.list.d/ceph.list... > Updating package list... > Non-zero exit code 100 from apt-get update > apt-get: stdout Hit:1 http://security.ubuntu.com/ubuntu noble-security > InRelease > apt-get: stdout Ign:2 https://download.ceph.com/debian-squid noble > InRelease > apt-get: stdout Err:3 https://download.ceph.com/debian-squid noble Release > apt-get: stdout 404 Not Found [IP: 158.69.68.124 443] > apt-get: stdout Hit:4 http://us.archive.ubuntu.com/ubuntu noble InRelease > apt-get: stdout Hit:5 http://us.archive.ubuntu.com/ubuntu noble-updates > InRelease > apt-get: stdout Hit:6 http://us.archive.ubuntu.com/ubuntu noble-backports > InRelease > apt-get: stdout Reading package lists... > apt-get: stderr E: The repository 'https://download.ceph.com/debian-squid > noble Release' does not have a Release file. > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File "/root/./cephadm/__main__.py", line 5579, in <module> > File "/root/./cephadm/__main__.py", line 5567, in main > File "/root/./cephadm/__main__.py", line 4575, in command_add_repo > File "/root/./cephadm/cephadmlib/packagers.py", line 192, in add_repo > File "/root/./cephadm/cephadmlib/packagers.py", line 213, in update > File "/root/./cephadm/cephadmlib/call_wrappers.py", line 307, in > call_throws > RuntimeError: Failed command: apt-get update: E: The repository ' > https://download.ceph.com/debian-squid noble Release' does not have a > Release file. >

1 0

Missing Release file? (cephadm add-repo —release squid fails on Ubuntu 24.04.1 LTS)
by Christian Kuhtz 30 Dec '24

30 Dec '24

Hello, Host is an Ubuntu 24.04.1 LTS server image. Following instructions at https://docs.ceph.com/en/squid/cephadm/install/ for cephadm install with curl. The command ./cephadm add-repo --release squid fails because it can’t find the Release file and is getting a 404 from the repo. Full output is below. It does install /etc/apt/sources.d/ceph.list with the line below deb https://download.ceph.com/debian-squid/ noble main before failing. If I modify ceph.list with bookworm instead of noble, apt update succeeds. If this is tested on Ubuntu 24.01.4 LTS as https://docs.ceph.com/en/squid/start/os-recommendations/ suggests, shouldn’t this work out of the box? Thanks, Christian # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 24.04.1 LTS Release: 24.04 Codename: noble # /cephadm add-repo --release squid Installing repo GPG key from https://download.ceph.com/keys/release.gpg... Installing repo file at /etc/apt/sources.list.d/ceph.list... Updating package list... Non-zero exit code 100 from apt-get update apt-get: stdout Hit:1 http://security.ubuntu.com/ubuntu noble-security InRelease apt-get: stdout Ign:2 https://download.ceph.com/debian-squid noble InRelease apt-get: stdout Err:3 https://download.ceph.com/debian-squid noble Release apt-get: stdout 404 Not Found [IP: 158.69.68.124 443] apt-get: stdout Hit:4 http://us.archive.ubuntu.com/ubuntu noble InRelease apt-get: stdout Hit:5 http://us.archive.ubuntu.com/ubuntu noble-updates InRelease apt-get: stdout Hit:6 http://us.archive.ubuntu.com/ubuntu noble-backports InRelease apt-get: stdout Reading package lists... apt-get: stderr E: The repository 'https://download.ceph.com/debian-squid noble Release' does not have a Release file. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/./cephadm/__main__.py", line 5579, in <module> File "/root/./cephadm/__main__.py", line 5567, in main File "/root/./cephadm/__main__.py", line 4575, in command_add_repo File "/root/./cephadm/cephadmlib/packagers.py", line 192, in add_repo File "/root/./cephadm/cephadmlib/packagers.py", line 213, in update File "/root/./cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: apt-get update: E: The repository ' https://download.ceph.com/debian-squid noble Release' does not have a Release file.

1 0

download.ceph.com TLS cert expired 29/12/2024
by Zakhar Kirpichenko 30 Dec '24

30 Dec '24

Hi! Looks like the TLS certificate at download.ceph.com expired on 29/12/2024: Certificate chain 0 s:CN = download.ceph.com i:C = US, O = Let's Encrypt, CN = R11 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Sep 30 23:02:29 2024 GMT; NotAfter: Dec 29 23:02:28 2024 GMT This affects multiple things, such as for example updating package repositories on multiple systems. Perhaps someone could please look into this. Best regards, Zakhar

3 4

cephadm rollout behavior and post adoption issues
by Nima AbolhassanBeigi 29 Dec '24

29 Dec '24

Hi everybody, I'm fairly new to cephadm. I'm trying to get some hands-on experience. I have a test cluster consisting of: 3 Monitor/Manager nodes. 6 OSD nodes 3 RGW nodes Pacific version containerized deployment using: quay.io/ceph/daemon:v6.0.11-stable-6.0-pacific-centos-stream8 I've adopted this cluster using the cephadm adopt playbook in the ceph-ansible project. infrastructure-playbooks/cephadm-adopt.yml I set the default container_image to a certain image and I checked the configuration database and it was set to the correct image. The adoption process for monitor, manager, and RGW daemons (redeployment in the case of RGWs) used the value for container_image, but the OSDs used the old image they were using before they were adopted. container_image value: quay.io/ceph/ceph:v16.2.13 I was wondering what could be the reason for the image not changing for OSDs. After the adoption, I wanted to mount the client.admin.keyring to monitor daemons. I used the extra_container_args service specification and added the volume. After I applied the mon service spec file to the cluster using ceph orch apply -i <filename>, I witnessed something odd. I read that cephadm has a rollout-like behavior for redeploying daemons. But the delay between the redeploys was 4 seconds and there was a period when all the monitor daemons were in the starting state. Is there any mechanism to increase the delay between the redeployments? Is there a way to enforce the redeployments to only 1 daemon? I've provided some logs, at the end of the message, that I think can be helpful. Thanks in advance. Regards Nima AbolhassanBeigi NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID mon.stg-pacific-mon1 stg-pacific-mon1 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon2 stg-pacific-mon2 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon3 stg-pacific-mon3 starting - - - 6144M <unknown> <unknon w> Every 1.0s: ceph orch ps --service_name=mon --refresh stg-pacific-mon1: Sun Dec 29 15:52:15 2024 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID mon.stg-pacific-mon1 stg-pacific-mon1 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon2 stg-pacific-mon2 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon3 stg-pacific-mon3 starting - - - 6144M <unknown> <unknow n> Every 1.0s: ceph orch ps --service_name=mon --refresh stg-pacific-mon1: Sun Dec 29 15:52:41 2024 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.stg-pacific-mon1 stg-pacific-mon1 running (37s) 2s ago 6d 30.2M 6144M 16.2.13 9e29efbb67d5 367a7f3bac31 mon.stg-pacific-mon2 stg-pacific-mon2 running (33s) 2s ago 11d 30.5M 6144M 16.2.13 9e29efbb67d5 43828e2453da mon.stg-pacific-mon3 stg-pacific-mon3 running (29s) 2s ago 11d 24.8M 6144M 16.2.13 9e29efbb67d5 d698a809370f 2024-12-29T12:18:16.974710+0000 mgr.stg-pacific-mon1 (mgr.173193) 322363 : cephadm [INF] Saving service mon spec with placement count:7;label:mons 2024-12-29T12:19:44.140240+0000 mgr.stg-pacific-mon1 (mgr.173193) 322545 : cephadm [INF] Saving service mon spec with placement count:3;label:mons 2024-12-29T12:19:47.497916+0000 mgr.stg-pacific-mon1 (mgr.173193) 322555 : cephadm [INF] Redeploying mon.stg-pacific-mon1, (container cli args changed) . . . 2024-12-29T12:19:47.507683+0000 mgr.stg-pacific-mon1 (mgr.173193) 322556 : cephadm [INF] Deploying daemon mon.stg-pacific-mon1 on stg-pacific-mon1 2024-12-29T12:19:50.970499+0000 mgr.stg-pacific-mon1 (mgr.173193) 322562 : cephadm [INF] Redeploying mon.stg-pacific-mon2, (container cli args changed) . . . 2024-12-29T12:19:50.982244+0000 mgr.stg-pacific-mon1 (mgr.173193) 322563 : cephadm [INF] Deploying daemon mon.stg-pacific-mon2 on stg-pacific-mon2 2024-12-29T12:19:55.774118+0000 mgr.stg-pacific-mon1 (mgr.173193) 322573 : cephadm [INF] Redeploying mon.stg-pacific-mon3, (container cli args changed) . . . 2024-12-29T12:19:55.782550+0000 mgr.stg-pacific-mon1 (mgr.173193) 322574 : cephadm [INF] Deploying daemon mon.stg-pacific-mon3 on stg-pacific-mon3 2024-12-29T12:21:56.620507+0000 mgr.stg-pacific-mon1 (mgr.173193) 322830 : cephadm [INF] Saving service mon spec with placement count:3;label:mons 2024-12-29T12:22:02.198770+0000 mgr.stg-pacific-mon1 (mgr.173193) 322844 : cephadm [INF] Redeploying mon.stg-pacific-mon1, (container cli args changed) . . . 2024-12-29T12:22:02.210484+0000 mgr.stg-pacific-mon1 (mgr.173193) 322845 : cephadm [INF] Deploying daemon mon.stg-pacific-mon1 on stg-pacific-mon1 2024-12-29T12:22:05.654926+0000 mgr.stg-pacific-mon1 (mgr.173193) 322851 : cephadm [INF] Redeploying mon.stg-pacific-mon2, (container cli args changed) . . . 2024-12-29T12:22:05.667782+0000 mgr.stg-pacific-mon1 (mgr.173193) 322852 : cephadm [INF] Deploying daemon mon.stg-pacific-mon2 on stg-pacific-mon2 2024-12-29T12:22:10.332756+0000 mgr.stg-pacific-mon1 (mgr.173193) 322862 : cephadm [INF] Redeploying mon.stg-pacific-mon3, (container cli args changed) . . . 2024-12-29T12:22:10.344298+0000 mgr.stg-pacific-mon1 (mgr.173193) 322863 : cephadm [INF] Deploying daemon mon.stg-pacific-mon3 on stg-pacific-mon3

1 0

Tpm2 in squid
by Ehsan Golpayegani 28 Dec '24

28 Dec '24

Hi ceph users, Has anyone successfully installed Ceph Squid with TPM2 enabled on an OSD? I'm encountering an error during installation: "stderr_lines": ["Error EINVAL: Failed to validate OSD spec "<unnamed>": Feature `tpm2` is not supported"]. I've also attempted to prepare the disks with TPM2 before adding them to the cluster, but I receive the following error: "/usr/bin/podman: stderr ceph-volume lvm prepare: error: unrecognized arguments: --with-tpm". For reference, my cluster is deployed via Cephadm and I'm using Squid 19.2.0. Any guidance would be greatly appreciated. Thanks, Ehsan -------------- Kind Regards <http://about.me/ehsan310?promo=email_sig> Ehsan Golpayegani [image: https://]www.ehsan.pw <https://ehsan.pw?promo=email_sig&utm_source=product&utm_medium=email_sig&ut…> SAVE PAPER - THINK BEFORE YOU PRINT!

2 3

Pls stuck in snaptrim
by bellow.oar_0t＠icloud.com 28 Dec '24

28 Dec '24

Greetings, I am having trouble with (the same) PGs stuck in snaptrrim and snaptrrim_wait for over half a year now. I don’t know how to debug/fix that, I tried bringing down the cluster, rebooted everything, failed the mds, mgr, upgraded to a new mentor version, upgraded to a new major version, all to no avail. Ceph version is 19.2.0 # ceph pg dump|grep snaptrim 9.7d 123817 0 0 0 0 479247022395 0 0 4375 346 4375 active+clean+snaptrim_wait 2024-10-13T00:00:12.050817+0000 134416'449058 134416:4144441 [9,14,0,6] 9 [9,14,0,6] 9 134054'434172 2024-10-11T01:26:12.751712+0000 134054'434172 2024-10-11T01:26:12.751712+0000 1256 138366 user requested deep scrub scheduled @ 2024-12-27T08:47:58.847491+0000 114488 0 9.74 123271 0 0 0 0 478810964688 0 0 4440 1140 4440 active+clean+snaptrim 2024-10-13T00:02:12.299691+0000 134416'451014 134416:4037366 [9,8,6,14] 9 [9,8,6,14] 9 134049'436191 2024-10-10T09:25:49.744261+0000 134049'436191 2024-10-10T09:25:49.744261+0000 1256 80746 user requested deep scrub scheduled @ 2024-12-27T08:48:59.094636+0000 114101 0 9.71 124359 0 0 0 0 482220121204 0 0 4374 878 4374 active+clean+snaptrim_wait 2024-10-13T00:02:05.517504+0000 134416'454057 134416:4334146 [9,8,7,3] 9 [9,8,7,3] 9 134049'438914 2024-10-10T05:16:04.290745+0000 134049'438914 2024-10-10T05:16:04.290745+0000 1256 65761 user requested deep scrub scheduled @ 2024-12-27T08:48:03.954808+0000 115086 0 9.6b 123942 0 0 0 0 480451675897 0 0 4358 345 4358 active+clean+snaptrim_wait 2024-10-13T00:00:12.540600+0000 134416'450439 134416:4705616 [9,4,0,11] 9 [9,4,0,11] 9 134049'435438 2024-10-10T18:35:30.564215+0000 134049'435438 2024-10-10T18:35:30.564215+0000 1256 113715 user requested deep scrub scheduled @ 2024-12-27T08:47:37.794121+0000 114709 0 9.6a 123870 0 0 0 0 480164531666 0 0 4364 879 4364 active+clean+snaptrim 2024-10-13T00:02:05.652753+0000 134416'456701 134416:4003484 [9,4,0,13] 9 [9,4,0,13] 9 134049'441302 2024-10-10T12:42:55.289279+0000 134049'441302 2024-10-10T12:42:55.289279+0000 1256 92560 user requested deep scrub scheduled @ 2024-12-27T08:48:09.914747+0000 114496 0 9.5f 123879 0 0 0 0 479941485759 0 0 4431 1141 4431 active+clean+snaptrim_wait 2024-10-13T00:00:13.340335+0000 134416'450791 134416:4123574 [9,3,2,6] 9 [9,3,2,6] 9 134054'435775 2024-10-11T01:27:38.427214+0000 134054'435775 2024-10-11T01:27:38.427214+0000 1256 138452 user requested deep scrub scheduled @ 2024-12-27T08:49:04.452674+0000 114542 0 27.19 7017 0 0 0 0 29241743329 0 0 4358 0 4358 active+clean+snaptrim_wait 2024-10-13T00:02:05.652729+0000 134416'5924654 134416:11859363 [9,7,3,4] 9 [9,7,3,4] 9 133910'5800741 2024-10-09T13:55:08.732933+0000 133910'5800741 2024-10-09T13:55:08.732933+0000 1256 10492 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053054+0000 6293 0 27.4 7072 0 0 0 0 29496160608 0 0 4401 0 4401 active+clean+snaptrim_wait 2024-10-13T00:00:13.740349+0000 134416'5920493 134416:12128517 [9,0,14,2] 9 [9,0,14,2] 9 133972'5796609 2024-10-09T12:20:39.099453+0000 133972'5796609 2024-10-09T12:20:39.099453+0000 1256 4833 user requested deep scrub scheduled @ 2024-12-27T08:47:47.863907+0000 6249 0 9.1c 124329 0 0 0 0 482320315530 0 0 4368 878 4368 active+clean+snaptrim_wait 2024-10-13T00:02:05.518217+0000 134416'453725 134416:3954611 [9,7,11,0] 9 [9,7,11,0] 9 134049'438455 2024-10-10T06:40:15.938302+0000 134049'438455 2024-10-10T06:40:15.938302+0000 1256 70803 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053068+0000 114948 0 9.19 123790 0 0 0 0 480077901246 0 0 4411 218 4411 active+clean+snaptrim_wait 2024-10-13T00:00:12.539642+0000 134416'451736 134416:3826333 [9,6,7,1] 9 [9,6,7,1] 9 134049'436829 2024-10-10T21:35:24.519252+0000 134049'436829 2024-10-10T21:35:24.519252+0000 1256 138616 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053063+0000 114595 0 dumped all 9.13 124027 0 0 0 0 480656352004 0 0 4370 0 4370 active+clean+snaptrim_wait 2024-10-13T00:00:06.753470+0000 134416'447134 134416:3903589 [9,14,13,0] 9 [9,14,13,0] 9 134049'431887 2024-10-10T03:55:32.120382+0000 134049'431887 2024-10-10T03:55:32.120382+0000 1256 60916 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053062+0000 114776 0 9.28 123470 0 0 0 0 478607727659 0 0 4355 0 4355 active+clean+snaptrim_wait 2024-10-13T00:02:12.298971+0000 134416'450238 134416:3500480 [9,0,1,3] 9 [9,0,1,3] 9 134054'435412 2024-10-11T01:23:22.978541+0000 134054'435412 2024-10-11T01:23:22.978541+0000 1256 138187 user requested deep scrub scheduled @ 2024-12-27T08:48:20.260350+0000 114182 0 9.37 123338 0 0 0 0 478217011616 0 0 4391 248 4391 active+clean+snaptrim_wait 2024-10-13T00:02:06.016700+0000 134416'448180 134416:3783995 [9,14,7,13] 9 [9,14,7,13] 9 134054'433338 2024-10-11T01:26:24.029875+0000 134054'433338 2024-10-11T01:26:24.029875+0000 1256 138368 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053069+0000 114085 0 9.42 123550 0 0 0 0 479437295594 0 0 4385 415 4385 active+clean+snaptrim_wait 2024-10-13T00:00:13.264210+0000 134416'446881 134416:3886466 [9,7,1,6] 9 [9,7,1,6] 9 134054'431973 2024-10-11T04:24:50.076511+0000 134054'431973 2024-10-11T04:24:50.076511+0000 1256 163177 user requested deep scrub scheduled @ 2024-12-27T08:49:10.053050+0000 114369 0 9.48 123504 0 0 0 0 478672624191 0 0 4370 0 4370 active+clean+snaptrim_wait 2024-10-13T00:00:15.366907+0000 134416'447888 134416:3476913 [9,2,13,7] 9 [9,2,13,7] 9 134049'432764 2024-10-10T18:43:14.290802+0000 134049'432764 2024-10-10T18:43:14.290802+0000 1256 114183 user requested deep scrub scheduled @ 2024-12-27T08:48:59.094642+0000 114094 0 9.4c 123799 0 0 0 0 480058080747 0 0 4367 298 4367 active+clean+snaptrim_wait 2024-10-13T00:00:15.365742+0000 134416'450187 134416:4460709 [9,7,4,0] 9 [9,7,4,0] 9 134049'435272 2024-10-10T12:45:20.588451+0000 134049'435272 2024-10-10T12:45:20.588451+0000 1256 92705 user requested deep scrub scheduled @ 2024-12-27T08:48:09.914749+0000 114623 0 9.4d 124344 0 0 0 0 480982998237 0 0 4349 878 4349 active+clean+snaptrim_wait 2024-10-13T00:02:12.299642+0000 134416'445413 134416:4170713 [9,14,0,7] 9 [9,14,0,7] 9 134049'430041 2024-10-10T18:35:55.877861+0000 134049'430041 2024-10-10T18:35:55.877861+0000 1255 113735 user requested deep scrub scheduled @ 2024-12-27T08:48:59.094640+0000 114850 0 9.5c 123379 0 0 0 0 478151706144 0 0 4377 0 4377 active+clean+snaptrim_wait 2024-10-13T00:02:06.017489+0000 134416'452726 134416:4446619 [9,11,6,13] 9 [9,11,6,13] 9 134049'437743 2024-10-10T18:43:28.133384+0000 134049'437743 2024-10-10T18:43:28.133384+0000 1255 114188 user requested deep scrub scheduled @ 2024-12-27T08:48:59.094632+0000 114239 0 pgs: 206 active+clean 16 active+clean+snaptrim_wait 2 active+clean+snaptrim 1 active+clean+scrubbing+deep Please advice. Daniel

2 1

Re: Ceph Reef 18.2.2 - stucked pgs in not scrubbed and deep-scrubbed in time
by Frédéric Nass 28 Dec '24

28 Dec '24

Hi, Can you try setting osd_scrub_disable_reservation_queuing to 'true' and report back? There's an ongoing bug [1] in Squid that closely matches what you're describing. This issue has been reported to affect multiple Ceph versions as well. Regards, Frédéric. [1] https://tracker.ceph.com/issues/69078 ________________________________ De : Saint Kid <saint8kid(a)gmail.com> Envoyé : samedi 28 décembre 2024 00:03 À : ceph-users(a)ceph.io Objet : [ceph-users] Ceph Reef 18.2.2 - stucked pgs in not scrubbed and deep-scrubbed in time Hello, I have adjusted the scrubbing schedule and manually kick started the process but it appears the warning remains on these 15 PGs. I am thinking of restarting the OSDs which those PGs are and hoping to clear this condition. Any suggestions? Best Regards, Saint Kid _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph Reef 18.2.2 - stucked pgs in not scrubbed and deep-scrubbed in time
by Saint Kid 27 Dec '24

27 Dec '24

Hello, I have adjusted the scrubbing schedule and manually kick started the process but it appears the warning remains on these 15 PGs. I am thinking of restarting the OSDs which those PGs are and hoping to clear this condition. Any suggestions? Best Regards, Saint Kid

1 0

OSD_FULL after OSD Node Failures
by Gerard Hand 27 Dec '24

27 Dec '24

Hi, We recently had problems that meant 3 out of 32 OSD hosts went offline for about 10 minutes. The hosts are now back in the cluster as expected and backfilling going on. However we are seeing a couple of problems. We are seeing: 1. Ceph is flagging a handfull of PGs as backfill_toofull when they aren't. https://tracker.ceph.com/issues/61839. 2. Periodically it generates an OSD_FULL error. Are there any plans to look at resolving bug #61839? When an OSD_FULL error has been reported, the OSD in question has been <75% usage. The OSD wasn't used by any of the PGs reporting backfill_toofull. I currently have the full_ratio set to 0.97 and nearfull_ratio set to 0.87 so the OSDs are nowhere near these levels. The %raw usage of the OSDs in the cluster ranges from about 60-80% and the raw usage of the cluster is about 75%. We do not get any "near full" warnings prior to OSD_FULL being set. Having a production system instantly go offline without warning isn't ideal and these things seem to know the least convenient moment. These problems only happen after a host failure. Each time we have added additional OSD hosts into the cluster, the backfilling has finished without problems. We are currently running Reef 18.2.4 but I have experienced these same problems on Pacific 16.2.10 Has anyone else seen this behaviour? Thanks Gerard

2 2

TPM2 in squid 19.2.0
by Ehsan Golpayegani 27 Dec '24

27 Dec '24

Hi ceph users, Has anyone successfully installed Ceph Squid with TPM2 enabled on an OSD? I'm encountering an error during installation: "stderr_lines": ["Error EINVAL: Failed to validate OSD spec "<unnamed>": Feature `tpm2` is not supported"]. I've also attempted to prepare the disks with TPM2 before adding them to the cluster, but I receive the following error: "/usr/bin/podman: stderr ceph-volume lvm prepare: error: unrecognized arguments: --with-tpm". For reference, my cluster is deployed via Cephadm and I'm using Squid 19.2.0. Any guidance would be greatly appreciated. Thanks, Ehsan -- -------------- Kind Regards <http://about.me/ehsan310?promo=email_sig> Ehsan Golpayegani [image: https://]www.ehsan.pw <https://ehsan.pw?promo=email_sig&utm_source=product&utm_medium=email_sig&ut…> SAVE PAPER - THINK BEFORE YOU PRINT!

1 0

RGW sizing in multisite and rgw_run_sync_thread
by Adam Prycki 26 Dec '24

26 Dec '24

Hello, I was recently reading rook ceph multisite documentation and I've found this rgw recommendation: "Scaling the number of gateways that run the synchronization thread to 2 or more can increase the latency of the replication of each S3 object. The recommended way to scale a multisite configuration is to dissociate the gateway dedicated to the synchronization from gateways that serve clients." Are there any official ceph recommendations about how many RGWs with rgw_run_sync_thread=true we should deploy in multisite zone? Have you ever experienced issues related to this in production? My plan for our multisite was to create 3 RGW daemons with rgw_run_sync_thread, put them behind load balancer and put load balancer VIP in zone endpoint. Rook documentation suggests it would be sub optimal configuration. Best regards Adam Prycki

2 1

2024-12-26 Perf Meeting Cancelled
by Matt Vandermeulen 26 Dec '24

26 Dec '24

Hi folks, the perf meeting for today will be cancelled for Boxing Day! Happy holidays! We will (probably) resume next week, on the 2nd. Thanks, Matt

1 0

radosgw stopped working
by Rok Jaklič 24 Dec '24

24 Dec '24

Hi, for some reason radosgw stopped working. Cluster status: [root@ctplmon1 ~]# ceph -v ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) [root@ctplmon1 ~]# ceph -s cluster: id: 0a6e5422-ac75-4093-af20-528ee00cc847 health: HEALTH_ERR 6 OSD(s) experiencing slow operations in BlueStore 2 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 32 pgs backfill_toofull Degraded data redundancy: 835306/1285383707 objects degraded (0.065%), 6 pgs degraded, 5 pgs undersized 76 pgs not deep-scrubbed in time 45 pgs not scrubbed in time Full OSDs blocking recovery: 1 pg recovery_toofull 9 pool(s) full 9 daemons have recently crashed services: mon: 3 daemons, quorum ctplmon1,ctplmon3,ctplmon2 (age 36m) mgr: ctplmon1(active, since 65m) mds: 1/1 daemons up osd: 193 osds: 191 up (since 8m), 191 in (since 9m); 267 remapped pgs rgw: 2 daemons active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 793 pgs objects: 257.08M objects, 292 TiB usage: 614 TiB used, 386 TiB / 1000 TiB avail pgs: 835306/1285383707 objects degraded (0.065%) 225512620/1285383707 objects misplaced (17.544%) 525 active+clean 230 active+remapped+backfilling 32 active+remapped+backfill_toofull 5 active+undersized+degraded+remapped+backfilling 1 active+recovery_toofull+degraded io: recovery: 978 MiB/s, 825 objects/s --- Do not know if it is related but the cluster has been rebalancing for a few days now, after I've set EC pool only to use hdd. --- If I start rgw with debug I get something like this in logs: [root@ctplmon2 ~]# radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n client.radosgw.moja.shramba.ctplmon2 -f -m 194.249.4.104:6789 --debug-rgw=99/99 2024-12-21T23:21:59.898+0100 7f659e380640 -1 Initialization timeout, failed to initialize In logs I get: 2024-12-21T23:16:59.898+0100 7f65a19257c0 0 deferred set uid:gid to 167:167 (ceph:ceph) 2024-12-21T23:16:59.898+0100 7f65a19257c0 0 ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable), process radosgw, pid 168935 2024-12-21T23:16:59.898+0100 7f65a19257c0 0 framework: beast 2024-12-21T23:16:59.898+0100 7f65a19257c0 0 framework conf key: port, val: 4444 2024-12-21T23:16:59.898+0100 7f65a19257c0 1 radosgw_Main not setting numa affinity 2024-12-21T23:16:59.901+0100 7f65a19257c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 2024-12-21T23:16:59.901+0100 7f65a19257c0 1 D3N datacache enabled: 0 2024-12-21T23:16:59.901+0100 7f658dffb640 20 reqs_thread_entry: start 2024-12-21T23:16:59.901+0100 7f658d7fa640 10 entry start 2024-12-21T23:16:59.908+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.914+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:16:59.914+0100 7f65a19257c0 20 rgw main: realm 2024-12-21T23:16:59.914+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.915+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:16:59.915+0100 7f65a19257c0 4 rgw main: RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-12-21T23:16:59.915+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.915+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:16:59.915+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.917+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:16:59.917+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.945+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=873 2024-12-21T23:16:59.945+0100 7f65a19257c0 20 rgw main: searching for the correct realm 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zone_info.c2c70444-7a41-4acd-a0d0-9f87d324ec72 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zonegroup_info.b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zone_names.default 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zonegroups_names.default 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.211+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:17:00.211+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.212+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=358 2024-12-21T23:17:00.212+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.213+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.213+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.214+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.214+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.215+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zone_info.c2c70444-7a41-4acd-a0d0-9f87d324ec72 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zonegroup_info.b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zone_names.default 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zonegroups_names.default 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.285+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:17:00.285+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.286+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=873 2024-12-21T23:17:00.286+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.287+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:17:00.287+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.293+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=358 2024-12-21T23:17:00.293+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.295+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.295+0100 7f65a19257c0 20 zone default found 2024-12-21T23:17:00.295+0100 7f65a19257c0 4 rgw main: Realm: () 2024-12-21T23:17:00.295+0100 7f65a19257c0 4 rgw main: ZoneGroup: default (b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578) 2024-12-21T23:17:00.295+0100 7f65a19257c0 4 rgw main: Zone: default (c2c70444-7a41-4acd-a0d0-9f87d324ec72) 2024-12-21T23:17:00.295+0100 7f65a19257c0 10 cannot find current period zonegroup using local zonegroup configuration 2024-12-21T23:17:00.295+0100 7f65a19257c0 20 rgw main: zonegroup default 2024-12-21T23:17:00.295+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.296+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.296+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.299+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.299+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.303+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.303+0100 7f65a19257c0 20 rgw main: started sync module instance, tier type = 2024-12-21T23:17:00.303+0100 7f65a19257c0 20 rgw main: started zone id=c2c70444-7a41-4acd-a0d0-9f87d324ec72 (name=default) with tier type = 2024-12-21T23:21:59.898+0100 7f659e380640 -1 Initialization timeout, failed to initialize --- Any ideas what might cause rgw to stop working? Kind regards, Rok

6 21

Issue With Dasboard TLS Certificate (Renewal)
by duluxoz 23 Dec '24

23 Dec '24

Hi All, So we've been using the Ceph (v18.2.4) Dashboard with internally generated TLS Certificates (via our Step-CA CA), one for each of our three Ceph Manager Nodes. Everything was working AOK. The TLS Certificates came up for renewal, which they were successfully renewed. Accordingly, the old Certificates & Keys were overwritten by the new ones and the commands `ceph mgr module disable/enabled dashboard` (respectively) were run. HOWEVER, the Ceph Dashboard stopped working / wouldn't use the renewed Certificates; as per the logs the Dashboard was still using the old Certificates and is now complaining that they have expired, and therefore the Dashboard won't run (unless I disable SSL via the CLI). I've been through the documentation and I can't work out what I've done wrong; according to this page (https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-ssl-tls-support) - in particular the Blue-Box Note - there's nothing else that needs to be done - EXCEPT that I've done all of that and the logs still tell me the Certificate(s) has expired - ie I can't get the Dashboard to recognise the new Certificates. Any help greatly appreciated. Thanks in advance Dulux-Oz

3 4

TPM2 capabilities
by Ehsan Golpayegani 22 Dec '24

22 Dec '24

Hi ceph users, Has anyone successfully installed Ceph Squid with TPM2 enabled on an OSD? I'm encountering an error during installation: "stderr_lines": ["Error EINVAL: Failed to validate OSD spec "<unnamed>": Feature `tpm2` is not supported"]. I've also attempted to prepare the disks with TPM2 before adding them to the cluster, but I receive the following error: "/usr/bin/podman: stderr ceph-volume lvm prepare: error: unrecognized arguments: --with-tpm". For reference, my cluster is deployed via Cephadm and I'm using Squid 19.2.0. Any guidance would be greatly appreciated. Thanks, Ehsan -------------- Kind Regards <http://about.me/ehsan310?promo=email_sig> Ehsan Golpayegani [image: https://]www.ehsan.pw <https://ehsan.pw?promo=email_sig&utm_source=product&utm_medium=email_sig&ut…> SAVE PAPER - THINK BEFORE YOU PRINT!

1 0

EC pool only for hdd
by Rok Jaklič 21 Dec '24

21 Dec '24

Hi, is it possible to set/change following already used rule to only use hdd? { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } Kind regards, Rok

4 15

Upgrade stalled after upgrading managers
by Torkil Svensgaard 20 Dec '24

20 Dec '24

Hi Running upgrade from 18.2.4 to 19.2.0 and it managed to upgrade the managers but no further progress. If I fail over the mgr it goes: " [root@ceph-flash1 ~]# ceph orch upgrade status Error ENOTSUP: Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it " From mgr log: " ... 2024-12-17T10:43:11.729+0000 7f70efafe640 0 log_channel(audit) log [DBG] : from='client.2110010386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-12-17T10:43:11.733+0000 7f70ebaf6640 0 [cephadm INFO cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Bus STARTING 2024-12-17T10:43:11.733+0000 7f70ebaf6640 0 log_channel(cephadm) log [INF] : [17/Dec/2024:10:43:11] ENGINE Bus STARTING 2024-12-17T10:43:11.811+0000 7f70e7aee640 0 [dashboard INFO dashboard.module] Engine started. 2024-12-17T10:43:11.861+0000 7f70ebaf6640 0 [cephadm INFO cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Serving on https://172.21.15.148:7150 2024-12-17T10:43:11.861+0000 7f70ebaf6640 0 log_channel(cephadm) log [INF] : [17/Dec/2024:10:43:11] ENGINE Serving on https://172.21.15.148:7150 2024-12-17T10:43:11.864+0000 7f70a2d7a640 0 [cephadm ERROR cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-12-17T10:43:11.865+0000 7f70a2d7a640 -1 log_channel(cephadm) log [ERR] : [17/Dec/2024:10:43:11] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-12-17T10:43:11.963+0000 7f70ebaf6640 0 [cephadm INFO cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Serving on http://172.21.15.148:8765 2024-12-17T10:43:11.963+0000 7f70ebaf6640 0 log_channel(cephadm) log [INF] : [17/Dec/2024:10:43:11] ENGINE Serving on http://172.21.15.148:8765 2024-12-17T10:43:11.963+0000 7f70ebaf6640 0 [cephadm INFO cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Bus STARTED 2024-12-17T10:43:11.964+0000 7f70ebaf6640 0 log_channel(cephadm) log [INF] : [17/Dec/2024:10:43:11] ENGINE Bus STARTED ... " It will recover after some timeout, maybe 5-10 mins, and then just sit there with no upgrade progress. Nothing in mgr/cephadm/osd_remove_queue. Suggestions? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

3 5

OSD stuck during a two-OSD drain
by Nicola Mori 20 Dec '24

20 Dec '24

Dear Ceph users, I'm upgrading some disks of my cluster (Squid 19.2.0 managed by cephadm, in which basically I have only a 6+2 EC pool over 12 hosts). To speed up the operations I issued a ceph orch osd rm --replace for two OSDs in two different hosts; the drain started for both and for one OSD finished smoothly and it is now in destroyed state. But for the second OSD it stopped with a single PG remaining to be moved away before the OSD is completely drained: # ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 31 rokanan draining 1 True False False 2024-12-19 08:57:36.458704+00:00 and there is no backfill activity going on, even i f the PG is labeled as backfilling : # ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 52 pgs not deep-scrubbed in time (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT) services: mon: 5 daemons, quorum bofur,fili,aka,bifur,romolo (age 7d) mgr: fili.olevnm(active, since 18h), standbys: bofur.tklnrn, bifur.htimkf mds: 2/2 daemons up, 1 standby osd: 124 osds: 123 up (since 4h), 122 in (since 22h); 1 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 27.11M objects, 78 TiB usage: 104 TiB used, 162 TiB / 266 TiB avail pgs: 53120/216457202 objects misplaced (0.025%) 302 active+clean 178 active+clean+scrubbing 48 active+clean+scrubbing+deep 1 active+remapped+backfilling Is all of the above normal? I guessed that maybe only one destroyed OSD at once can exist in the cluster, and that after replacing its disk and recreating it the drain for the second one would resume and finish, is this plausible? Thanks, Nicola

2 2

Cephadm multi zone rgw_dns_name setting
by Huseyin Cotuk 19 Dec '24

19 Dec '24

Hello Cephers, I am running a multi-site Ceph cluster with multiple zones in both regions. Specifically, site A has two zones named tr-ist-01 and tr-ist-02. The first zone (tr-ist-01) is used for local object storage needs and not synced with the remote site. The second zone (tr-ist-02) is used for multi-site and synced with remote site. When using cephadm with single zone radosgw, the fqdn can be set as follows. ceph config set global rgw_dns_name {rgw_service_fqdn} My question is, how can I set the fqdn of each radosgw service in a multiple zone environment? Rather than setting a global rgw_dns_name, is it possible to set this setting for each zone? It is possible to set this for each pod with specific podname but it can change in runtime. Any help will be appreciated. Thanks in advance Best regards, Huseyin Cotuk hcotuk(a)gmail.com

2 2

[RGW] multisite sync, stall recovering shards
by Gilles Mocellin 19 Dec '24

19 Dec '24

Hi there ! Being suspicious about replication between two clusters, I've done a radosgw-admin data sync init on the secondary zone. Since then, after a lot of activity, I'm stuck with recovering shards, nothing moves. Incremental sync still work. Wondering if I had a bad state also on the primary, I also did a data sync init on the primary... And now, it's also stuck with recovering shard ! In sync error list, I can find some "failed to sync bucket instance: (125) Operation canceled" errors I also tried to rewrite some buckets chown in those errors, but nothing changes. Strange, in those errors, the objects names are not real objects, example : "name": "replic_cfn_rec/cfb0047:aefd400 3-1866-4b16-b1b3-2f308075cd1c.20298566.4:11[0]" I wonder what is this ending ":10[0]". I also tried to remove stale instances, but nothing. I've still not retry a data sync init on secondary, perhaps I should, but the generated activity is impactfull. Can we reduce that resync activity priority ? ah, My primary cluster is on Reef 18.2.4, the secondary still on 18.2.2 (needs OS upgrade, Ubuntu 18.04). -- Gilles -- Gilles

1 0

(no subject)
by Gilles Mocellin 19 Dec '24

19 Dec '24

Hi there ! Being suspicious about replication between two clusters, I've done a radosgw-admin data sync init on the secondary zone. Since then, after a lot of activity, I'm stuck with recovering shards, nothing moves. Incremental sync still work. Wondering if I had a bad state also on the primary, I also did a data sync init on the primary... And now, it's also stuck with recovering shard ! In sync error list, I can find some "failed to sync bucket instance: (125) Operation canceled" errors I also tried to rewrite some buckets chown in those errors, but nothing changes. Strange, in those errors, the objects names are not real objects, example : "name": "replic_cfn_rec/cfb0047:aefd400 3-1866-4b16-b1b3-2f308075cd1c.20298566.4:11[0]" I wonder what is this ending ":10[0]". I also tried to remove stale instances, but nothing. I've still not retry a data sync init on secondary, perhaps I should, but the generated activity is impactfull. Can we reduce that resync activity priority ? ah, My primary cluster is on Reef 18.2.4, the secondary still on 18.2.2 (needs OS upgrade, Ubuntu 18.04). -- Gilles

1 0

January 2025 Ceph Meetup in Berlin, Germany and Frankfurt/Main, Germany - interested people welcome !
by Matthias Muench 19 Dec '24

19 Dec '24

Hi people in Germany, we run two meetup groups in Berlin and Frankfurt with meetings nearly quarterly. For each location, we provide also a quick introduction to Ceph before the main event. Currently, we have a new set of appointments. You can join us by RSVP to the events: - Ceph Frankfurt: 21.01.2025 in Frankfurt/Main, Germany Introduction: https://www.meetup.com/ceph-frankfurt/events/305161885/ Main meetup: https://www.meetup.com/ceph-frankfurt/events/305161680/ - Ceph Berlin: 20.01.2025 in Berlin, Germany Introduction: https://www.meetup.com/ceph-berlin/events/305161635/ Main meetup: https://www.meetup.com/ceph-berlin/events/305161618/ To get the invites also for any future meetings, one could also join the meetup groups: - Ceph Frankfurt: https://www.meetup.com/de-DE/ceph-frankfurt/ - Ceph Berlin: https://www.meetup.com/de-DE/ceph-berlin/ Happy to see you soon, -matt -- —————————————————— Matthias Muench Principal Specialist Solution Architect, OpenShift EMEA Storage Specialist matthias.muench(a)redhat.com Phone: +49-160-92654111 Red Hat GmbH Technopark II Werner-von-Siemens-Ring 12 85630 Grasbrunn Germany _______________________________________________________________________ Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

1 0

MONs not trimming
by Janek Bevendorff 19 Dec '24

19 Dec '24

Hi all, We moved our Ceph cluster to a new data centre about three months ago, which completely changed its physical topology. I changed the CRUSH map accordingly so that the CRUSH location matches the physical location again and the cluster has been rebalancing ever since. Due to capacity limits, the rebalancing requires rather frequent manual reweights so that individual OSDs don't run full. We started at around 65% of remapped PGs and are now down to 12% (it goes a little faster now that the bulk is done). Unfortunately, the MONs refuse to trim their stores while there are remapped PGs in the cluster, so their disk usage has increased gradually from 600MB to 17GB. I assume the rebalancing will finish before we run out of disk space, but the MON restart times have become unsustainably long and compacts now take up to 20 minutes or so. I'm a bit worried that at some point the MONs will start becoming unstable or worse: the stores may get corrupted. Is there anything that I can do to safely (!) get the MON stores back to a more manageable size even without all PGs being active+clean? I would like to forgo any potential disaster, especially before the holidays. Thanks Janek

6 13

CRC Bad Signature when using KRBD
by jsterr＠deckplosion.de 19 Dec '24

19 Dec '24

Hello Ceph Users, * Problem: we get the following errors when using krbd, we are using rbd for vms. * Workaround: by switching to librbd the errors disappear. * Software: ** Kernel: 6.8.8-2 (parameters: intel_iommu=on iommu=pt pcie_aspm.policy=performance) ** Ceph: 18.2.2 Description/Details: Errors from using krbd with ceph. Side-effects: [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000015af2284 data crc 1221767919 != exp. 282251377 [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000066b200ab data crc 3817026135 != exp. 3925662391 [Wed Aug 21 03:04:17 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:17 2024] libceph: osd13 (1)10.1.4.13:6809 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000008a131738 data crc 2612835980 != exp. 917302924 [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000005160776b data crc 2965872045 != exp. 563323792 [Wed Aug 21 03:04:21 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: osd6 (1)10.1.4.12:6843 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000007e548354 data crc 1265032637 != exp. 2426281931 [Wed Aug 21 03:06:44 2024] libceph: osd0 (1)10.1.4.11:6835 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000009214d802 data crc 2596010853 != exp. 1221875667 [Wed Aug 21 03:06:44 2024] libceph: osd10 (1)10.1.4.12:6809 bad crc/signature [Wed Aug 21 03:06:47 2024] libceph: read_partial_message 000000000f9edc73 data crc 1326019705 != exp. 3079604517 [Wed Aug 21 03:06:47 2024] libceph: osd3 (1)10.1.4.11:6803 bad crc/signature [Wed Aug 21 03:06:50 2024] libceph: read_partial_message 000000004769da61 data crc 3421275194 != exp. 4183754554 [Wed Aug 21 03:06:50 2024] libceph: osd8 (1)10.1.4.12:6835 bad crc/signature [Wed Aug 21 03:06:51 2024] libceph: read_partial_message 0000000044db9a59 data crc 2603270708 != exp. 4150529351 [Wed Aug 21 03:06:51 2024] libceph: osd14 (1)10.1.4.13:6806 bad crc/signature Description/Details 2: vms get problems with buffer i/o errors on rbd-backed virtual disks: Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=885s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=847s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Add. Sense: I/O process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 CDB: Write(10) 2a 00 34 87 48 08 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 CDB: Write (10) 2a 00 34 81 52 70 00 00 58 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281032 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 880890480 op 0x1: (WRITE) flags 0x103000 phys_seg 11 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: EXT4-fs warning (device sdb1): ext4_end_bio:343: I/0 error 10 writing to inode 27525908 starting Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111054, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: buffer_io_error: 21 callbacks suppressed Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on device sdbl, logical block 110159873 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on dev sdbl, logical block 110111055, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=875s Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111056, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111057, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111058, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 CDB: Write(10) 2a 00 34 87 48 10 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111059, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281040 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111060, lost async page write Thanks for helping out. Greetings Jonas

3 4

Squid: deep scrub issues
by Laimis Juzeliūnas 19 Dec '24

19 Dec '24

Hello Ceph community, Wanted to highlight one observation and gather any Squid users having similar experiences. Since upgrading to 19.2.0 (from 18.4.0) we have observed that pg deep scrubbing times have drastically increased. Some pgs take 2-5 days to complete deep scrubbing while others increase to 20+ days. This causes the deep scrubbing queue to fill up and the cluster almost constantly has 'pgs not deep-scrubbed in time' alerts. We have on average 67 pgs/osd: running on 15TB hdd disks this results in 200GB-ish pgs. While fairly large - these pgs did not cause such increase in deep scrubs when on Reef. "ceph pg dump | grep 'deep scrubbing for'" will always have a few entries of quite morbid scrubs like the following: 7.3e 121289 0 0 0 0 225333247207 0 0 127 0 127 active+clean+scrubbing+deep 2024-11-13T09:37:42.549418+0000 490179'5220664 490179:23902923 [268,27,122] 268 [268,27,122] 268 483850'5203141 2024-11-02T11:33:57.835277+0000 472713'5197481 2024-10-11T04:30:00.639763+0000 0 21873 deep scrubbing for 1169147s 34.247 62618 0 0 0 0 179797964677 0 0 101 50 101 active+clean+scrubbing+deep 2024-11-05T06:27:52.288785+0000 490179'22729571 490179:80672442 [34,97,25] 34 [34,97,25] 34 481331'22436869 2024-10-23T16:06:50.092439+0000 471395'22289914 2024-10-07T19:29:26.115047+0000 0 204864 deep scrubbing for 1871733s Not pointing any fingers but Squid release had "better scrub scheduling" announced. Though this is not scheduling directly, but maybe this change had any impact causing such behaviour? Scrubbing configurations: ceph config get osd | grep scrub global advanced osd_deep_scrub_interval 2678400.000000 global advanced osd_deep_scrub_large_omap_object_key_threshold 500000 global advanced osd_max_scrubs 5 global advanced osd_scrub_auto_repair true global advanced osd_scrub_max_interval 2678400.000000 global advanced osd_scrub_min_interval 172800.000000 Cluster details (backfilling expected and caused by some manual reweights): cluster: id: 96df99f6-fc1a-11ea-90a4-6cb3113cb732 health: HEALTH_WARN 24 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum ceph-node004,ceph-node003,ceph-node001,ceph-node005,ceph-node002 (age 4d) mgr: ceph-node001.hgythj(active, since 11d), standbys: ceph-node002.jphtvg mds: 20/20 daemons up, 12 standby osd: 384 osds: 384 up (since 25h), 384 in (since 5d); 5 remapped pgs rbd-mirror: 2 daemons active (2 hosts) rgw: 64 daemons active (32 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 8681 pgs objects: 758.42M objects, 1.5 PiB usage: 4.6 PiB used, 1.1 PiB / 5.7 PiB avail pgs: 275177/2275254543 objects misplaced (0.012%) 6807 active+clean 989 active+clean+scrubbing+deep 880 active+clean+scrubbing 5 active+remapped+backfilling io: client: 37 MiB/s rd, 59 MiB/s wr, 1.72k op/s rd, 439 op/s wr recovery: 70 MiB/s, 38 objects/s One thread of other users experiencing same 19.2.0 prolonged deep scrub issues: https://www.reddit.com/r/ceph/comments/1guynak/strange_issue_where_scrubdee… Any hints or help would be greately appreciated! Thanks in advance, Laimis J. laimis.juzeliunas(a)oxylabs.io

6 13

Re: Squid Manager Daemon: balancer crashing orchestrator and dashboard
by Alexander Closs 18 Dec '24

18 Dec '24

Just chiming in to say this also affected our cluster, same symptoms and a temporary fix of disabling the balancer. Happy to add my cluster's logs to the issue, though I suspect they'll look the same as Laimis' cluster. -Alex MIT CSAIL

7 7

pgs not deep-scrubbed in time
by Jan Kasprzak 18 Dec '24

18 Dec '24

Hello, Ceph users, a question/problem related to deep scrubbing: I have a HDD-based Ceph 18 cluster currently with 34 osds and 600-ish pgs. In order to avoid latency peaks which apparently correlate with HDD being 100 % busy for several hours during a deep scrub, I wanted to relax the scrubbing frequency and concurrency. Six days ago I have modified the following config parameters: ceph config set osd osd_scrub_max_interval 2592000 # was 604800 ceph config set osd osd_deep_scrub_interval 2592000 # was 604800 ceph config set osd osd_max_scrubs 1 # was 3 The intervals are four times longer and max_scrubs is three times lower, so the total scrubbing load should _decrease_. However, just several hours after the config change, my cluster went to HEALTH_WARN with "XX pgs not deep-scrubbed in time". I thought this was something temporary, but six days later, the number of pgs not scrubbed in time still grows - now it says 58. In "ceph -s" there are about 6-8 pgs active+clean+scrubbing, and 4-6 active+clean+scrubbing+deep all the time, so scrubbing still happens. According to "ceph pg dump" a deep scrub of a pg takes about 9000 seconds. It seems all pgs are scheduled to be scrubbed in the next two days: # for i in `seq 18 24`; do echo -n "2024-12-$i "; ceph pg dump 2>/dev/null | grep -c "scheduled @ 2024-12-$i"; done 2024-12-18 152 2024-12-19 422 2024-12-20 7 2024-12-21 0 2024-12-22 0 2024-12-23 0 2024-12-24 0 The positive thing is that latency-wise it helped: with at most one pg being deep-scrubbed per OSD at any time, the utilization of the HDD never grows near 100 %, it stays at ~60 % when deep scrubbing is in progress on that OSD. Is there any other config parameter which I should modify together with the above three parameters? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

Announcing go-ceph v0.31.0
by Anoop C S 18 Dec '24

18 Dec '24

Hi, We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.31.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S.

1 0

Update host operating system - Ceph version 18.2.4 reef
by alessandro＠universonet.com.br 18 Dec '24

18 Dec '24

Ceph version 18.2.4 reef (cephadm) Hello, We have a cluster running with 6 Ubuntu 20.04 servers and we would like to add another host but with Ubuntu 22.04, will we have any problems? We would like to add new HOST with Ubuntu 22.04 and deactivate the Ubuntu 20.04 ones, our idea would be to update the hosts from Ubuntu 20.04 to 22.04. If anyone can give us some guidance, thank you very much. Best regards, Alessandro

4 4

ceph network acl: multiple network prefixes possible?
by Stefan Kooman 18 Dec '24

18 Dec '24

Hi List, Is it possible to specify multiple network prefixes for a given ceph user? I.e. osd 'allow {access-spec} [{match-spec}] [network {network1/prefix} {network2/prefix} {network3/prefix}] ' osd 'profile {name} [pool={pool-name} [namespace={namespace-name}]] [network {network1/prefix} {network2/prefix} {network3/prefix}' Or in some other syntax (comma separated, multiple network tags, etc.? I have tried some different formats but they either result in "1 auth entities have invalid capabilities" or only one of three prefixed being used. Gr. Stefan [1]: https://docs.ceph.com/en/reef/rados/operations/user-management/#authorizati…

1 0

Erasure coding best practice
by Szabo, Istvan (Agoda) 17 Dec '24

17 Dec '24

Hi, Couple of time I saw some use-case in the mail list where people using erasure code on a "weird" way. Normally we use more data chunks than coding chunks, like k=4 m=2 or k=8 m=3. What I mean "weird" I saw someone using k=3 and m=8 (I might remember wrong). I'm trying to understand what is the benefit of the higher coding chunks? You can use smaller object size? Let's say 4:2 the minimum object size should be 24K at least or with 8:3 it would be 44K because nothing will be stored on smaller space. In case of k=3 m=8, smallest object can be 12K, but you can lose 8 nodes (pgs) and data still there? Ty

5 6

stray host with daemons
by Chris Webb 17 Dec '24

17 Dec '24

Hi, I managed to install our 15th server with a fully qualified hostname. I rectified this after adding the disks to the cluster (17.2.6) by changing the hostname. 1) ceph orch host ls - returns the correct (shrot) hostnames 2) ceph orch ps - returns the correct (short) hostname for all demons 3) ceph health detail returns the following ( despite failing the mgr ) [root@cephp15 ~]# ceph health detail HEALTH_WARN 1 stray host(s) with 12 daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 12 daemon(s) not managed by cephadm stray host cephp15.xxx.xxx.xxx has 12 stray daemons: ['osd.168', 'osd.169', 'osd.170', 'osd.171', 'osd.172', 'osd.173', 'osd.174', 'osd.175', 'osd.176', 'osd.177', 'osd.178', 'osd.179'] [root@cephp15 ~]# 4) ceph device ls - returns the incorrect fully qualified hostname ( despite rebooting the host ) 5) I've tried a "ceph host rescan", this does not appear to make any difference. Does anyone have any suggestions on how to resolve this in a non-destructive way? Thanks, Chris

3 3

Erasure coding issue
by Deba Dey 17 Dec '24

17 Dec '24

See I have total host 5, each host holding 24 HDD and each HDD is of size 9.1TiB. So, a total of 1.2PiB out of which i am getting 700TiB. I did erasure coding 3+2 and placement group 128. But, the issue i am facing is when I turn off one node write is completely disabled. Erasure coding 3+2 can handle two nodes failure but it's not working in my case. I request this community to help me tackle this issue. The min size is 3 and 4 pools are there.

3 2

Tracing Ceph with LTTng-UST issue
by IslamChakib Kedadsa 17 Dec '24

17 Dec '24

Hello, We are writing to you regarding an issue we encountered while attempting to trace Ceph with LTTng-UST. Below are the steps we have followed so far: 1. *Compiling Ceph with LTTng Support*: - We modified the debian/rules file to enable LTTng support using the following flags: -DWITH_LTTNG=ON and -DWITH_OSD_INSTRUMENT_FUNCTIONS=ON. - The build steps can be found in our GitHub <https://github.com/KedadsaChakib/ceph-configuration/blob/main/build/README.…> repository. 2. *Deploying a Ceph Cluster*: - We deployed the cluster manually (1MON,1MGR,2OSDs). Please find the script here <https://github.com/KedadsaChakib/ceph-configuration/blob/main/cluster/clust…> . 3. *Attempting to Trace with LTTng-UST*: - We tried listing the tracepoints using the following command : # lttng list -u - This command did not return any Ceph-related tracepoints: UST events: ------------- NONE 4. *Preloading liblttng-ust-fork.so*: - We attempted to preload liblttng-ust-fork.so before starting the Ceph daemons (MON, OSD, MGR) as follows: # systemctl edit ceph-mon@debian - We added the following configuration: [Service] Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so - After restarting the daemons, we executed lttng list -u, which displayed tracepoints related to the libraries (lttng_ust_lib:*), trace logs (lttng_ust_tracelog:*), and state dump events ( lttng_ust_statedump:*), instead of the expected Ceph-related tracepoints. *Expected Result*: UST events: ------------- PID: 100859 - Name: /path/to/ceph-osd pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) ... *Actual Result*: UST events: ------------- lttng_ust_lib:* lttng_ust_tracelog:* lttng_ust_statedump:* We would greatly appreciate any help you could provide to resolve this problem.

1 0

Correct way to replace working OSD disk keeping the same OSD ID
by Nicola Mori 17 Dec '24

17 Dec '24

Dear Ceph users, I'm struggling to unbderstand what is the correct way to remove a working disk and replace it (e.g. for a disk upgrade) while keeping the same OSD ID. I did this several times following this procedure: https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd every time facing a different issue. This time it happened that after removal the OSD was marked as destroyed, so I went on with disk replacement and then with: ceph orch apply osd --all-available-devices but this created a new OSD with a new ID instead of reusing the old one, which remained in destroyed state. I must be missing something but I cannot figure out what. Thanks, Nicola

5 6

we cannot read the prometheus Metrics
by Giovanna Ratini 17 Dec '24

17 Dec '24

Hello, we cannot read the prometheus Metrics anymore. ceph config get mgr mgr/cephadm/container_image_node_exporter quay.io/prometheus/node-exporter:v1.5.0 curl localhost:9926/metrics --> works curl 10.244.62.35:9926/metrics --> works (IP = POD IP) same namespace another pod ( rook-tools ) curl 10.244.62.35:9926/metrics --> timeout Any Idea? kubectl describe svc rook-ceph-exporter -n rook-ceph Name: rook-ceph-exporter Namespace: rook-ceph Labels: app=rook-ceph-exporter rook_cluster=rook-ceph Annotations: <none> Selector: app=rook-ceph-exporter,rook_cluster=rook-ceph Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.108.230.88 IPs: 10.108.230.88 Port: ceph-exporter-http-metrics 9926/TCP TargetPort: 9926/TCP Endpoints: 10.244.115.233:9926,10.244.132.1:9926,10.244.139.103:9926 + 11 more... Session Affinity: None Events: <none> Best whishes, Gio kubectl get endpoints rook-ceph-exporter -n rook-ceph NAME ENDPOINTS AGE rook-ceph-exporter 10.244.115.233:9926,10.244.132.1:9926,10.244.139.103:9926 + 11 more... 273d

1 0

doc: https://docs.ceph.com/ root URL still redirects to Reef
by Florian Haas 16 Dec '24

16 Dec '24

Hi everyone, A little while back I noticed that the docs.ceph.com root URL still redirects to the docs for Reef: $ curl -I https://docs.ceph.com/ HTTP/2 302 date: Fri, 06 Dec 2024 13:04:15 GMT content-type: text/html; charset=utf-8 content-length: 0 location: https://docs.ceph.com/en/reef/ This is still the case as of today. Considering the current major release is Squid, I am guessing we'll want this to point to https://docs.ceph.com/en/squid/ instead. Does that sound like a good suggestion to others too? Cheers, Florian

1 0

OSD bind ports min/max sizing
by Szabo, Istvan (Agoda) 16 Dec '24

16 Dec '24

Hi, Going through this documentation: https://docs.redhat.com/en/documentation/red_hat_ceph_storage/2/html/config… Wonder the 6800-7100 range consist how many osd normally? How do you size this in ceph? Thank you

3 2

failed to load OSD map for epoch 2898146, got 0 bytes
by Frank Schilder 15 Dec '24

15 Dec '24

Hi all, I have a strange problem on an octopus latest cluster. We had a couple of SSD OSDs down for a while and brought them up today again. For some reason, these OSDs don't come up and flood the log with messages like osd.1004 2971464 failed to load OSD map for epoch 2898146, got 0 bytes These messages cycle through the same epochs over and over again. I did not really fine too much help, there is an old thread about a similar/the same problem on a home lab cluster, with new OSDs though, I believe. I couldn't really find useful information. The OSDs seem to boot fine and then end up in something like a death loop. Below some snippets from the OSD log. Any hints appreciated. Thanks and best regards, Frank After OSD start, everything looks normal up to here: 2024-10-21T17:41:39.136+0200 7fad73cf6f00 0 osd.1004 2971464 load_pgs opened 205 pgs 2024-10-21T17:41:39.140+0200 7fad73cf6f00 -1 osd.1004 2971464 log_to_monitors {default=true} 2024-10-21T17:41:39.150+0200 7fad73cf6f00 -1 osd.1004 2971464 mon_cmd_maybe_osd_create fail: 'osd.1004 has already bound to class 'fs_meta', can not reset class to 'ssd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy 2024-10-21T17:41:39.155+0200 7fad519a3700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898134, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898135, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898136, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4f99f700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4f99f700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898134, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898135, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898136, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898137, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad73cf6f00 0 osd.1004 2971464 done with init, starting boot process 2024-10-21T17:41:39.155+0200 7fad73cf6f00 1 osd.1004 2971464 start_boot 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898138, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898139, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898140, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898141, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898142, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898143, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898144, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898145, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898146, got 0 bytes These messages repeat over and over again with some others of this form showing up every now and then: 2024-10-21T17:41:39.476+0200 7fad651ca700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 12] Generated table #82879: 76571 keys, 67866714 bytes 2024-10-21T17:41:39.688+0200 7fad651ca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1729525299690000, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 82879, "file_size": 67866714, "table_properties": {"data_size": 67111697, "index_size": 562601, "filter_size": 191557, "raw_key_size": 4823973, "raw_average_key_size": 63, "raw_value_size": 62631087, "raw_average_value_size": 817, "num_data_blocks": 15644, "num_entries": 76571, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} And another occasion: 2024-10-21T17:41:40.520+0200 7fad651ca700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 12] Generated table #82880: 76774 keys, 67868330 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899234, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899235, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899236, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad651ca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1729525300521403, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 82880, "file_size": 67868330, "table_properties": {"data_size": 67113021, "index_size": 562509, "filter_size": 191941, "raw_key_size": 4836742, "raw_average_key_size": 62, "raw_value_size": 62623274, "raw_average_value_size": 815, "num_data_blocks": 15630, "num_entries": 76774, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

6 12

OSD process in the "weird" state
by Jan Marek 13 Dec '24

13 Dec '24

Hello, we have sometimes problem with OSD process in the "weird" state - its "flapping" between dead and health state. CEPH cluster is in the version 18.2.2, installed by 'cephadm' bootstrap process. Cluster hosts uses for internal communication RoCE rdma: # ceph config dump: global advanced ms_cluster_type async+rdma * global advanced ms_public_type async+posix * Our netwok cards are: 43:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 43:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] (2x100Gbps) We have all-flash NVMe configuration. When some OSD goes to the "half-dead" state, we have in our syslog this messages: 2024-12-10T00:56:55.025681+01:00 c1-dc1 ceph-osd[3346501]: RDMAStack handle_async_event QP not dead, event associate qp number: 80235 Queue Pair status: IBV_QPS_ERR Event : last WQE reached 2024-12-10T00:56:55.025777+01:00 c1-dc1 ceph-89ec5e54-dba6-11ee-9c3e-72791a95392b-osd-1[3346299]: 2024-12-09T23:56:55.018+0000 7f5b5950c700 -1 RDMAStack handle_async_event QP not dead, event associate qp number: 80235 Queue Pair status: IBV_QPS_ERR Event : last WQE reached We are using podman as a container solution. When we do restart this problematic container, its start and everythink goes smoothly. If we don't restart this container, in log we have a lot of messages, that some OSD maked this OSD as dead, but this OSD complaints about it and cluster goes to the problematic state. We are using this CEPH cluster as a storage for ProxMox virtualization and some virtuals don't "survive" this situation, as they have not accessible theire "disks" :-(. Is there a some solution, which we can a try? Many thanks for advices. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

2 1

[RGW] Never ending PUT requests
by Gilles Mocellin 13 Dec '24

13 Dec '24

Hello ! Since we have upgraded our Ceph cluster from Ceph Pacific 16.2.15 to Ceph 18.2.2. We can see more requests in timeout. Especially PUT from backup tools, Nexus Repository... Een with 10min timeouts on haproxy for the server backends. We are upgrading Ubuntu from 18.04 to 22.04 to be able to upgrade to Ceph 18.2.4, but not finished yet. With debug log level, I can see only one thing concerning a never ending request. Here are the last logs : Dec 09 12:50:20 server-xxx bash[876499]: debug 2024-12-09T11:50:20.189+0000 7f5af9890700 10 req 7017125235642647451 0.004000126s s3:put_obj setting object write_tag=3e3b0a2b-1864-4923-93a6-237a13b51594.81430478.7017125235642647451 Dec 09 12:50:20 server-xxx bash[876499]: debug 2024-12-09T11:50:20.189+0000 7f5af9890700 20 req 7017125235642647451 0.004000126s s3:put_obj bucket index object: MYREGION.rgw.buckets.index:.dir.3e3b0a2b-1864-4923-93a6-237a13b51594.80569777.1.5 Dec 09 12:50:20 server-xxx bash[876499]: debug 2024-12-09T11:50:20.341+0000 7f5a83fa5700 20 req 7017125235642647451 0.156004906s s3:put_obj WARNING: blocking librados call I wonder if that last line is the culprit, end if the prcess ends or stay blocked indefinitelly... --Gilles

1 0

Dashboard redirection changed after upgrade octopus to pacific
by Frank Schilder 13 Dec '24

13 Dec '24

Hi all, this here are 2 questions, one out of curiosity and one about performance when having dashboard access behind an nginx reverse proxy. We follow instructions here: https://docs.ceph.com/en/pacific/mgr/dashboard/#proxy-configuration. Question 2 about nginx: Is anyone using dashboard access via an nginx reverse proxy? My interpretation of the documentation is, that it will work (nginx handles the redirect response transparently) as long as the one MGR pointed to is up. However, I wonder how smart the handling of redirects is. Does nginx remember a redirected connection or will it go through the "ask configured host"->"receive redirect"->"ask redirection target" procedure for every single http(s) request, or will it remember the redirection for established connections? Question 1 out of curiosity: On octopus MGRs would return redirection URLs containing the FQDN of the host. Under pacific this does not work anymore. One can enable hostname resolution (https://docs.ceph.com/en/pacific/mgr/dashboard/#resolve-ip-address-to-hostn…) however, this requires a reverse DNS lookup and is off by default. How did octopus do that? Was it reverse DNS lookup and it was just on by default? If not, why is the simple "use result of hostname -f" that does not require DNS not an option? Thanks for answers to either or both of these questions! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

19.2.1 reediness for QE Validation
by Yuri Weinstein 13 Dec '24

13 Dec '24

All PRs in the 19.2.1 milestone have been merged. And I've run preliminary tests on "squid-release" branch.: https://pulpito.ceph.com/?sha1=a4b5f6b5b3e7cf446c6f6bafb60001513356af2c There are lots of errors 'DevicePath' in the different suites. Suspect is https://tracker.ceph.com/issues/69067 We will continue testing when this is addressed. Thx YuriW

3 4

Ceph Cluster slowness in production
by Michel Niyoyita 12 Dec '24

12 Dec '24

Hello team, I have configured a ceph cluster running on ubuntu 20.04 , with 16 HDD for data and 4 SSD for journaling on each host. the total cluster size is 150 TB usable with a replica of 3. The cluster is in production , integrated with openstack with different pools: Volumes, vms, cinder and cinder-backup. The problem I have is this : For linux servers there is no slowness , but on windows servers is worse , they are very very slow, some of those servers have been migrated from vmware to openstack using coriolis , the network interfaces and switches are 10Gbps. Kindly advise the way I can speed up my cluster to avoid the slowness. Best regards Michel

3 2

How to list pg-upmap-items
by Frank Schilder 12 Dec '24

12 Dec '24

Dear all, during our upgrade from octopus to pacific the MGR suddenly started logging messages like this one to audit.log: 2024-12-10T10:30:01.105524+0100 mon.ceph-03 (mon.2) 3004 : audit [INF] from='mgr.424622547 192.168.32.67:0/63' entity='mgr.ceph-03' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.60", "id": [1054, 1125]}]: dispatch Apparently, the balancer got enabled and tried to do something. However, we never enabled pg-upmap on our cluster, because we still have jewel clients from the museum connected. Therefore, I'm pretty certain that all of these upmap requests either failed or are scheduled and pending. To be sure, I would like to confirm that nothing happened. How can I list upmap items and scheduled+pending upmap operations? If there are any, how do I delete these? I really would like to avoid that these requests start hurting in a few years from now. I looked at the documentation. Unfortunately, its the usual disease[1], commands for setting all sorts of stuff are documented, but commands to query anything seem to be missing. This workaround [root@gnosis osdmaps]# ceph osd dump -f json-pretty | grep upmap "pg_upmap": [], "pg_upmap_items": [], indicates nothing is screwed up yet. However, I would really like to know what happened to the MGR commands and where they are now. How do I confirm they went to digital heaven? [1] There are "ceph osd pg-upmap-items : set upmap items" and "ceph osd rm-pg-upmap-items : clear upmap items" commands. Why would anyone ever need a "ceph osd ls-pg-upmap-items"?? I found out that I can write it myself (https://ceph-users.ceph.narkive.com/h7y24SDg/stale-pg-upmap-items-entries-a… and https://gitlab.cern.ch/ceph/ceph-scripts/blob/master/tools/upmap/upmap-rema…) However, a good API is always symmetric to make it *easy* for users to check and fix screw-ups. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 3

The Object Gateway Service is not configured, Credentials not found for RGW Daemon
by Malte Stroem 12 Dec '24

12 Dec '24

Hello, this issue hit lots of people in the past. Now, I am on 18.2.2 and after configuring multisite between two clusters the RGWs/Object Gateways are not visible in the dashboard anymore. There are two realms, two zonegroups now and for each realm three RGWs running on one cluster. One of the two clusters works fine and shows the RGWs in the dashboard. The other one shows: Message: The Object Gateway Service is not configured Credentials not found for RGW Daemon: '' Please consult the documentation on how to configure and enable the Object Gateway management functionality. Setting the credentials again did not solve the problem. Debugging the dashboard shows: mgr get_typed_config RGW_API_ADMIN_RESOURCE not found mgr get_typed_config ALERTMANAGER_API_SSL_VERIFY not found mgr get_typed_config PROMETHEUS_API_SSL_VERIFY not found However: ceph dashboard get-rgw-api-admin-resource admin and: ceph dashboard get-alertmanager-api-ssl-verify True and: ceph dashboard get-prometheus-api-ssl-verify True and: ceph dashboard get-rgw-api-ssl-verify False Some ideas for debugging deeper? Javascript console in the browser did not show anything. Best, Malte

2 2

NFS cluster
by Rok Jaklič 12 Dec '24

12 Dec '24

Hi, I am trying to create nfs cluster with following command: ceph nfs cluster create cephnfs But I get an error like: Error EPERM: osd pool create failed: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1 retval: -34 Any ideas why? I also tried adding placement number at the end like: ceph nfs cluster create cephnfs 128 ... but I get the same error. Ceph version quincy: v17.2.8 Kind regards, Rok

2 1

admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
by Frank Schilder 12 Dec '24

12 Dec '24

Hi all, two days ago we upgraded our cluster from octopus to pacific. Everything went well and we see lots of improvements. Thanks for releasing the last stable version with all its fixes. I do have some questions though and this hiccup is one for starters: After the upgrade to pacific we started getting the error message "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory" when using the ceph daemon command. Here is the output of a full session: [root@ceph-adm:ceph-26 ~]# ceph daemon mon.ceph-26 version | jq .release "pacific" [root@ceph-adm:ceph-26 ~]# ceph --id admin daemon mon.ceph-26 version | jq .release admin_socket: exception getting command descriptions: [Errno 2] No such file or directory [root@ceph-adm:ceph-26 ~]# ceph --id admin daemon /var/run/ceph/ceph-mon.ceph-26.asok version | jq .release "pacific" [root@ceph-adm:ceph-26 ~]# ceph daemon /var/run/ceph/ceph-mon.ceph-26.asok version | jq .release "pacific" We observe that it is impossible to use the ceph daemon command in its simple form whenever a --id argument is present. This, unfortunately, creates an unnecessary restrictions, we can't use non-admin users any more. here is why this fails: [root@ceph-adm:ceph-26 ~]# strace ceph daemon mon.ceph-26 version |& grep asok stat("/var/run/ceph/ceph-mon.ceph-26.asok", {st_mode=S_IFSOCK|0755, st_size=0, ...}) = 0 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.ceph-26.asok"}, 37) = 0 getpeername(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.ceph-26.asok"}, [110 => 38]) = 0 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.ceph-26.asok"}, 37) = 0 getpeername(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.ceph-26.asok"}, [110 => 38]) = 0 [root@ceph-adm:ceph-26 ~]# strace ceph --id admin daemon mon.ceph-26 version |& grep asok stat("/var/run/ceph/ceph-mon.admin.asok", 0x7fffa65e9f00) = -1 ENOENT (No such file or directory) connect(3, {sa_family=AF_UNIX, sun_path="/var/run/ceph/ceph-mon.admin.asok"}, 35) = -1 ENOENT (No such file or directory) As you can see, the daemon name "ceph-26" was replaced with the user name "admin" passed with the argument to --id. As a result the command looks for a non-existent file. Passing the full path "fixes" this. This is clearly a bug and I wonder if there is a way out, for example, by setting an explicit daemon path template in the config. I will open a tracker if a user on quincy or newer confirms that this is present in newer versions as well. I wonder if this is a fall-out of https://docs.ceph.com/en/latest/releases/pacific/#id39 Point 3: "$pid expansion in config paths like admin_socket will now properly expand to the daemon pid for commands like ceph-mds or ceph-osd. Previously only ceph-fuse/rbd-nbd expanded $pid with the actual daemon pid." Thanks for any pointers on how to work around this issue. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

MDS crashing and stuck in replay(laggy) ( "batch_ops.empty()", "p->first <= start" )
by Enrico Favero 10 Dec '24

10 Dec '24

Hi all, at the University of Zurich we run a cephfs cluster of ~12PB raw size. We currently run Pacific 16.2.15 and our clients (Ubuntu 20.04) mount cephfs using the kernel driver. The cluster was deployed in Mimic and subsequently to Nautilus (14.2.22) and then Pacific (16.2.15). Last Wednesday (4 Dec.) the kernel clients have been upgraded from 5.15.0-124 to 5.15.0-126. We run 6 active MDSs and 2 standby with static pinning of directory subtrees. Yesterday (actually it happened on Friday evening) we experienced a problem where an MDS serving rank.4 crashed and then the two standby MDS in turn crashed when trying to replay the journal. The other ranks are not affected. We would appreciate any help to debug and recover the failed filesystem. Thank you. Following some info on the crash and status of the cluster. The oldest and initial crash info is: { "assert_condition": "batch_ops.empty()", "assert_file": "/build/ceph-16.2.15/src/mds/CDentry.h", "assert_func": "virtual CDentry::~CDentry()", "assert_line": 130, "assert_msg": "/build/ceph-16.2.15/src/mds/CDentry.h: In function 'virtual CDentry::~CDentry()' thread 7f14fb70e700 time 2024-12-06T19:08:09.292770+0100\n/build/ceph-16.2.15/src/mds/CDentry.h: 130: FAILED ceph_assert(batch_ops.empty())\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f1507109980]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f15077b4a2e]", "(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f15077b4bb8]", "(CDentry::~CDentry()+0x4d9) [0x564e03dec519]", "(CDir::remove_dentry(CDentry*)+0x17e) [0x564e03df672e]", "(MDCache::trim_dentry(CDentry*, std::map<int, boost::intrusive_ptr<MCacheExpire>, std::less<int>, std::allocator<std::pair<int const, boost::intrusive_ptr<MCacheExpire> > > >&)+0xd4) [0x564e03cb9b54]", "(MDCache::trim_lru(unsigned long, std::map<int, boost::intrusive_ptr<MCacheExpire>, std::less<int>, std::allocator<std::pair<int const, boost::intrusive_ptr<MCacheExpire> > > >&)+0x91a) [0x564e03cbb36a]", "(MDCache::trim(unsigned long)+0xc1) [0x564e03cbbde1]", "(MDCache::upkeep_main()+0x94a) [0x564e03cf8aea]", "/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df) [0x7f1506c276df]", "/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f15070fe6db]", "clone()" ], "ceph_version": "16.2.15", "crash_id": "2024-12-06T18:08:09.295408Z_e6f3560d-21aa-425d-a1b2-f0b5fb5bd103", "entity_name": "mds.mds-l20-33", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "18.04.6 LTS (Bionic Beaver)", "os_version_id": "18.04", "process_name": "ceph-mds", "stack_sig": "5e65f2db93b88e8fb398b21684d658a9ef1cec94845c09c581a0d8a80722960a", "timestamp": "2024-12-06T18:08:09.295408Z", "utsname_hostname": "mds-l20-33", "utsname_machine": "x86_64", "utsname_release": "5.4.0-146-generic", "utsname_sysname": "Linux", "utsname_version": "#163~18.04.1-Ubuntu SMP Mon Mar 20 15:02:59 UTC 2023" } and it looks similar to https://tracker.ceph.com/issues/66967 even though the "assert_line" is different. But all the following crash are different and look like: { "assert_condition": "p->first <= start", "assert_file": "/build/ceph-16.2.15/src/include/interval_set.h", "assert_func": "void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]", "assert_line": 568, "assert_msg": "/build/ceph-16.2.15/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7fdd353b9700 time 2024-12-06T19:08:24.550104+0100\n/build/ceph-16.2.15/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start)\n", "assert_thread_name": "md_log_replay", "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fdd445bb980]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fdd44c66a2e]", "(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fdd44c66bb8]", "(interval_set<inodeno_t, std::map>::erase(inodeno_t, inodeno_t, std::function<bool (inodeno_t, inodeno_t)>)+0x296) [0x55876572c9f6]", "(interval_set<inodeno_t, std::map>::erase(inodeno_t)+0x2f) [0x55876572ca3f]", "(EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x588b) [0x558765a0868b]", "(EUpdate::replay(MDSRank*)+0x5d) [0x558765a0a0bd]", "(MDLog::_replay_thread()+0x78e) [0x55876598d64e]", "(MDLog::ReplayThread::entry()+0xd) [0x55876567f4ad]", "/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fdd445b06db]", "clone()" ], "ceph_version": "16.2.15", "crash_id": "2024-12-06T18:08:24.552401Z_35179c40-1fb8-473e-84d6-c013b27f341f", "entity_name": "mds.mds-l20-33", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "18.04.6 LTS (Bionic Beaver)", "os_version_id": "18.04", "process_name": "ceph-mds", "stack_sig": "96ed41e917b880ad1183c7acd926e3259e37e1511eb6de7730a21d14121679b2", "timestamp": "2024-12-06T18:08:24.552401Z", "utsname_hostname": "mds-l20-33", "utsname_machine": "x86_64", "utsname_release": "5.4.0-146-generic", "utsname_sysname": "Linux", "utsname_version": "#163~18.04.1-Ubuntu SMP Mon Mar 20 15:02:59 UTC 2023" } that looks like https://tracker.ceph.com/issues/61009 even if it has not been reported for Pacific. Our current status is: root@mon-l19-31:~# ceph -s cluster: id: 6ef3c8fe-33e9-4db3-80a7-c64d8991b9f0 health: HEALTH_WARN 1054 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats 1 filesystem is degraded insufficient standby MDS daemons available 4 MDSs report slow requests 2 MDSs behind on trimming 4 pgs not deep-scrubbed in time 2 pools have too many placement groups 33 daemons have recently crashed services: mon: 3 daemons, quorum mon-l19-31,mon-l20-41,mon-l15-41 (age 2M) mgr: mon-l20-41(active, since 2M), standbys: mon-l19-31, mon-l15-41 mds: 6/6 daemons up osd: 1216 osds: 1213 up (since 11d), 1213 in (since 11d) data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 9729 pgs objects: 2.78G objects, 5.3 PiB usage: 8.4 PiB used, 3.3 PiB / 12 PiB avail pgs: 9655 active+clean 74 active+clean+scrubbing+deep root@mon-l19-31:~# ceph fs status spindlefs - 611 clients ========= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active mds-l21-34-2 Reqs: 1 /s 248k 222k 119k 5107 1 active mds-l19-33 Reqs: 1 /s 15.5M 14.5M 2049k 1836k 2 active mds-l21-33 Reqs: 0 /s 25.7M 20.5M 2765k 65.5k 3 active mds-l19-33-2 Reqs: 0 /s 20.4M 20.4M 582k 10.3k 4 replay(laggy) mds-l21-33-2 0 0 0 0 5 active mds-l21-34 Reqs: 1 /s 304k 274k 139k 9112 POOL TYPE USED AVAIL spindle_metadata metadata 592G 4321G spindle data 8384T 1092T vhp data 119T 10.2T MDS version: ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) root@mon-l19-31:~# ceph health detail HEALTH_WARN 1054 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats; 1 filesystem is degraded; insufficient standby MDS daemons available; 4 MDSs report slow requests; 2 MDSs behind on trimming; 4 pgs not deep-scrubbed in time; 2 pools have too many placement groups; 33 daemons have recently crashed [WRN] BLUESTORE_NO_PER_POOL_OMAP: 1054 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats osd.13 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics ... ... ... osd.1226 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics [WRN] FS_DEGRADED: 1 filesystem is degraded fs spindlefs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [WRN] MDS_SLOW_REQUEST: 4 MDSs report slow requests mds.mds-l21-33(mds.2): 953 slow requests are blocked > 30 secs mds.mds-l19-33(mds.1): 35 slow requests are blocked > 30 secs mds.mds-l21-34-2(mds.0): 19 slow requests are blocked > 30 secs mds.mds-l19-33-2(mds.3): 3 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 2 MDSs behind on trimming mds.mds-l19-33(mds.1): Behind on trimming (449/128) max_segments: 128, num_segments: 449 mds.mds-l19-33-2(mds.3): Behind on trimming (7099/128) max_segments: 128, num_segments: 7099 [WRN] PG_NOT_DEEP_SCRUBBED: 4 pgs not deep-scrubbed in time pg 11.140a not deep-scrubbed since 2024-11-28T08:56:45.425135+0100 pg 11.694 not deep-scrubbed since 2024-11-27T22:18:56.908953+0100 pg 11.21b not deep-scrubbed since 2024-11-28T04:48:14.734599+0100 pg 11.ad8 not deep-scrubbed since 2024-11-27T19:59:32.851964+0100 [WRN] POOL_TOO_MANY_PGS: 2 pools have too many placement groups Pool spindle has 8192 placement groups, should have 8192 Pool vhp has 1024 placement groups, should have 1024 [WRN] RECENT_CRASH: 33 daemons have recently crashed mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-06T18:08:50.420719Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-06T18:08:33.409537Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-06T18:08:27.335401Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-06T18:08:53.156894Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-06T18:08:31.359872Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-06T18:08:29.331676Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-06T18:08:09.295408Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-06T18:08:24.552401Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-06T18:09:13.029070Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-06T18:09:10.641714Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-06T18:09:18.205911Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-06T18:09:15.176043Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-08T13:33:16.196251Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-08T13:33:18.242351Z mds.mds-l20-33 crashed on host mds-l20-33 at 2024-12-08T13:33:13.762433Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-08T15:22:20.143608Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-08T15:22:18.083712Z mds.mds-l20-33-2 crashed on host mds-l20-33 at 2024-12-08T15:22:15.248451Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:19:10.277110Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:19:06.202423Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:23:24.469758Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:23:22.372727Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:19:08.271763Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:23:26.559707Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:50:21.508069Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:50:19.417250Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:50:17.469305Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:59:42.693519Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:59:40.494913Z mds.mds-l21-33-2 crashed on host mds-l21-33 at 2024-12-09T09:59:44.763196Z and 3 more Best regards, Enrico -- Enrico Favero S3IT Services and Support for Science IT Office Y11 F 52 University of Zürich Winterthurerstrasse 190, CH-8057 Zürich (Switzerland) Tel: +41 44 635 42 22

1 0

Performance Discrepancy Between rbd bench and fio on Ceph RBD
by Ml Ml 10 Dec '24

10 Dec '24

Hi all, I have a question regarding the performance differences between rbd bench and a fio test on a Ceph RBD. During testing of my Ceph environment, I noticed that rbd bench reports significantly higher write speeds compared to fio, which is run on an XFS filesystem mounted on the same RBD. Here are the test details: fio test: ------------ fio --name=ceph_bench --rw=write --bs=4M --direct=1 --ioengine=libaio --size=4G --numjobs=1 --filename=/mnt/ceph10/backup-cluster1/testfile Results: Write speed: ~70.3 MiB/s (73.7 MB/s) IOPS: ~17 Latency: ~56.4 ms (average) rbd bench: -------------- rbd bench --io-type write --io-size 4M --io-threads 16 --io-total 12G backup-proxmox/cluster1-new --keyring /etc/ceph/ceph10/ceph.client.admin.keyring --conf /etc/ceph/ceph10/ceph.conf Results: Write speed: ~150–180 MiB/s (sustained after initial peak) IOPS: ~43–46 Threads: 16 ceph tell osd.X bench: --------------------------- osd.0: IOPS: 20.03, Throughput: 80.11 MB/s osd.1: IOPS: 23.08, Throughput: 92.33 MB/s osd.2: IOPS: 22.01, Throughput: 88.02 MB/s osd.3: IOPS: 17.65, Throughput: 70.61 MB/s osd.4: IOPS: 20.15, Throughput: 80.59 MB/s osd.5: IOPS: 19.71, Throughput: 78.82 MB/s osd.6: IOPS: 18.96, Throughput: 75.84 MB/s osd.7: IOPS: 20.13, Throughput: 80.52 MB/s osd.8: IOPS: 16.45, Throughput: 65.79 MB/s (hm) osd.9: IOPS: 31.16, Throughput: 124.65 MB/s osd.10: IOPS: 23.19, Throughput: 92.76 MB/s osd.12: IOPS: 19.10, Throughput: 76.38 MB/s osd.13: IOPS: 28.02, Throughput: 112.09 MB/s osd.14: IOPS: 19.07, Throughput: 76.27 MB/s osd.15: IOPS: 21.40, Throughput: 85.59 MB/s osd.16: IOPS: 20.24, Throughput: 80.95 MB/s I understand that rbd bench is designed to test the raw performance of the RBD layer, while fio runs at the filesystem level. However, the large discrepancy between the two results has left me curious. What could be causing this difference? Are there any specific factors related to caching, journaling, or XFS itself that could explain the significantly lower performance in the fio test? Any insights or recommendations for further troubleshooting would be greatly appreciated! Thanks in advance! Best regards, Mario

4 4

is replica pool required to store metadata for EC pool?
by Tony Liu 10 Dec '24

10 Dec '24

Hi, https://docs.ceph.com/en/reef/rados/operations/erasure-code/#erasure-coding… A replica pool is required to store metadata for EC pool? My case is RBD with OpenStack. With EC pool, the RBD image has to be created like this? rbd create --size 1G --data-pool ec_pool replicated_pool/image_name What about all other RBD operations, like snapshot, import, export, etc. Thanks! Tony

2 1

About erasure code for larger hdd
by Phong Tran Thanh 10 Dec '24

10 Dec '24

Hi community, Please help with advice on selecting an erasure coding algorithm for a 6-node cluster with 540 OSDs. What would be the appropriate values for *k* and *m*? The cluster requires a high level of HA and consistent throughput. Email: tranphong079(a)gmail.com

3 3

Single unfound object in cluster with no previous version - is there anyway to recover rather than deleting the object?
by Ivan Clayson 10 Dec '24

10 Dec '24

Hello, We have an Alma8.9 (version 4 kernel) quincy (17.2.7) CephFS cluster with spinners for our bulk data and SSDs for the metadata where we have a single unfound object in the bulk pool: [root@ceph-n30 ~]# ceph -s cluster: id: fa7cf62b-e261-49cd-b00e-383c36b79ef3 health: HEALTH_ERR 1/849660811 objects unfound (0.000%) Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 9/8468903874 objects degraded (0.000%), 1 pg degraded services: mon: 3 daemons, quorum ceph-s2,ceph-s3,ceph-s1 (age 44h) mgr: ceph-s2(active, since 45h), standbys: ceph-s3, ceph-s1 mds: 1/1 daemons up, 3 standby osd: 439 osds: 439 up (since 43h), 439 in (since 43h); 176 remapped pgs data: volumes: 1/1 healthy pools: 9 pools, 4321 pgs objects: 849.66M objects, 2.3 PiB usage: 3.0 PiB used, 1.7 PiB / 4.6 PiB avail pgs: 9/8468903874 objects degraded (0.000%) 36630744/8468903874 objects misplaced (0.433%) 1/849660811 objects unfound (0.000%) 4122 active+clean 174 active+remapped+backfill_wait 22 active+clean+scrubbing+deep 2 active+remapped+backfilling 1 active+recovery_unfound+degraded io: client: 669 MiB/s rd, 87 MiB/s wr, 302 op/s rd, 77 op/s wr recovery: 175 MiB/s, 59 objects/s [root@ceph-n30 ~]# ceph health detail | grep unfound HEALTH_ERR 1/849661114 objects unfound (0.000%); Possible data damage: 1 pg recovery_unfound; Degraded data redundancy: 9/8468906904 objects degraded (0.000%), 1 pg degraded [WRN] OBJECT_UNFOUND: 1/849661114 objects unfound (0.000%) pg 2.c90 has 1 unfound objects [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound pg 2.c90 is active+recovery_unfound+degraded, acting [259,210,390,209,43,66,322,297,25,374], 1 unfound pg 2.c90 is active+recovery_unfound+degraded, acting [259,210,390,209,43,66,322,297,25,374], 1 unfound We've tried deep-scrubbing and repairing the PG as well as rebooting the entire cluster but unfortunately this has not resolved our issue. The primary OSD (259) log reports that our 1009e1df26d.000000c9 object is missing where when we do rados commands on the object that command just hangs: [root@ceph-n30 ~]# grep 2.c90 /var/log/ceph/ceph-osd.259.log ... 2024-11-25T11:38:33.860+0000 7fd409870700 1 osd.259 pg_epoch: 512353 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512348/512349 n=211842 ec=1175/1168 lis/c=512348/472766 les/c/f=512349/472770/232522 sis=512353 pruub=11.010143280s) [259,210,390,209,43,66,322,297,NONE,374]p259(0) r=0 lpr=512353 pi=[472766,512353)/11 crt=512310'8145216 mlcod 0'0 unknown pruub 205.739364624s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:38:54.926+0000 7fd409870700 1 osd.259 pg_epoch: 512356 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512353/512354 n=211842 ec=1175/1168 lis/c=512353/472766 les/c/f=512354/472770/232522 sis=512356 pruub=11.945847511s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active pruub 227.741577148s@ m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(0+0)=1},9={(1+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,2147483647,374] -> [259,210,390,209,43,66,322,297,25,374], acting [259,210,390,209,43,66,322,297,2147483647,374] -> [259,210,390,209,43,66,322,297,25,374], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:38:54.926+0000 7fd409870700 1 osd.259 pg_epoch: 512356 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512353/512354 n=211842 ec=1175/1168 lis/c=512353/472766 les/c/f=512354/472770/232522 sis=512356 pruub=11.945847511s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 unknown pruub 227.741577148s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:38:59.910+0000 7fd409870700 0 osd.259 pg_epoch: 512359 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512356) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] get_remaining_shards not enough shards left to try for 2:0930c16c:::1009e1df26d.000000c9:head read result was read_result_t(r=0, errors={25(8)=-2,43(4)=-2,66(5)=-2,209(3)=-2,210(1)=-2,297(7)=-2,322(6)=-2,390(2)=-2}, noattrs, returned=(0, 8388608, [])) 2024-11-25T11:38:59.911+0000 7fd409870700 0 osd.259 pg_epoch: 512359 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512356) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] on_failed_pull 2:0930c16c:::1009e1df26d.000000c9:head from shard 25(8),43(4),66(5),209(3),210(1),297(7),322(6),390(2), reps on 374(9) unfound? 1 2024-11-25T11:39:01.435+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost 2024-11-25T11:39:02.193+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost 2024-11-25T11:39:03.590+0000 7fd409870700 1 osd.259 pg_epoch: 512362 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512362 pruub=8.352689743s) [259,210,390,209,43,66,322,297,25,NONE]p259(0) r=0 lpr=512362 pi=[472766,512362)/11 crt=512310'8145216 mlcod 0'0 active pruub 232.812744141s@ m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,25,374] -> [259,210,390,209,43,66,322,297,25,2147483647], acting [259,210,390,209,43,66,322,297,25,374] -> [259,210,390,209,43,66,322,297,25,2147483647], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:39:03.591+0000 7fd409870700 1 osd.259 pg_epoch: 512362 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512362 pruub=8.352689743s) [259,210,390,209,43,66,322,297,25,NONE]p259(0) r=0 lpr=512362 pi=[472766,512362)/11 crt=512310'8145216 mlcod 0'0 unknown pruub 232.812744141s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:39:24.954+0000 7fd409870700 1 osd.259 pg_epoch: 512365 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512362/512363 n=211842 ec=1175/1168 lis/c=512362/472766 les/c/f=512363/472770/232522 sis=512365 pruub=11.731550217s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active pruub 257.554870605s@ m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(0+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,25,2147483647] -> [259,210,390,209,43,66,322,297,25,374], acting [259,210,390,209,43,66,322,297,25,2147483647] -> [259,210,390,209,43,66,322,297,25,374], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:39:24.954+0000 7fd409870700 1 osd.259 pg_epoch: 512365 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512362/512363 n=211842 ec=1175/1168 lis/c=512362/472766 les/c/f=512363/472770/232522 sis=512365 pruub=11.731550217s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 unknown pruub 257.554870605s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:39:30.679+0000 7fd409870700 0 osd.259 pg_epoch: 512368 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512365/512366 n=211842 ec=1175/1168 lis/c=512365/472766 les/c/f=512366/472770/232522 sis=512365) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] get_remaining_shards not enough shards left to try for 2:0930c16c:::1009e1df26d.000000c9:head read result was read_result_t(r=0, errors={25(8)=-2,43(4)=-2,66(5)=-2,209(3)=-2,210(1)=-2,297(7)=-2,322(6)=-2,390(2)=-2}, noattrs, returned=(0, 8388608, [])) 2024-11-25T11:39:30.679+0000 7fd409870700 0 osd.259 pg_epoch: 512368 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512365/512366 n=211842 ec=1175/1168 lis/c=512365/472766 les/c/f=512366/472770/232522 sis=512365) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] on_failed_pull 2:0930c16c:::1009e1df26d.000000c9:head from shard 25(8),43(4),66(5),209(3),210(1),297(7),322(6),390(2), reps on 374(9) unfound? 1 2024-11-25T11:40:11.652+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost [root@ceph-n30 ~]# rados -p ec82pool stat 1009e1df26d.000000c9 ... # hanging indefinitely Similarly if we do any object store commands, then we get crashes and seg faults as well: [root@ceph-n25 ~]# ceph pg 2.c90 list_unfound { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "1009e1df26d.000000c9", "key": "", "snapid": -2, "hash": 914558096, "max": 0, "pool": 2, "namespace": "" }, "need": "510502'8140206", "have": "0'0", "flags": "none", "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", "locations": [ "374(9)" ] } ], "state": "NotRecovering", "available_might_have_unfound": true, "might_have_unfound": [], "more": false } [root@ceph-n11 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-374 --debug --pgid 2.c90 1009e1df26d.000000c9 dump ... -6> 2024-11-25T15:54:57.387+0000 7f8b6a0a7340 1 bluestore(/var/lib/ceph/osd/ceph-374) _upgrade_super from 4, latest 4 -5> 2024-11-25T15:54:57.387+0000 7f8b6a0a7340 1 bluestore(/var/lib/ceph/osd/ceph-374) _upgrade_super done -4> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 prioritycache tune_memory target: 4294967296 mapped: 132694016 unmapped: 618364928 heap: 751058944 old mem: 134217728 new mem: 2761652735 -3> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0540541 -2> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 prioritycache tune_memory target: 4294967296 mapped: 132939776 unmapped: 618119168 heap: 751058944 old mem: 2761652735 new mem: 2842823159 -1> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 bluestore.MempoolThread(0x55cd3a807b40) _resize_shards cache_size: 2842823159 kv_alloc: 1241513984 kv_used: 2567024 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 1174405120 meta_used: 13360 data_alloc: 218103808 data_u sed: 0 0> 2024-11-25T15:54:57.445+0000 7f8b6a0a7340 -1 *** Caught signal (Segmentation fault) ** in thread 7f8b6a0a7340 thread_name:ceph-objectstor ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8b67941cf0] 2: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x4c) [0x55cd37e34f5c] 3: (_action_on_all_objects_in_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x13b4) [0x55cd377fdf64] 4: (action_on_all_objects_in_exact_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x64) [0x55cd377fe274] 5: main() 6: __libc_start_main() 7: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. We don't have a previous verison of this object and trying to fix-lost with the object store command seg faults: [root@ceph-n11 ~] ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-374 --pgid 2.c90 --op fix-lost --dry-run *** Caught signal (Segmentation fault) ** in thread 7f45d9890340 thread_name:ceph-objectstor ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f45d712acf0] 2: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x4c) [0x5556f3c36f5c] 3: (_action_on_all_objects_in_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x13b4) [0x5556f35fff64] 4: (action_on_all_objects_in_exact_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x64) [0x5556f3600274] 5: main() 6: __libc_start_main() 7: _start() Segmentation fault (core dumped) The recommended solution seems to be to use "mark_unfound_lost revert" but for our object, there is no previous version so I think this command will discard and delete the object. The lost object is on our live filesystem and there seems to be no easily way to find the backup version as I can't access the path and or filename associated with the object to recover it from the backup. Is there any way for us to recover this object without discarding it? Or should we just accept our losses and delete it? Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N269 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

2 2

ceph multisite lifecycle not working
by Christopher Durham 09 Dec '24

09 Dec '24

I have 18.2.4 on Rocky 9 Linux.This system has been updated from octopus -> pacific -> quincy (18.2.2) -> (el8->el9 reinstall of each server, but ceph osd and mon survival) -> reef (18.2.4) over several years. It appears that I have two probably related problems with lifecycle expiration in a multsite configuration. I have two zones, one on each side of a multisite. I recently discovered (about a month after the el9 and reef 18.2.4 updates) that lifecycle expiration was (mostly) not working on the secondary zone side.I had thought initially that there may be replication issues, but while there are replication issues on individual buckets that required me to full sync individual buckets, the majority of the issues are becauselifecycle expiration is not working on the secondary side. The observation that caused me to think lifecycle is the issue is that based on a lifecycle policy for a given bucket, all objects in that bucket should be already deleted.What we are seeing is that all objects have been deleted from the bucket on the master zone, but NONE of them have been deleted on the slave side.This may vary based on the date the objects were created across multiple lifecycle runs on the master side, but objects never get deleted/expired on the slave side. I tracked this down to one of two causes, let's say for a given bucket bucket1 1. radosgw-admin lc list on the master shows that the bucket completes its lifecycle processing periodically. But on the slave side, it shows: "started": "Thu, 01 Jan 1970 ...""status": "UNINITIAL" If I run: radosgw-admin lc process --bucket bucket1 that particular bucket flushes all of its expired objects (takes awhile). But as far as I can tell at this point, it never runs lifecycle again on the slave side Now, let's say I have bucket2. 2. radosgw-admin lc list on the slave side does NOT show the bucket in the json output, yet the same command on the master side shows it! Given this, if I run radosgw-admin lc process --bucket2 causes C++ exceptions and the command crashes on the slave side (makes sense, actually) Yet in this case if I do: aws --profile bucket2_owner s3api get-bucket-lifecycle-configuration --bucket bucket2 it shows the lifecycle configuration for the bucket, regardless whether I point the awscli to the master or slave zone. In this case, if I redeploy the lifecycle with put-bucket-lifecycle-configuration to the master side, then thelifecycle status shows up in radosgw-admin lc list on the slave side (as well as on the master) as UNINITIAL, and this issue devolves to #1 above, Note that lifecycle expiration on the slave side does work for some number of buckets, but most remain in the UNINITIAL state, and others not there at all until Iredeploy the lifecycle. The slave side is a lot more active in reading and writing. So, why would the bucket not show up in lc list on the slave side, where it had before (I can't say how long ago 'before' was)?How can I get it to automatically perform lifecycle on the slave side? Would this perhaps be related to rgw_lc_max_workerrgw_lc_max_wp_workerrgw_lifecycle_work_time It appears that lifecycle processing is independent on each side, meaning that a lifecycle processing of bucket A on one side runs separately from lifecycle processing of bucket A on the other side, and as such an object may exist on one side for a time when it has been already deleted on the other side. How does rgw_lifecycle_work_time work? Does it mean that outside of the work_time window no new lifecyle processing starts, or that those in process abort/stop? Either way this may explain my observations as to too many buckets staying in UNINITIAL when those that are processing have a lot of data to delete. And why is this last one rgw_lifecycle_work_time and not rgw_lc_work_time? Anyway, any help on theses issues would be appreciated. Thanks \-Chris

2 2

Replacing Ceph Monitors for Openstack
by Adrien Georget 09 Dec '24

09 Dec '24

Hello, We are using Ceph as a storage backend for Openstack (Cinder, Nova, Glance, Manila) and we are replacing old hardware hosting Ceph monitors (MON,MGR,MDS) to new ones. I have already added the new ones in production, monitors successfully joined the quorum and new MGR/MDS are standby. For the monitors, I'm sure that the monmap is already up to date and Openstack clients are already aware of the change and it should not be a problem when I will next shut down the old monitors. The ceph.conf will be updated in all Openstack controllers to replace "mon host" with the new ones before shutting old mons down. But I have some doubts with the resilience of Openstack Manila service because the IP addresses of the monitors look hardcoded in the export location of the manila share : The manila show command returns for example : | export_locations | | | | path = 134.158.208.140:6789,134.158.208.141:6789,134.158.208.142:6789:/volumes/EC_manila/_nogroup/7a6c05d9-2fea-43b1-a6d4-06eec1e384f2 | | | share_instance_id = 7a6c05d9-2fea-43b1-a6d4-06eec1e384f2 | Has anyone already done this kind of migration in the past and can confirm my doubts? Is there any process to update shares? Cheers, Adrien

4 4

Ceph Steering Committee Election: Ceph Executive Council
by Patrick Donnelly 08 Dec '24

08 Dec '24

The Ceph Steering Committee completed an election [1] on November 18th for the next 2 year term of the Ceph Executive Council. Congratulations to Dan, Josh, and Neha on their reelection. [1] https://vote.heliosvoting.org/helios/elections/b2fa20e6-1a5d-4f49-a747-0f8f… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Slow ops during index pool recovery causes cluster performance drop to 1%
by Szabo, Istvan (Agoda) 07 Dec '24

07 Dec '24

Hi, I'm updating from octopus to quincy and all in our cluster when index pool recovery kicks off, cluster operation drops to 1%, slow ops comes non-stop. The recovery takes 1-2 hours/nodes. What I can see the iowait on the nvme drives which belongs to the index pool is pretty high, however the throughput is less than 500MB/s, the iops is less than 5000/sec. The index pool is a 3:2 replica pool with 2048pg on 156 osd (1 nvme drive has 4 osds due to we experienced latency issue with 1 or 2 osd/nvme). If we consider let's say the nvme drive still slow with these really small load, how would that be possible to somehow ease and get rid of this cluster performance drop? If I increase replica to 4-5 would that help? It could tolerate more pg slowness maybe? FYI we have many objects in our cluster, more than 4Billions: objects: 4.06G objects, 616 TiB However I think it should still tolerate cluster recovery without penalty. What I can see in the slow osd log with default debug value is about "get_health_metrics" so far : 2024-11-02T12:38:40.762+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 6 slow requests (by type [ 'delayed' : 6 ] most affected pool [ 'hkg.rgw.buckets.index' : 6 ]) 2024-11-02T12:38:41.802+0700 7f241bc25640 -1 osd.110 626281 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:41.802+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) 2024-11-02T12:38:42.782+0700 7f241bc25640 -1 osd.110 626282 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:42.782+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) 2024-11-02T12:38:43.802+0700 7f241bc25640 -1 osd.110 626282 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:43.802+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) How we also try to make it smoother, after update and machine reboot compaction kicks off which generates 30-40 iowait on the node, we prevent with "noup" flag to put these osds into the cluster until compaction finished, however when we have 0 iowait after compaction, I unset noup so recovery can start which causes the above issue. If I wouldn't set noup it would cause even bigger issue. Thank you for help ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 15

Dump/Add users yaml/json
by Albert Shih 06 Dec '24

06 Dec '24

Hi everyone, Stupid question....after some test I was able to dump a user caps with ceph auth get --format json but I'wasn't able to find the other way, something ceph auth add fubar.json Is they are any way to add a user (without given a key and with a key). Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 29 nov. 2024 18:12:35 CET

4 7

CephFS: Revert snapshot
by Andre Tann 05 Dec '24

05 Dec '24

Hi all, in CephFS, creating and removing a snapshot is as easy as - mkdir .snap/SNAPSHOT_NAME - rmdir .snap/SNAPSHOT_NAME Both steps are completed in no time, because no data has to be moved around. However, what I cannot find is a way to revert to a snapshot. The only way I see is to rsync … .snap/SNAPSHOT_NAME/ . In this case all the data has to be moved around, with two main disadvantages: - takes forever - consumes double the space than before Is there a better way to do this? Cheers -- Andre Tann

2 1

Weird pg degradation behavior
by Szabo, Istvan (Agoda) 04 Dec '24

04 Dec '24

Hi, I lost a node due to cpu failure so until hardware fix, I let it down as Ceph marked osds out, however I keep it in the tree because in 1-2 days will be online again. This is the current state: health: HEALTH_WARN Degraded data redundancy: 1726/23917349718 objects degraded(0.000%), 1 pg degraded, 1 pg undersized Out of the 9x osd nodes 2x of them have 4x nvme osd (temporarily) and 7x has only 2x (for index pool on 3 replicas) where this pg is located. (The down osd node is a 2x nvme osd one) This is some part of the query of that specific degraded pg: ... "up": [ 233, 202 ], "acting": [ 233, 202 ], "avail_no_missing": [ "233", "202" ], ... The above mentioned osds are on the 2x osd nodes with 4x nvme. What is not clear, I don't see any probing osd or any indicator for the 3rd missing part of that specific pg. This is the pgs_brief: PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 10.f6 active+undersized+degraded [233,202] 233 [233,202] 233 Shouldn't have any indicator what it is looking for? Which osd or some probing osd? (It's quincy 17.2.7). If I purge the down osds on the down server, I guess it would kick off recovery, I'm just curious why don't show anywhere what it is looking for? Ty

1 0

what's the minimum m to keep cluster functioning when 2 OSDs are down?
by Tony Liu 04 Dec '24

04 Dec '24

Hi, With Reef, I'd like to keep the cluster functioning while 2 OSDs are down. The cluster has more than 10 nodes, each of which has 64x 2.5G threads, 256GB memory, 24x NVMe drives and 2x 25GB networking. Will k=4 and m=2 achieve what I want? I know no data loss with m=2 when 2 OSDs are down, but is the cluster still functioning/writable in that case? Or m=3 is required to keep cluster functioning while 2 OSDs are down? In case m=3, what would be the reasonable k? Overhead 1.5 or 1.6 is efficient enough for me. So, k=5 or k=6? Any comments is welcome. Thanks! Tony

2 1

Mailing List Issues
by Kozakis, Anestis 03 Dec '24

03 Dec '24

Apologies for posting this here but was unsure where else I could dot his. I am being made redundant in my current position (Last day is this Friday). I have been able to successfully subscribe using my personal e-mail address to the ceph-announce list, but every time I try to subscribe to ceph-users from the web page, I keep getting an API/Mailman Core error. Anestis Kozakis Systems Administrator - Multi-Level Security Solutions P: + 61 2 6122 0205 M: +61 4 88 376 339 anestis.kozakis(a)raytheon.com.au<mailto:anestis.kozakis@raytheon.com.au> Raytheon Australia Cybersecurity and Information Assurance 4 Brindabella Cct Brindabella Business Park Canberra Airport, ACT 2609 www.raytheonaustralia.com.au<http://www.raytheonaustralia.com.au/> LinkedIn<https://au.linkedin.com/company/raytheon-australia> | Twitter<https://twitter.com/RaytheonAU> | Facebook<https://www.facebook.com/RaytheonAustralia> | Instagram<https://www.instagram.com/raytheonaustralia/>

2 1

Update host operating system - Ceph version 18.2.4 reef
by alessandro＠universonet.com.br 02 Dec '24

02 Dec '24

Ceph version 18.2.4 reef (cephadm) Hello, We have a cluster running with 6 Ubuntu 20.04 servers and we would like to add another host but with Ubuntu 22.04, will we have any problems? We would like to add new HOST with Ubuntu 22.04 and deactivate the Ubuntu 20.04 ones, our idea would be to update the hosts from Ubuntu 20.04 to 22.04. If anyone can give us some guidance, thank you very much.

1 0

lifecycle processing in multisite
by Christopher Durham 02 Dec '24

02 Dec '24

Hello, I have reef 18.2.4 on Rocky9. I have a multi-site environment with one zone on each side of the multi-site. Replication appears to be working in both directions. My question is, is lifecycle processing independent on each side? I have a lifecycle json installed to delete all objects in a given bucket and prefix after say, 180 days. (6 months). I have noticed that if I run: # radosgw-admin lc process --bucket <bucket> --rgw-debug 20 on the master zone, then the objects this process deletes do not go away on the other side. I expect that perhaps tonight, when lifecycle runs on the slave side, that these objects will disappear there, butwanted to confirm that lifecycle processing is independent on each zone. It appears that the s3api that puts the lifecycle json config ends up pushing it to each zone, so that the lifecycle is the same on all zones But the lifecycle cleanup may not actually execute at exactly the same time on all zones. Am I correct? If not, what should I be seeing? Thanks -Chris

1 0

down OSDs, Bluestore out of space, unable to restart
by John Jasen 02 Dec '24

02 Dec '24

Ceph version 17.2.6 After a power loss event affecting my ceph cluster, I've been putting humpty dumpty back together since. One problem I face is that with objects degraded, rebalancing doesn't run -- and this resulted in several of my fast OSDs filling up. I have 8 OSDs currently down, 100% full (exceeding all the full ratio settings on by default or I toggled to try and keep it together), and when I try to restart them, they fail out. Is there any way to bring these back from the dead? Here's some interesting output from journalctl -xeu on the failed OSD: ceph-osd[2383080]: bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) ceph-osd[2383080]: bluestore(/var/lib/ceph/osd/ceph-242) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... ceph-osd[2389725]: bluefs _allocate allocation failed, needed 0x3000 ceph-6ab85342-53d6-11ee-88a7-e43d1a153e91-osd-242[2389718]: -2> 2024-11-25T18:31:42.070+0000 7f0adfdef540 -1 bluefs _flush_range_F allocated: 0x0 offset: 0x0 length: 0x230f ceph-osd[2389725]: bluefs _flush_range_F allocated: 0x0 offset: 0x0 length: 0x230f Followed quickly by an abort: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: In funct> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: 3380: ce> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd7) [0x559bf4361d2f] 2: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x7a9) [0x559bf4b225f9] 3: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2) [0x559bf4b22812] 4: (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e] 5: (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] 6: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] 7: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) [0x559bf51a880a] 8: (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0] 9: (rocksdb::SyncManifest(rocksdb::Env*, rocksdb::ImmutableDBOptions const*, rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] 10: (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, std::allocator<rocksdb::VersionSet::ManifestWriter> >&, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks> 11: (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> const&, rocksdb::autovector<rocksdb::autovector<rocksdb::> 12: (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocksdb::ColumnFamilyOptions const> 13: (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) [0x559bf50bd250] 14: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x13f1) [0x559bf50d3f21] 15: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::Colu> 16: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFa> 17: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x77a) [0x559bf503766a] 18: (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4] 19: (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0] 20: (BlueStore::_mount()+0x396) [0x559bf4a795d6] 21: (OSD::init()+0x556) [0x559bf44a0eb6] 22: main() 23: __libc_start_main() 24: _start() *** Caught signal (Aborted) ** in thread 7f0adfdef540 thread_name:ceph-osd ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f0addff1cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x559bf4361def] 5: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x7a9) [0x559bf4b225f9] 6: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2) [0x559bf4b22812] 7: (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e] 8: (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] 9: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] 10: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) [0x559bf51a880a] 11: (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0] 12: (rocksdb::SyncManifest(rocksdb::Env*, rocksdb::ImmutableDBOptions const*, rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] 13: (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, std::allocator<rocksdb::VersionSet::ManifestWriter> >&, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks> 14: (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> const&, rocksdb::autovector<rocksdb::autovector<rocksdb::> 15: (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocksdb::ColumnFamilyOptions const> 16: (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) [0x559bf50bd250] 17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x13f1) [0x559bf50d3f21] 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::Colu> 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFa> 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x77a) [0x559bf503766a] 21: (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4] 22: (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0] 23: (BlueStore::_mount()+0x396) [0x559bf4a795d6] 24: (OSD::init()+0x556) [0x559bf44a0eb6] 25: main() 26: __libc_start_main() 27: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

5 21

mount path missing for subvolume
by bruno.pessanha＠gmail.com 02 Dec '24

02 Dec '24

I'm using: ``` # ceph version ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) ``` I get the following error when trying to run: # ceph fs subvolume info volume01 vol01 group01 Error ENOENT: mount path missing for subvolume 'vol01' The volume exists and it's mounted on the clients: # ceph fs subvolume ls volume01 group01 [ { "name": "vol02" }, { "name": "vol01" } ] Would you know what could be happening here? Regards, Bruno

1 0

Ceph mirror failing on /archive/el6/x86_64/ceph-0.67.10-0.el6.x86_64.rpm
by Rouven Seifert 02 Dec '24

02 Dec '24

Hi there, we are operating a mirror of downloads.ceph.com that is used as package source for some internal hosts. Recently the rsync script began failing with the following error: rsync: send_files failed to open "/archive/el6/x86_64/ceph-0.67.10-0.el6.x86_64.rpm" (in ceph): Permission denied (13) Who would be the best to report this to? Best regards, Rouven -- Rouven Seifert Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) E-Mail: rouven.seifert(a)agdsn.de https://agdsn.de

1 0

Issue creating LVs within cephadm shell
by Ed Krotee 02 Dec '24

02 Dec '24

Trying to create the Block and Block.DB devices for BlueStore. Within the cephadm shell able to run the vgcreate but get the following errors and we cannot see the vg device in /dev so doesn't seem to actually create the vg, but vgs within the cephadm shell shows it, but vgs at the o/s level does not. FYI - SE Linux is disabled. stdout: Physical volume "/dev/sda" successfully created. Not creating system devices file due to existing VGs. stdout: Volume group "ceph-b80b7206-0c2e-4770-9895-51077b1d59d4" successfully created Running command: lvcreate --yes -l 5245439 -n osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9 ceph-b80b7206-0c2e-4770-9895-51077b1d59d4 stderr: /dev/ceph-b80b7206-0c2e-4770-9895-51077b1d59d4/osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9: not found: device not cleared Aborting. Failed to wipe start of new LV. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 --> RuntimeError: Unable to find any LV for zapping OSD: 0 [ceph: root@ritcephstrdata09 /]# lvcreate --yes -l 5245439 -n osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9 ceph-b80b7206-0c2e-4770-9895-51077b1d59d4 /dev/ceph-b80b7206-0c2e-4770-9895-51077b1d59d4/osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9: not found: device not cleared Aborting. Failed to wipe start of new LV. Any thoughts?

2 1

[Cephadm] Bootstrap Ceph with alternative data directory
by Jinfeng Biao 02 Dec '24

02 Dec '24

We’d like to bootstrap and deploy Ceph into alternative directory. By keeping Ceph relevant data to a separate directory mounted from a separate disk, Ceph services will start right after operating system reprovision. While deploying with –data-dir and other directory parameters, cephadm --data-dir /pd/var/lib/ceph --unit-dir /pd/etc/systemd/system --log-dir /pd/var/log/ceph --logrotate-dir /pd/etc/logrotate.d bootstrap --mon-ip xx.xx.xx.xx The bootstrap failed at mon creation. Ceph version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Non-zero exit code 1 from systemctl disable ceph.target systemctl: stderr Failed to disable unit: Unit file ceph.target does not exist. RuntimeError: Failed command: systemctl disable ceph.target: Failed to disable unit: Unit file ceph.target does not exist. Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 10889, in <module> File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 10877, in main File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 6318, in _rollback File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 2626, in _default_image File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 6463, in command_bootstrap File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 5848, in create_mon File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 3964, in deploy_daemon File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 4191, in deploy_daemon_units File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 4459, in install_base_units File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 2266, in call_throws RuntimeError: Failed command: systemctl disable ceph.target: Failed to disable unit: Unit file ceph.target does not exist. Ceph.target does exist in the directory specified by –unit-dir $ pwd /pd/etc/systemd/system $ ls -al total 4 drwxr-x---. 2 root root 25 Dec 2 04:12 . drwxr-x---. 3 root root 20 Nov 29 06:39 .. -rw-r-----. 1 root root 88 Dec 2 04:12 ceph.target Is this a supported approach and is it a bug? regards James Biao ************** IMPORTANT MESSAGE ************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Opt Out in the subject line. **************************************************

1 0