November 2024 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

constant increase in osdmap epoch
by Frank Schilder 15 Apr '25

15 Apr '25

Hi all, we observe a problem that has been reported before, but I can't find a resolution. This is related to an earlier thread "failed to load OSD map for epoch 2898146, got 0 bytes" (https://www.spinics.net/lists/ceph-users/msg84485.html) We run an octopus latest cluster and observe a constant increase in osdmap epoch every few seconds. There is no change in the contents between two successive epochs: # diff map.3075085.txt map.3075086.txt 1c1 < epoch 3075085 --- > epoch 3075086 4c4 < modified 2024-11-18T15:38:45.512100+0100 --- > modified 2024-11-18T15:38:47.858092+0100 This is exactly what others reported too, for example, "steady increasing of osd map epoch since octopus" (https://www.spinics.net/lists/ceph-users/msg69443.html) Its a real problem since it dramatically shortens the time window an OSD can be down before its latest OSD map is purged from the cluster. This, in turn, leads to serious follow-up problems with OSD restart as reported in the thread I'm referring to at the beginning. Related to that I also see the mgrs increasing the pgmap version constantly every 2 seconds. However, I believe this is intentional. I don't see this redundant pgp_num_actual setting by the mgrs reported here: https://tracker.ceph.com/issues/51433 . I can't find a resolution anywhere. Any help would be very much appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 7

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

Move block.db to new ssd
by Roland Giesler 11 Mar '25

11 Mar '25

I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions. The new 4TB SSD's I have split into volumes with: # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 & osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this: # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami I then take osd4 out of the cluster and stop it. Then I do this: # ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami Thereafter I do this to load the changes: # ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 --path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami After that the configuration change has reverted to what it was before. I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 What am I missing to get these changes to be permanent? regards Roland

6 13

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

Snaptriming speed degrade with pg increase
by Szabo, Istvan (Agoda) 14 Jan '25

14 Jan '25

Hi, When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04) -osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2 We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours. Is there any good solution that we are missing to fix this? On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still. Thank you

4 9

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

EC pool only for hdd
by Rok Jaklič 21 Dec '24

21 Dec '24

Hi, is it possible to set/change following already used rule to only use hdd? { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } Kind regards, Rok

4 15

CRC Bad Signature when using KRBD
by jsterr＠deckplosion.de 19 Dec '24

19 Dec '24

Hello Ceph Users, * Problem: we get the following errors when using krbd, we are using rbd for vms. * Workaround: by switching to librbd the errors disappear. * Software: ** Kernel: 6.8.8-2 (parameters: intel_iommu=on iommu=pt pcie_aspm.policy=performance) ** Ceph: 18.2.2 Description/Details: Errors from using krbd with ceph. Side-effects: [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000015af2284 data crc 1221767919 != exp. 282251377 [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000066b200ab data crc 3817026135 != exp. 3925662391 [Wed Aug 21 03:04:17 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:17 2024] libceph: osd13 (1)10.1.4.13:6809 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000008a131738 data crc 2612835980 != exp. 917302924 [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000005160776b data crc 2965872045 != exp. 563323792 [Wed Aug 21 03:04:21 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: osd6 (1)10.1.4.12:6843 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000007e548354 data crc 1265032637 != exp. 2426281931 [Wed Aug 21 03:06:44 2024] libceph: osd0 (1)10.1.4.11:6835 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000009214d802 data crc 2596010853 != exp. 1221875667 [Wed Aug 21 03:06:44 2024] libceph: osd10 (1)10.1.4.12:6809 bad crc/signature [Wed Aug 21 03:06:47 2024] libceph: read_partial_message 000000000f9edc73 data crc 1326019705 != exp. 3079604517 [Wed Aug 21 03:06:47 2024] libceph: osd3 (1)10.1.4.11:6803 bad crc/signature [Wed Aug 21 03:06:50 2024] libceph: read_partial_message 000000004769da61 data crc 3421275194 != exp. 4183754554 [Wed Aug 21 03:06:50 2024] libceph: osd8 (1)10.1.4.12:6835 bad crc/signature [Wed Aug 21 03:06:51 2024] libceph: read_partial_message 0000000044db9a59 data crc 2603270708 != exp. 4150529351 [Wed Aug 21 03:06:51 2024] libceph: osd14 (1)10.1.4.13:6806 bad crc/signature Description/Details 2: vms get problems with buffer i/o errors on rbd-backed virtual disks: Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=885s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=847s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Add. Sense: I/O process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 CDB: Write(10) 2a 00 34 87 48 08 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 CDB: Write (10) 2a 00 34 81 52 70 00 00 58 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281032 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 880890480 op 0x1: (WRITE) flags 0x103000 phys_seg 11 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: EXT4-fs warning (device sdb1): ext4_end_bio:343: I/0 error 10 writing to inode 27525908 starting Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111054, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: buffer_io_error: 21 callbacks suppressed Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on device sdbl, logical block 110159873 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on dev sdbl, logical block 110111055, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=875s Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111056, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111057, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111058, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 CDB: Write(10) 2a 00 34 87 48 10 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111059, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281040 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111060, lost async page write Thanks for helping out. Greetings Jonas

3 4

Squid: deep scrub issues
by Laimis Juzeliūnas 19 Dec '24

19 Dec '24

Hello Ceph community, Wanted to highlight one observation and gather any Squid users having similar experiences. Since upgrading to 19.2.0 (from 18.4.0) we have observed that pg deep scrubbing times have drastically increased. Some pgs take 2-5 days to complete deep scrubbing while others increase to 20+ days. This causes the deep scrubbing queue to fill up and the cluster almost constantly has 'pgs not deep-scrubbed in time' alerts. We have on average 67 pgs/osd: running on 15TB hdd disks this results in 200GB-ish pgs. While fairly large - these pgs did not cause such increase in deep scrubs when on Reef. "ceph pg dump | grep 'deep scrubbing for'" will always have a few entries of quite morbid scrubs like the following: 7.3e 121289 0 0 0 0 225333247207 0 0 127 0 127 active+clean+scrubbing+deep 2024-11-13T09:37:42.549418+0000 490179'5220664 490179:23902923 [268,27,122] 268 [268,27,122] 268 483850'5203141 2024-11-02T11:33:57.835277+0000 472713'5197481 2024-10-11T04:30:00.639763+0000 0 21873 deep scrubbing for 1169147s 34.247 62618 0 0 0 0 179797964677 0 0 101 50 101 active+clean+scrubbing+deep 2024-11-05T06:27:52.288785+0000 490179'22729571 490179:80672442 [34,97,25] 34 [34,97,25] 34 481331'22436869 2024-10-23T16:06:50.092439+0000 471395'22289914 2024-10-07T19:29:26.115047+0000 0 204864 deep scrubbing for 1871733s Not pointing any fingers but Squid release had "better scrub scheduling" announced. Though this is not scheduling directly, but maybe this change had any impact causing such behaviour? Scrubbing configurations: ceph config get osd | grep scrub global advanced osd_deep_scrub_interval 2678400.000000 global advanced osd_deep_scrub_large_omap_object_key_threshold 500000 global advanced osd_max_scrubs 5 global advanced osd_scrub_auto_repair true global advanced osd_scrub_max_interval 2678400.000000 global advanced osd_scrub_min_interval 172800.000000 Cluster details (backfilling expected and caused by some manual reweights): cluster: id: 96df99f6-fc1a-11ea-90a4-6cb3113cb732 health: HEALTH_WARN 24 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum ceph-node004,ceph-node003,ceph-node001,ceph-node005,ceph-node002 (age 4d) mgr: ceph-node001.hgythj(active, since 11d), standbys: ceph-node002.jphtvg mds: 20/20 daemons up, 12 standby osd: 384 osds: 384 up (since 25h), 384 in (since 5d); 5 remapped pgs rbd-mirror: 2 daemons active (2 hosts) rgw: 64 daemons active (32 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 8681 pgs objects: 758.42M objects, 1.5 PiB usage: 4.6 PiB used, 1.1 PiB / 5.7 PiB avail pgs: 275177/2275254543 objects misplaced (0.012%) 6807 active+clean 989 active+clean+scrubbing+deep 880 active+clean+scrubbing 5 active+remapped+backfilling io: client: 37 MiB/s rd, 59 MiB/s wr, 1.72k op/s rd, 439 op/s wr recovery: 70 MiB/s, 38 objects/s One thread of other users experiencing same 19.2.0 prolonged deep scrub issues: https://www.reddit.com/r/ceph/comments/1guynak/strange_issue_where_scrubdee… Any hints or help would be greately appreciated! Thanks in advance, Laimis J. laimis.juzeliunas(a)oxylabs.io

6 13

Re: Squid Manager Daemon: balancer crashing orchestrator and dashboard
by Alexander Closs 18 Dec '24

18 Dec '24

Just chiming in to say this also affected our cluster, same symptoms and a temporary fix of disabling the balancer. Happy to add my cluster's logs to the issue, though I suspect they'll look the same as Laimis' cluster. -Alex MIT CSAIL

7 7

failed to load OSD map for epoch 2898146, got 0 bytes
by Frank Schilder 15 Dec '24

15 Dec '24

Hi all, I have a strange problem on an octopus latest cluster. We had a couple of SSD OSDs down for a while and brought them up today again. For some reason, these OSDs don't come up and flood the log with messages like osd.1004 2971464 failed to load OSD map for epoch 2898146, got 0 bytes These messages cycle through the same epochs over and over again. I did not really fine too much help, there is an old thread about a similar/the same problem on a home lab cluster, with new OSDs though, I believe. I couldn't really find useful information. The OSDs seem to boot fine and then end up in something like a death loop. Below some snippets from the OSD log. Any hints appreciated. Thanks and best regards, Frank After OSD start, everything looks normal up to here: 2024-10-21T17:41:39.136+0200 7fad73cf6f00 0 osd.1004 2971464 load_pgs opened 205 pgs 2024-10-21T17:41:39.140+0200 7fad73cf6f00 -1 osd.1004 2971464 log_to_monitors {default=true} 2024-10-21T17:41:39.150+0200 7fad73cf6f00 -1 osd.1004 2971464 mon_cmd_maybe_osd_create fail: 'osd.1004 has already bound to class 'fs_meta', can not reset class to 'ssd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy 2024-10-21T17:41:39.155+0200 7fad519a3700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898134, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898135, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898136, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4f99f700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4f99f700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898134, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898135, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898136, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898137, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad73cf6f00 0 osd.1004 2971464 done with init, starting boot process 2024-10-21T17:41:39.155+0200 7fad73cf6f00 1 osd.1004 2971464 start_boot 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898138, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898139, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898140, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898141, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898142, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898143, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898144, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898145, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898146, got 0 bytes These messages repeat over and over again with some others of this form showing up every now and then: 2024-10-21T17:41:39.476+0200 7fad651ca700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 12] Generated table #82879: 76571 keys, 67866714 bytes 2024-10-21T17:41:39.688+0200 7fad651ca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1729525299690000, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 82879, "file_size": 67866714, "table_properties": {"data_size": 67111697, "index_size": 562601, "filter_size": 191557, "raw_key_size": 4823973, "raw_average_key_size": 63, "raw_value_size": 62631087, "raw_average_value_size": 817, "num_data_blocks": 15644, "num_entries": 76571, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} And another occasion: 2024-10-21T17:41:40.520+0200 7fad651ca700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 12] Generated table #82880: 76774 keys, 67868330 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899234, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899235, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899236, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad651ca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1729525300521403, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 82880, "file_size": 67868330, "table_properties": {"data_size": 67113021, "index_size": 562509, "filter_size": 191941, "raw_key_size": 4836742, "raw_average_key_size": 62, "raw_value_size": 62623274, "raw_average_value_size": 815, "num_data_blocks": 15630, "num_entries": 76774, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

6 12

Single unfound object in cluster with no previous version - is there anyway to recover rather than deleting the object?
by Ivan Clayson 10 Dec '24

10 Dec '24

Hello, We have an Alma8.9 (version 4 kernel) quincy (17.2.7) CephFS cluster with spinners for our bulk data and SSDs for the metadata where we have a single unfound object in the bulk pool: [root@ceph-n30 ~]# ceph -s cluster: id: fa7cf62b-e261-49cd-b00e-383c36b79ef3 health: HEALTH_ERR 1/849660811 objects unfound (0.000%) Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 9/8468903874 objects degraded (0.000%), 1 pg degraded services: mon: 3 daemons, quorum ceph-s2,ceph-s3,ceph-s1 (age 44h) mgr: ceph-s2(active, since 45h), standbys: ceph-s3, ceph-s1 mds: 1/1 daemons up, 3 standby osd: 439 osds: 439 up (since 43h), 439 in (since 43h); 176 remapped pgs data: volumes: 1/1 healthy pools: 9 pools, 4321 pgs objects: 849.66M objects, 2.3 PiB usage: 3.0 PiB used, 1.7 PiB / 4.6 PiB avail pgs: 9/8468903874 objects degraded (0.000%) 36630744/8468903874 objects misplaced (0.433%) 1/849660811 objects unfound (0.000%) 4122 active+clean 174 active+remapped+backfill_wait 22 active+clean+scrubbing+deep 2 active+remapped+backfilling 1 active+recovery_unfound+degraded io: client: 669 MiB/s rd, 87 MiB/s wr, 302 op/s rd, 77 op/s wr recovery: 175 MiB/s, 59 objects/s [root@ceph-n30 ~]# ceph health detail | grep unfound HEALTH_ERR 1/849661114 objects unfound (0.000%); Possible data damage: 1 pg recovery_unfound; Degraded data redundancy: 9/8468906904 objects degraded (0.000%), 1 pg degraded [WRN] OBJECT_UNFOUND: 1/849661114 objects unfound (0.000%) pg 2.c90 has 1 unfound objects [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound pg 2.c90 is active+recovery_unfound+degraded, acting [259,210,390,209,43,66,322,297,25,374], 1 unfound pg 2.c90 is active+recovery_unfound+degraded, acting [259,210,390,209,43,66,322,297,25,374], 1 unfound We've tried deep-scrubbing and repairing the PG as well as rebooting the entire cluster but unfortunately this has not resolved our issue. The primary OSD (259) log reports that our 1009e1df26d.000000c9 object is missing where when we do rados commands on the object that command just hangs: [root@ceph-n30 ~]# grep 2.c90 /var/log/ceph/ceph-osd.259.log ... 2024-11-25T11:38:33.860+0000 7fd409870700 1 osd.259 pg_epoch: 512353 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512348/512349 n=211842 ec=1175/1168 lis/c=512348/472766 les/c/f=512349/472770/232522 sis=512353 pruub=11.010143280s) [259,210,390,209,43,66,322,297,NONE,374]p259(0) r=0 lpr=512353 pi=[472766,512353)/11 crt=512310'8145216 mlcod 0'0 unknown pruub 205.739364624s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:38:54.926+0000 7fd409870700 1 osd.259 pg_epoch: 512356 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512353/512354 n=211842 ec=1175/1168 lis/c=512353/472766 les/c/f=512354/472770/232522 sis=512356 pruub=11.945847511s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active pruub 227.741577148s@ m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(0+0)=1},9={(1+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,2147483647,374] -> [259,210,390,209,43,66,322,297,25,374], acting [259,210,390,209,43,66,322,297,2147483647,374] -> [259,210,390,209,43,66,322,297,25,374], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:38:54.926+0000 7fd409870700 1 osd.259 pg_epoch: 512356 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512353/512354 n=211842 ec=1175/1168 lis/c=512353/472766 les/c/f=512354/472770/232522 sis=512356 pruub=11.945847511s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 unknown pruub 227.741577148s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:38:59.910+0000 7fd409870700 0 osd.259 pg_epoch: 512359 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512356) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] get_remaining_shards not enough shards left to try for 2:0930c16c:::1009e1df26d.000000c9:head read result was read_result_t(r=0, errors={25(8)=-2,43(4)=-2,66(5)=-2,209(3)=-2,210(1)=-2,297(7)=-2,322(6)=-2,390(2)=-2}, noattrs, returned=(0, 8388608, [])) 2024-11-25T11:38:59.911+0000 7fd409870700 0 osd.259 pg_epoch: 512359 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512356) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512356 pi=[472766,512356)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] on_failed_pull 2:0930c16c:::1009e1df26d.000000c9:head from shard 25(8),43(4),66(5),209(3),210(1),297(7),322(6),390(2), reps on 374(9) unfound? 1 2024-11-25T11:39:01.435+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost 2024-11-25T11:39:02.193+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost 2024-11-25T11:39:03.590+0000 7fd409870700 1 osd.259 pg_epoch: 512362 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512362 pruub=8.352689743s) [259,210,390,209,43,66,322,297,25,NONE]p259(0) r=0 lpr=512362 pi=[472766,512362)/11 crt=512310'8145216 mlcod 0'0 active pruub 232.812744141s@ m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,25,374] -> [259,210,390,209,43,66,322,297,25,2147483647], acting [259,210,390,209,43,66,322,297,25,374] -> [259,210,390,209,43,66,322,297,25,2147483647], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:39:03.591+0000 7fd409870700 1 osd.259 pg_epoch: 512362 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512356/512357 n=211842 ec=1175/1168 lis/c=512356/472766 les/c/f=512357/472770/232522 sis=512362 pruub=8.352689743s) [259,210,390,209,43,66,322,297,25,NONE]p259(0) r=0 lpr=512362 pi=[472766,512362)/11 crt=512310'8145216 mlcod 0'0 unknown pruub 232.812744141s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:39:24.954+0000 7fd409870700 1 osd.259 pg_epoch: 512365 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512362/512363 n=211842 ec=1175/1168 lis/c=512362/472766 les/c/f=512363/472770/232522 sis=512365 pruub=11.731550217s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active pruub 257.554870605s@ m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(0+0)=1}}] start_peering_interval up [259,210,390,209,43,66,322,297,25,2147483647] -> [259,210,390,209,43,66,322,297,25,374], acting [259,210,390,209,43,66,322,297,25,2147483647] -> [259,210,390,209,43,66,322,297,25,374], acting_primary 259(0) -> 259, up_primary 259(0) -> 259, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 2024-11-25T11:39:24.954+0000 7fd409870700 1 osd.259 pg_epoch: 512365 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512362/512363 n=211842 ec=1175/1168 lis/c=512362/472766 les/c/f=512363/472770/232522 sis=512365 pruub=11.731550217s) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 unknown pruub 257.554870605s@ m=1 mbc={}] state<Start>: transitioning to Primary 2024-11-25T11:39:30.679+0000 7fd409870700 0 osd.259 pg_epoch: 512368 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512365/512366 n=211842 ec=1175/1168 lis/c=512365/472766 les/c/f=512366/472770/232522 sis=512365) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 mbc={0={(0+0)=1},1={(1+0)=1},2={(1+0)=1},3={(1+0)=1},4={(1+0)=1},5={(1+0)=1},6={(1+0)=1},7={(1+0)=1},8={(1+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] get_remaining_shards not enough shards left to try for 2:0930c16c:::1009e1df26d.000000c9:head read result was read_result_t(r=0, errors={25(8)=-2,43(4)=-2,66(5)=-2,209(3)=-2,210(1)=-2,297(7)=-2,322(6)=-2,390(2)=-2}, noattrs, returned=(0, 8388608, [])) 2024-11-25T11:39:30.679+0000 7fd409870700 0 osd.259 pg_epoch: 512368 pg[2.c90s0( v 512310'8145216 lc 0'0 (511405'8142151,512310'8145216] local-lis/les=512365/512366 n=211842 ec=1175/1168 lis/c=512365/472766 les/c/f=512366/472770/232522 sis=512365) [259,210,390,209,43,66,322,297,25,374]p259(0) r=0 lpr=512365 pi=[472766,512365)/10 crt=512310'8145216 mlcod 0'0 active+recovering+degraded rops=1 m=1 u=1 mbc={0={(0+0)=1},1={(0+0)=1},2={(0+0)=1},3={(0+0)=1},4={(0+0)=1},5={(0+0)=1},6={(0+0)=1},7={(0+0)=1},8={(0+0)=1},9={(1+0)=1}} trimq=[13f6e~134]] on_failed_pull 2:0930c16c:::1009e1df26d.000000c9:head from shard 25(8),43(4),66(5),209(3),210(1),297(7),322(6),390(2), reps on 374(9) unfound? 1 2024-11-25T11:40:11.652+0000 7fd409870700 -1 log_channel(cluster) log [ERR] : 2.c90 has 1 objects unfound and apparently lost [root@ceph-n30 ~]# rados -p ec82pool stat 1009e1df26d.000000c9 ... # hanging indefinitely Similarly if we do any object store commands, then we get crashes and seg faults as well: [root@ceph-n25 ~]# ceph pg 2.c90 list_unfound { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "1009e1df26d.000000c9", "key": "", "snapid": -2, "hash": 914558096, "max": 0, "pool": 2, "namespace": "" }, "need": "510502'8140206", "have": "0'0", "flags": "none", "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", "locations": [ "374(9)" ] } ], "state": "NotRecovering", "available_might_have_unfound": true, "might_have_unfound": [], "more": false } [root@ceph-n11 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-374 --debug --pgid 2.c90 1009e1df26d.000000c9 dump ... -6> 2024-11-25T15:54:57.387+0000 7f8b6a0a7340 1 bluestore(/var/lib/ceph/osd/ceph-374) _upgrade_super from 4, latest 4 -5> 2024-11-25T15:54:57.387+0000 7f8b6a0a7340 1 bluestore(/var/lib/ceph/osd/ceph-374) _upgrade_super done -4> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 prioritycache tune_memory target: 4294967296 mapped: 132694016 unmapped: 618364928 heap: 751058944 old mem: 134217728 new mem: 2761652735 -3> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0540541 -2> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 prioritycache tune_memory target: 4294967296 mapped: 132939776 unmapped: 618119168 heap: 751058944 old mem: 2761652735 new mem: 2842823159 -1> 2024-11-25T15:54:57.439+0000 7f8b5d925700 5 bluestore.MempoolThread(0x55cd3a807b40) _resize_shards cache_size: 2842823159 kv_alloc: 1241513984 kv_used: 2567024 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 1174405120 meta_used: 13360 data_alloc: 218103808 data_u sed: 0 0> 2024-11-25T15:54:57.445+0000 7f8b6a0a7340 -1 *** Caught signal (Segmentation fault) ** in thread 7f8b6a0a7340 thread_name:ceph-objectstor ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8b67941cf0] 2: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x4c) [0x55cd37e34f5c] 3: (_action_on_all_objects_in_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x13b4) [0x55cd377fdf64] 4: (action_on_all_objects_in_exact_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x64) [0x55cd377fe274] 5: main() 6: __libc_start_main() 7: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. We don't have a previous verison of this object and trying to fix-lost with the object store command seg faults: [root@ceph-n11 ~] ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-374 --pgid 2.c90 --op fix-lost --dry-run *** Caught signal (Segmentation fault) ** in thread 7f45d9890340 thread_name:ceph-objectstor ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f45d712acf0] 2: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x4c) [0x5556f3c36f5c] 3: (_action_on_all_objects_in_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x13b4) [0x5556f35fff64] 4: (action_on_all_objects_in_exact_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x64) [0x5556f3600274] 5: main() 6: __libc_start_main() 7: _start() Segmentation fault (core dumped) The recommended solution seems to be to use "mark_unfound_lost revert" but for our object, there is no previous version so I think this command will discard and delete the object. The lost object is on our live filesystem and there seems to be no easily way to find the backup version as I can't access the path and or filename associated with the object to recover it from the backup. Is there any way for us to recover this object without discarding it? Or should we just accept our losses and delete it? Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N269 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

2 2

Replacing Ceph Monitors for Openstack
by Adrien Georget 09 Dec '24

09 Dec '24

Hello, We are using Ceph as a storage backend for Openstack (Cinder, Nova, Glance, Manila) and we are replacing old hardware hosting Ceph monitors (MON,MGR,MDS) to new ones. I have already added the new ones in production, monitors successfully joined the quorum and new MGR/MDS are standby. For the monitors, I'm sure that the monmap is already up to date and Openstack clients are already aware of the change and it should not be a problem when I will next shut down the old monitors. The ceph.conf will be updated in all Openstack controllers to replace "mon host" with the new ones before shutting old mons down. But I have some doubts with the resilience of Openstack Manila service because the IP addresses of the monitors look hardcoded in the export location of the manila share : The manila show command returns for example : | export_locations | | | | path = 134.158.208.140:6789,134.158.208.141:6789,134.158.208.142:6789:/volumes/EC_manila/_nogroup/7a6c05d9-2fea-43b1-a6d4-06eec1e384f2 | | | share_instance_id = 7a6c05d9-2fea-43b1-a6d4-06eec1e384f2 | Has anyone already done this kind of migration in the past and can confirm my doubts? Is there any process to update shares? Cheers, Adrien

4 4

Slow ops during index pool recovery causes cluster performance drop to 1%
by Szabo, Istvan (Agoda) 07 Dec '24

07 Dec '24

Hi, I'm updating from octopus to quincy and all in our cluster when index pool recovery kicks off, cluster operation drops to 1%, slow ops comes non-stop. The recovery takes 1-2 hours/nodes. What I can see the iowait on the nvme drives which belongs to the index pool is pretty high, however the throughput is less than 500MB/s, the iops is less than 5000/sec. The index pool is a 3:2 replica pool with 2048pg on 156 osd (1 nvme drive has 4 osds due to we experienced latency issue with 1 or 2 osd/nvme). If we consider let's say the nvme drive still slow with these really small load, how would that be possible to somehow ease and get rid of this cluster performance drop? If I increase replica to 4-5 would that help? It could tolerate more pg slowness maybe? FYI we have many objects in our cluster, more than 4Billions: objects: 4.06G objects, 616 TiB However I think it should still tolerate cluster recovery without penalty. What I can see in the slow osd log with default debug value is about "get_health_metrics" so far : 2024-11-02T12:38:40.762+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 6 slow requests (by type [ 'delayed' : 6 ] most affected pool [ 'hkg.rgw.buckets.index' : 6 ]) 2024-11-02T12:38:41.802+0700 7f241bc25640 -1 osd.110 626281 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:41.802+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) 2024-11-02T12:38:42.782+0700 7f241bc25640 -1 osd.110 626282 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:42.782+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) 2024-11-02T12:38:43.802+0700 7f241bc25640 -1 osd.110 626282 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.3641786447.0:2194661324 26.588 26:11aa561a:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.503457179.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e626262) 2024-11-02T12:38:43.802+0700 7f241bc25640 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'hkg.rgw.buckets.index' : 7 ]) How we also try to make it smoother, after update and machine reboot compaction kicks off which generates 30-40 iowait on the node, we prevent with "noup" flag to put these osds into the cluster until compaction finished, however when we have 0 iowait after compaction, I unset noup so recovery can start which causes the above issue. If I wouldn't set noup it would cause even bigger issue. Thank you for help ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 15

Dump/Add users yaml/json
by Albert Shih 06 Dec '24

06 Dec '24

Hi everyone, Stupid question....after some test I was able to dump a user caps with ceph auth get --format json but I'wasn't able to find the other way, something ceph auth add fubar.json Is they are any way to add a user (without given a key and with a key). Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 29 nov. 2024 18:12:35 CET

4 7

down OSDs, Bluestore out of space, unable to restart
by John Jasen 02 Dec '24

02 Dec '24

Ceph version 17.2.6 After a power loss event affecting my ceph cluster, I've been putting humpty dumpty back together since. One problem I face is that with objects degraded, rebalancing doesn't run -- and this resulted in several of my fast OSDs filling up. I have 8 OSDs currently down, 100% full (exceeding all the full ratio settings on by default or I toggled to try and keep it together), and when I try to restart them, they fail out. Is there any way to bring these back from the dead? Here's some interesting output from journalctl -xeu on the failed OSD: ceph-osd[2383080]: bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) ceph-osd[2383080]: bluestore(/var/lib/ceph/osd/ceph-242) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... ceph-osd[2389725]: bluefs _allocate allocation failed, needed 0x3000 ceph-6ab85342-53d6-11ee-88a7-e43d1a153e91-osd-242[2389718]: -2> 2024-11-25T18:31:42.070+0000 7f0adfdef540 -1 bluefs _flush_range_F allocated: 0x0 offset: 0x0 length: 0x230f ceph-osd[2389725]: bluefs _flush_range_F allocated: 0x0 offset: 0x0 length: 0x230f Followed quickly by an abort: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: In funct> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: 3380: ce> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd7) [0x559bf4361d2f] 2: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x7a9) [0x559bf4b225f9] 3: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2) [0x559bf4b22812] 4: (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e] 5: (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] 6: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] 7: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) [0x559bf51a880a] 8: (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0] 9: (rocksdb::SyncManifest(rocksdb::Env*, rocksdb::ImmutableDBOptions const*, rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] 10: (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, std::allocator<rocksdb::VersionSet::ManifestWriter> >&, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks> 11: (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> const&, rocksdb::autovector<rocksdb::autovector<rocksdb::> 12: (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocksdb::ColumnFamilyOptions const> 13: (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) [0x559bf50bd250] 14: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x13f1) [0x559bf50d3f21] 15: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::Colu> 16: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFa> 17: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x77a) [0x559bf503766a] 18: (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4] 19: (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0] 20: (BlueStore::_mount()+0x396) [0x559bf4a795d6] 21: (OSD::init()+0x556) [0x559bf44a0eb6] 22: main() 23: __libc_start_main() 24: _start() *** Caught signal (Aborted) ** in thread 7f0adfdef540 thread_name:ceph-osd ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f0addff1cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x559bf4361def] 5: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x7a9) [0x559bf4b225f9] 6: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2) [0x559bf4b22812] 7: (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e] 8: (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] 9: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] 10: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) [0x559bf51a880a] 11: (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0] 12: (rocksdb::SyncManifest(rocksdb::Env*, rocksdb::ImmutableDBOptions const*, rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] 13: (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, std::allocator<rocksdb::VersionSet::ManifestWriter> >&, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks> 14: (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> const&, rocksdb::autovector<rocksdb::autovector<rocksdb::> 15: (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocksdb::ColumnFamilyOptions const> 16: (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) [0x559bf50bd250] 17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x13f1) [0x559bf50d3f21] 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::Colu> 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFa> 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x77a) [0x559bf503766a] 21: (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4] 22: (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0] 23: (BlueStore::_mount()+0x396) [0x559bf4a795d6] 24: (OSD::init()+0x556) [0x559bf44a0eb6] 25: main() 26: __libc_start_main() 27: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

5 21

Issue creating LVs within cephadm shell
by Ed Krotee 02 Dec '24

02 Dec '24

Trying to create the Block and Block.DB devices for BlueStore. Within the cephadm shell able to run the vgcreate but get the following errors and we cannot see the vg device in /dev so doesn't seem to actually create the vg, but vgs within the cephadm shell shows it, but vgs at the o/s level does not. FYI - SE Linux is disabled. stdout: Physical volume "/dev/sda" successfully created. Not creating system devices file due to existing VGs. stdout: Volume group "ceph-b80b7206-0c2e-4770-9895-51077b1d59d4" successfully created Running command: lvcreate --yes -l 5245439 -n osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9 ceph-b80b7206-0c2e-4770-9895-51077b1d59d4 stderr: /dev/ceph-b80b7206-0c2e-4770-9895-51077b1d59d4/osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9: not found: device not cleared Aborting. Failed to wipe start of new LV. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 --> RuntimeError: Unable to find any LV for zapping OSD: 0 [ceph: root@ritcephstrdata09 /]# lvcreate --yes -l 5245439 -n osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9 ceph-b80b7206-0c2e-4770-9895-51077b1d59d4 /dev/ceph-b80b7206-0c2e-4770-9895-51077b1d59d4/osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9: not found: device not cleared Aborting. Failed to wipe start of new LV. Any thoughts?

2 1

classes crush rules new cluster
by Marc 29 Nov '24

29 Nov '24

It looks like it is not possible to create crush rules when you don't have harddrives active in this class. I am testing with new squid and did not add ssd's yet, eventhough I added class like this. ceph osd crush class create ssd I can't execute this ceph osd crush rule create-replicated replicated_ssd default host ssd Is there any way around this?

4 8

new cluser ceph osd perf = 0
by Marc 29 Nov '24

29 Nov '24

My ceph osd perf are all 0, do I need to enable module for this? osd_perf_query? Where should I find this in manuals? Or do I just need to wait? [@ target]# ceph osd perf osd commit_latency(ms) apply_latency(ms) 25 0 0 24 0 0 23 0 0 22 0 0 21 0 0 20 0 0 19 0 0 18 0 0

4 3

Additional rgw pool
by Rok Jaklič 29 Nov '24

29 Nov '24

Hi, we are already running the "default" rgw pool with some users. Data is stored in pool: pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 309346 lfor 0/127784/214408 flags hashpspool,ec_overwrites,backfillfull stripe_width 12288 application rgw Is it possible to create another rgw pool with diferent ec profile and associate those pools to specific s3 users? Kind regards, Rok

3 2

nfs-ganesha 5 changes
by Marc 29 Nov '24

29 Nov '24

In my old environment I have simple nfs-ganesha export like this, which is sufficent and mounts. EXPORT { Export_Id = 200; Path = /backup; Pseudo = /backup; FSAL { Name = CEPH; Filesystem = "xxxx"; User_Id = "cephfs.xxxx.bakup"; Secret_Access_Key = "xxxxx=="; } Disable_ACL = FALSE; CLIENT { Clients = 192.168.11.200; access_type = "RW"; } CLIENT { Clients = *; Access_Type = NONE; } } In the new ganesha 5 I am getting these errors. Don't really get why it wants to create a pool rados_kv_connect :CLIENT ID :EVENT :Failed to create pool: -34 rados_ng_init :CLIENT ID :EVENT :Failed to connect to cluster: -34 main :NFS STARTUP :CRIT :Recovery backend initialization failed! cephfs kernel mount with this userid is ok. User only has access to this dir. Anyone an idea what config I need to update?

2 3

internal communication network
by Michel Niyoyita 29 Nov '24

29 Nov '24

Hello team , I am creating new cluster which will be created using CEPHADM, I will use 192.168.1.0/24 as public network and 10.10.90.0/24 as internal network for osd , mon communication . would like if this command is helpful as it is my first time to use caphadm . sudo cephadm bootstrap --mon-ip 192.168.1.23 --cluster-network 10.10.90.0/24 . Kindly help me if there is alternative. Best regards Michel

3 3

[CephFS] Completely exclude some MDS rank from directory processing
by Александр Руденко 29 Nov '24

29 Nov '24

Hi, I try to distribute all top level dirs in CephFS by different MDS ranks. I have two active MDS with rank *0* and *1 *and I have 2 top dirs like */dir1* and* /dir2*. After pinning: setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir1 setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir2 I can see next INOS and DNS distribution: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active c Reqs: 127 /s 12.6k 12.5k 333 505 1 active b Reqs: 11 /s 21 24 19 1 When I write to dir1 I can see a small amount on Reqs: in rank 1. Events in journal of MDS with rank 1: cephfs-journal-tool --rank=fs1:1 event get list 2024-11-20T12:24:42.045056+0300 0xc5c1cb UPDATE: (scatter_writebehind) A2037D53 2024-11-20T12:24:46.935934+0300 0xc5c629 SESSION: () 2024-11-20T12:24:47.192012+0300 0xc5c7cd UPDATE: (lock inest accounted scatter stat update) 2024-11-20T12:24:47.904717+0300 0xc5ca0b SESSION: () 2024-11-20T12:26:46.912719+0300 0xc5ca98 SESSION: () 2024-11-20T12:26:47.910806+0300 0xc5cc3c SESSION: () 2024-11-20T12:27:35.746239+0300 0xc5ccc9 SESSION: () 2024-11-20T12:28:46.923812+0300 0xc5ce63 SESSION: () 2024-11-20T12:28:47.903066+0300 0xc5d007 SESSION: () 2024-11-20T12:29:08.063326+0300 0xc5d094 EXPORT: () di1/A2037D53 2024-11-20T12:30:46.909621+0300 0xc5d96f SESSION: () 2024-11-20T12:30:47.908050+0300 0xc5db13 SESSION: () 2024-11-20T12:32:46.907649+0300 0xc5dba0 SESSION: () 2024-11-20T12:32:47.905962+0300 0xc5dd44 SESSION: () 2024-11-20T12:34:44.349348+0300 0xc5ddd1 SESSIONS: () But the main problem, when I stop MDS rank 1 (without any kind of standby) - FS hangs for all actions. Is this correct? Is it possible to completely exclude rank 1 from processing dir1 and not stop io when rank 1 goes down?

4 19

2024-11-28 Perf Meeting Cancelled
by Matt Vandermeulen 28 Nov '24

28 Nov '24

Hi folks, the perf meeting for today will be cancelled for US thanksgiving! As a heads up, next week will also be cancelled for Cephalocon. Thanks, Matt

1 0

rgw multisite excessive data usage on secondary zone
by Adam Prycki 28 Nov '24

28 Nov '24

Hi, I've just configured a second zones for 2 of our ceph s3 deployments and I've noticed that after initial sync secondary zone data pool are much bigger than ones on master zones. My setup consists of main zone, archive zone and sync_policy which configure directional sync from main zone to archive. here is an example from the zone I've configured today. Master zone looks like this X.rgw.buckets.data 42 1024 361 GiB 254.26k 542 GiB 0.01 3.1 PiB secondary archive zone looks like this. X-archive.rgw.buckets.data 32 32 755 GiB 716.68k 1007 GiB 0.03 2.4 PiB This archive zone was created few hours ago. Users didn't overwrite so much data to double archive zone size. (object count is almost trippled) I've checked gc list --include-all on archive zones and it's empty. I'm not sure why zone is this big. Few days ago I've also configured archive zone for different deployment. I've configured archive-zone lifecycle policy to 1 day and tried to cleanup all the buckets on archive zone. It didn't help. My other archive zone is 150% of the size of it's master zone. I've tried to force sync with `radosgw-admin data sync init` Sync worked but didn't help with excess data on the pool. I suspect it's an error during initial multisite synchronization. I've restarted RGW daemon on archive zone during initial synchronization in both cases. What else could have caused this? Are RGW daemons in multisite setups sensitive to restarts? Could similar issues happen during normal rgw restart during multisite operations? Best regards Adam Prycki

1 0

Migrated to cephadm, rgw logs to file even when rgw_ops_log_rados is true
by Paul JURCO 28 Nov '24

28 Nov '24

Hi, we recently migrated to cephadm from ceph-deploy a 18.2.2 ceph cluster (Ubuntu with docker). RGWs are separate vms. We noticed syslog increased a lot due to rgw's access logs sent to it. And because we use to log ops, a huge ops log file on /var/log/ceph/cluster-id/ops-log-ceph-client.rgw.hostname-here.log. While having "rgw_ops_log_rados": "true", oplogs goes to both file and rados pool for logs. If false it doesn't log anything, as expected. How to stop dockered rgws to log to syslog and to a file on disk, but to keep opslog in logs pool? Config is: global basic log_to_journald false global advanced rgw_enable_ops_log false global advanced rgw_ops_log_rados true A few hours later after after enabling it back, after massive cleanup, it does logging ops but only to files. How to get ops logs in rados pool and access log to a file on disk but not on syslog? I have add this to daemon.json to limit access logs to accumulate on /var/log/docker/containers/rand/rand/json.log file: { "log-driver": "local", "log-opts": { "max-size": "512m", "max-file": "3" } } Thank you! Paul

2 3

testing with tcmu-runner vs rbd map
by Marc 27 Nov '24

27 Nov '24

I know this is not a good test, but when I dd to a rbd image like this [@ ~]# dd if=/dev/zero of=/dev/rbd0 bs=1M count=100 status=progress 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.399629 s, 262 MB/s It is a big cache difference doing this via the tcmu-runner [@~]# dd if=/dev/zero of=/dev/sdac bs=1M count=100 status=progress 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3571 s, 10.1 MB/s Is there anything I can change so the default config of tcmu-runner behaves like the rbdmap?

1 0

Ceph Nautilus packages for ubuntu 20.04
by Pardhiv Karri 27 Nov '24

27 Nov '24

Hi, I am in a tricky situation. Our current OSD nodes (luminous version) are on the latest Dell servers, which only support Ubuntu 20.04. The Luminous packages were installed on 16.04, so the packages are still Xenial; I later upgraded the OS to 20.04 and added OSDs to the cluster. Now, I am trying to upgrade to Nautilus while on Ubuntu 20.04, but there are no Nautilus packages for 20.04. What is the best way to upgrade OSD nodes to Nautilus on 20.04? The Mon nodes have already been upgraded to Nautilus, as they are still on Ubuntu 16.04 (older Dell nodes). Thanks, Pardhiv

3 2

ceph cluster planning size / disks
by Marc 27 Nov '24

27 Nov '24

I was wondering if there is some online tool that can help you with calculating usable storage from nodes, drives per node, replication/erasure used etc.

7 10

Balancer: Unable to find further optimization
by sinan＠turka.nl 27 Nov '24

27 Nov '24

Hi, My Ceph cluster is out-of-balance. The amount of PG's per OSD ranges from about 50 up to 100 PG's per OSD. This is far from balanced. Today, I have enabled the balancer module. But unfortunately, it doesn't want to balance: "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect" The upmap_max_deviation setting was set to default (5). Changing this to 1, triggered the balancer module. It did 1 run, balanced a few PG's (less than 5 PG's in total), then again it got stuck with the same message: "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect" I have about 280 OSD's: 24 OSD's per node, 4 nodes per rack, 3 racks in total. Replica = 3. 1 replica per rack. My disk sizes differs from 1.6T up to 2.4T. Any advice on this? Thanks! Sinan

2 4

iscsi-ceph
by Marc 27 Nov '24

27 Nov '24

Failed: Clients can not be defined until a HA configuration has been defined (>2 gateways) Who cares when I am testing???? I am fully aware I only entered 1.

1 0

iscsi testing
by Marc 27 Nov '24

27 Nov '24

what is the best location to get tcmu-runner rpms from? (Does not seem to be available in el9) create ceph-gw-1 10.172.19.21 skipchecks=true returns: The first gateway defined must be the local machine I can only put here full domainname not even only the hostname, does not seem to match the manual.

1 0

macos rbd client
by Marc 27 Nov '24

27 Nov '24

Does it make sense to try and see if I can connect macos client to an rbd device, or is this never going to be a stable supported environment? Are people doing this?

1 0

CephFS 16.2.10 problem
by Alexey.Tsivinsky＠baikalelectronics.ru 27 Nov '24

27 Nov '24

Good evening! The following problem occurred. There is a cluster ceph 16.2.10 The cluster was operating normally on Friday. Shut down cluster: -Excluded all clients Executed commands: ceph osd set noout ceph osd set nobackfill ceph osd set norecover ceph osd set norebalance ceph osd set nodown ceph osd set pause Turned off the cluster, checked server maintenance. Enabled cluster. He gathered himself, found all the nodes, and here the problem began. After all OSD went up and all pg became available, cephfs refused to start. Now mds are in the replay status, and do not go to the ready status. Previously, one of them was in the replay (laggy) status, but we executed command: ceph config set mds mds_wipe_sessions true After that, mds switched to the status of replays, the third in standby status started, and mds crashes with an error stopped. But cephfs is still unavailable. What else can we do? The cluster is very large, almost 200 million files. Best regards A.Tsivinsky e-mail: alexey.tsivinsky(a)baikalelectronics.com

3 8

config set -> ceph.conf
by Marc 27 Nov '24

27 Nov '24

How should I rewrite this to ceph.conf ceph config set mon mon_warn_on_insecure_global_id_reclaim false ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false

2 1

Upgrade of OS and ceph during recovery
by Rok Jaklič 27 Nov '24

27 Nov '24

Hi, right now the cluster is doing recovery for the last two weeks and it seems it will be doing so for the next week or so also. Meanwhile a new quincy update came, which fixes some of the things for us but we would need to upgrade to AlmaLinux 9. Has anyone done maintainace or upgrade of nodes during recovery? Kind regards, Rok

1 0

CephFS empty files in a Frankenstein system
by Linas Vepstas 27 Nov '24

27 Nov '24

Don't laugh. I am experimenting with Ceph in an enthusiast, small-office, home-office setting. Yes, this is not the conventional use case, but I think Ceph almost is, almost could be used for this. Do I need to explain why? These kinds of people (i.e. me) already run RAID. And maybe CIFS/Samba or NFS. On half-a-dozen or more machines. And RAID is OK, I guess, as long as your mirrors are in sync. And if you have three disks in a mirror, do they vote as to who has the best data? Not mdraid, no. And fsck takes hours on a multi-terabyte system with old disks. There's no CRC on data, just metadata. Maybe there's silent corruption in the background. Who knows. And being admin for CIFS/NFS in a small network is not exactly a fulfilling experience. So, hey, you know what? Perhaps CephFS can replace these. Perhaps CephFS is better. Perhaps CephFS is ready for this. I mean, I like ext4fs, but only because btrfs hates me. I'm loyal. But I'm here now. Seriously, I think that, with just a little bit of polishing and automation, Ceph could be deployed in the small-office/home-office setting. Don't laugh. This could happen. Sadly, I am seeing a tiny bit of data corruption. I said don't laugh. I'm resource constrained, server and disk constrained, and time-unconstrained. So I thought I'd bootstrap up from nothing. Manually.. First, one server, with two OSD's from two partitions on two different disks. Works great! For the record, this should be no worse than RAID mirroring on one host, and so is 100% entirely acceptable in the home/small-office environment. I move (not copy, but move) a few 100 GB of data I can afford to lose. Of course there are "warning: degraded+undersize" warnings, but hardly a surprise. I add a second server, and a 3rd OSD. The initial system is running Ceph "reef", because it's running Debian testing which has reef in it. The second system has ceph "pacific", because it's Debian stable, and that's what Debian stable currently packages. (Yes, pacific is past EOL. But it's on Debian stable, so....) Next version of Debian stable won't show up till next summer. I thought reef and pacific might not talk to one-another, but they do, and happily so. Excellent! +1 ceph! The system starts rebalancing onto the third OSD. After running all night, it gets to 2/3rds or 3/4 clean, and maybe 1/4th degraded-undersize. Surprising at first, because I had size/min_size set to 3/2 and I had 3 OSD's so .. ??? Later I understood that CRUSH wants three hosts, but whatever. I experiment with changing size/min_size to 2/1 and the warnings go away! Going back to 3/2 and they reappear. I am somewhat unhappy: before switching to 2/1, I saw most PG's were on most OSD's. Good! Switching to 2/1 made many of them disappear (!) and switching back to 3/2 did not make them reappear. The OSD's seem mildly underpopulated, and seem to remain that way. Huh. Who am I to say, but I would have liked to have seen one copy of *every* PG on *every* OSD, because that's what size=3 should mean, right? Even if there are only two servers? Or should I select a different crush algo? (I have not played with this, yet). I would sleep better if all three disks had a copy, because that would be an actual improvement over bog-standard raid mirroring. Anyway, I add a 4th OSD on the second server (so that there are 4 total) and I add a third server with zero OSD's (so that ceph-mon can create a proper quorum.) And by now I have over a TB of data on there (that I can afford to lose). So that's the backstory. Here's the problem. Last night, I move (not copy, but move) a 33 GB directory onto CephFS. This morning I see: mv: cannot overwrite directory '/dest/.snap' with non-directory '/source/.snap' wtf? I cd to /dest/.snap and indeed it is a directory. And it's empty. The source .snap is a file, 736 bytes in size. Directory timestamp and source-file timestamp are identical. I attempt a hand repair. I cd to the parent dir, do rmdir .snap and rmdir says "no such directory". wtf? So then `ls -la` and indeed, there is no such directory. But I can still cd into it! Huh. I cannot rmdir it, I cannot cp or mv it, because "it doesn't exist". If I cd into it, I cannot create any files in it, I get "permission denied", even as root. So it's a directory, it won't show in the listing of the parent dir. I can cd to it, but I cannot put anything in it. This, to me, says "CephFS bug". Other culprits could have been: (a) bad disk (b) corruption during copy, i.e. bad fs stat during copy due to (b1) bad kernel (b2) bad SATA traffic (b3) corrupt kernel page table, buffer, block cache, etc. (b4) bad data on the source disk. But all this is in the past: I can look at the source file: it's there, it's readable, it has valid contents, (It's ascii text. It looks just fine.) Clearly some transient error perturbed things. However, I now have a CephFS with a directory that I cannot remove, and I cannot place anything into it, and I cannot create a file with the same name as the directory. So that's not good. Sometime after I hit send on this email, ceph will automatically run scrub and/or deep scrub, and maybe this problem will fix itself. If there's something I should do to preserve this broken-ness for future posterity debugging, tell me now before it's too late. While I'm on this topic: The failed `mv` means that the source was still there, and so I was able to run `diff -rq --no-dereference` of the original source and the copy. I discovered that there were 5 files that were borked. They all had the same timestamps as the original, but the content was all zeros (as reported by od) I found that I could repair these by hand-copying the original over them, and the result is fine. Yes, this is a frankenstein system. Yes, it has 3 mons and three mgrs and three mds running on three machines, but OSD's only on two of them. I'll add more OSD's maybe tomorrow. But meanwhile, I have a system I can lose sleep over. The five files that were corrupted were corrupted *silently*, I would never have known if not for the weird-dir bug allowing me to go do the diff and find the corruption. This was on a copy of 33GB containing 412K files. I'd previously moved over a TB of files. Are any of these damaged? God knows. I was sloppy, I did not keep the originals, I did not do a diff. I trusted CephFS to do the right thing, and it seems it didn't. That trust is shattered. How often does ext4fs do something like this? I don't know. I've used ext2/3/4fs for 25 years on maybe a hundred different machines, and never lost data (that wasn't user error, i.e. rm-r* in the wrong directory) This is with 25 years of consumer-grade, low-quality pata/sata and sometimes scsi gear plugged into consumer-grade boxes cobbled from parts bought on newegg or maybe refurbished from Dell. (Yeah I also get to use huge high-end machines, but this is "for example") If there was dataloss during these decades, I'm not aware of it. Was there any? Laws of probability say "yes". Once upon a time, I managed to corrupt an ext2fs disk so that e2fsck could not fix it. Yes, *hit happens. But I've been using Ceph for barely a week, and its 2024 so it should be mature, so I'm not thrilled that I hit a bug so early and so easily. WTF. I did read the paper https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution" which is an awesome paper, by the way, kudos to the authors. It does have one mis-representation, though. It says, paraphrasing: "it typically takes ten years for a file system to mature, and we here at CephFS/bluestore did it in only two (glow)" That paper was published in 2019 and it's now 2024, and off-the-shelf CephFS is clearly and blatantly buggy. Ceph is still awesome, but you cannot blame this bug on crappy hardware or a frankenstein sysconfig. Either the corner cases work, or they don't, and the authors of other filesystems take a decade or two to get through all of those corner cases. It would seem that CephFS has not evaded this iron law. Sorry for ending on such a downer, but hey... I'm one of those perfectionists that wreck things for the good-enough people. So it goes. -- Linas -- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us.

3 2

How to modify the destination pool name in the rbd-mirror configuration?
by David Yang 27 Nov '24

27 Nov '24

Hello, everyone. Under normal circumstances, we synchronize from PoolA of ClusterA to PoolA of ClusterB (same pool name), which is also easy to configure. Now the requirements are as follows: ClusterA/Pool synchronizes to BackCluster/PoolA ClusterB/Pool synchronizes to BackCluster/PoolB After reading rbd-mirror and other documents, I did not find the configuration of different pool names.

2 4

Re: 4k IOPS: miserable performance in All-SSD cluster
by Peter Linder 26 Nov '24

26 Nov '24

In my experience, ceph will add around 1ms even if only on localhost. If this is in the client code or on the OSD's, I dont really know. I don't even know the precise reason, but the latency is there nevertheless. Perhaps you can find the reason here among the tradeoffs ceph and similar systems have to make to ensure consistency even if a partition can happen at any time: https://en.wikipedia.org/wiki/PACELC_theorem With size=3, a write will go first to the primary OSD for the PG, (0,1ms), then from there to two more PGs (in parallell), so 0,2ms more total RTT. Then back to the client, 0,1ms. That is, very roughly, 1,4ms even if storage latency is 0 which it never is even for ssds. If you set size=1, you can skip the step where the primary OSD replicates to the 2 replicas, but you still have cephs internal latency as well as the network latency to reach the primary OSD for whatever PG the object will belong to which could be on any server. So expect a small improvement but not too much. With that said, a single thread will not exceed 1000 iops ever in a typical setup. /Peter Den 2024-11-26 kl. 21:09, skrev Martin Gerhard Loschwitz: > that would mean 2-3ms latency between hosts hanging above each other > in the same rack connected to the same switches. > > Ping shows 0,2ms of latency though for all three affected clusters. > > So roughly 5000 iops. We can certainly add Ceph latency to that, but > that would mean Ceph eats 99% of the available performance, wouldn't it? > > Also, that wouldn't explain why we're seeing a bit of improvement with > size=1 for a specific pool but not a massive improvement, given that > at least half of the latency is taken out of the equation in that case. > > Best regards > Martin > > Peter Linder <peter.linder(a)fiberdirekt.se> schrieb am Di. 26. Nov. > 2024 um 20:52: > > With qd=1 (queue depth?) and a single thread, this isn't totally > unreasonable. > > Ceph will have an internal latency of around 1ms or so, add some > network > to that and an operation can take 2-3ms. With a single operation in > flight all the time, this means 333-500 operations per second. With > hdds, even fewer. > > What happens if you try again with many more threads? > > > Den 2024-11-25 kl. 15:22, skrev Martin Gerhard Loschwitz: > > Folks, > > > > I am getting somewhat desperate debugging multiple setups here > within the same environment. Three clusters, two SSD-only, one > HDD-only, and what they all have in common is abysmal 4k IOPS > performance when measuring with „rados bench“. Abysmal means: In > an All-SSD cluster I will get roughly 400 IOPS over more than 250 > devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on > the low side of things to me. > > > > In the second cluster, also All-SSD based, I get roughly 120 4k > IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter > both with substantially fewer devices, granted. But even with 20 > HDDs, 68 4k IOPS seems like a very bad value to me. > > > > I’ve tried to rule out everything I know of: BIOS > misconfigurations, HBA problems, networking trouble (I am seeing > comparably bad values with a size=1 pool) and so further and so > on. But to no avail. Has anybody dealt with something similar on > Dell hardware or in general? What could cause such extremely bad > benchmark results? > > > > I measure with rados bench and qd=1 at 4k block size. „ceph tell > osd bench“ with 4k blocks yields 30k+ IOPS for every single device > in the big cluster, and all that leads to is 400 IOPS in total > when writing to it? Even with no replication in place? That looks > a bit off, doesn't it? Any help will be greatly appreciated, thank > you very much in advance. Even a pointer to the right direction > would be held in high esteem right now. Thank you very much in > advance! > > > > Best regards > > Martin > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

4 5

v17.2.8 Quincy released
by Yuri Weinstein 26 Nov '24

26 Nov '24

We're happy to announce the 8th backport release in the Quincy series. https://ceph.io/en/news/blog/2024/v17-2-8-quincy-released/ v17.2.8 will have RPM/centos 9 packages instead of RPM/centos 8 built. v17.2.8 container images, now based on CentOS 9, may be incompatible on older kernels (e.g., Ubuntu 18.04) due to differences in thread creation methods. Users upgrading to v17.2.8 container images with older OS versions may encounter crashes during `pthread_create`. However, we recommend upgrading your OS to avoid this unsupported combination. Users should expect to see the el8 rpm subdirectory empty and the "dnf" commands are expected to fail with 17.2.8. They can choose to use 17.2.8 RPM packages for centos 8/el8 provided by CERN as a community member or continue to stay at 17.2.7 following instructions from https://docs.ceph.com/en/latest/install/get-packages/#rhel, the ceph.repo file should point to https://download.ceph.com/rpm-17.2.7/el8 instead of https://download.ceph.com/rpm-quincy/el8 These CERN packages come with no warranty and have not been tested. The software in them has been tested by Ceph according to [platforms](https://docs.ceph.com/en/latest/start/os-recommendations/#platfo…. The repository for el8 builds is hosted by CERN on [Linux @ CERN](https://linuxsoft.cern.ch/repos/ceph-ext-quincy8el-stable/). The public part of the GPG key used to sign the packages is available at [RPM-GPG-KEY-Ceph-Community](https://linuxsoft.cern.ch/repos/RPM-GPG-KEY-Ceph-Community). Notable Changes --------------- * RADOS: `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated due to being prone to false negative results. It's safer replacement is `pool_is_in_selfmanaged_snaps_mode`. * RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled and valid), diff-iterate is now guaranteed to execute locally if exclusive lock is available. This brings a dramatic performance improvement for QEMU live disk synchronization and backup use cases. * RBD: The option ``--image-id`` has been added to `rbd children` CLI command, so it can be run for images in the trash. * RBD: `RBD_IMAGE_OPTION_CLONE_FORMAT` option has been exposed in Python bindings via `clone_format` optional parameter to `clone`, `deep_copy` and `migration_prepare` methods. * RBD: `RBD_IMAGE_OPTION_FLATTEN` option has been exposed in Python bindings via `flatten` optional parameter to `deep_copy` and `migration_prepare` methods. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.8.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: f817ceb7f187defb1d021d6328fa833eb8e943b3

3 3

RGW Daemons Crash After Adding Secondary Zone with Archive module
by mahnoosh shahidi 26 Nov '24

26 Nov '24

Hi everyone, I encountered an issue while setting up a multisite configuration in Ceph. After adding the second secondary zone and enabling the archive module, the RGW daemons started crashing repeatedly after running the period update command. The crash log shows the following error: **** Caught signal (Aborted) ** * *in thread 7fc7d20c7640 thread_name:data-sync * *ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) * *1: /lib64/libc.so.6(+0x3e6f0) [0x7fc7f5c916f0] * *2: /lib64/libc.so.6(+0x8b94c) [0x7fc7f5cde94c] * *3: raise() * *4: abort() * *5: /usr/bin/radosgw(+0x30e668) [0x55983a811668] * *6: /usr/bin/radosgw(+0x9236c7) [0x55983ae266c7] * *7: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x125) [0x55983aa484b5] * *8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x489) [0x55983aa49529] * *9: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x55983aa4a31d] * *10: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x3e0) [0x55983ae36c20] * *11: /usr/bin/radosgw(+0x705e88) [0x55983ac08e88] * *12: (RGWRadosThread::Worker::entry()+0xb3) [0x55983ac0c863] * *13: /lib64/libc.so.6(+0x89c02) [0x7fc7f5cdcc02] * *14: /lib64/libc.so.6(+0x10ec40) [0x7fc7f5d61c40] * *NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. * Any idea about this? Thank you.

1 0

Multisite RGW-SYNC error: failed to remove omap key from error repo
by Szabo, Istvan (Agoda) 26 Nov '24

26 Nov '24

Hi, I see in our logs continuously these kind of messaages: RGW-SYNC:data:sync:shard[7]:entry: ERROR: failed to remove omap key from error repo (hkg.rgw.log:datalog.sync-status.shard.61c9d940-fde4-4bed-9389-edc8d7741817.7.retry retcode=-1 And we started to receive large omaps in the log pool which we trim and scrub the pg to get rid of it, however this is not the root cause. What I can see in this line is the remote site id: 61c9d940-fde4-... Which has 1 bucket to sync and it looks like syncing so don't understand what is this error. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

CephFS 16.2.10 problem
by Alexey.Tsivinsky＠baikalelectronics.ru 26 Nov '24

26 Nov '24

Good morning! It's a novelty for our situation. The list of cluster locks now contains entries for the addresses of the servers on which the mds is running. The lifetime of the records is a day. For now, we decided to wait until the records are deleted and see how the cluster behaves. Best regards! Alexey Tsivinsky e-mail:a.tsivinsky@baikalelectronics.com

1 0

Cephalocon Update - New Users Workshop and Power Users Session
by Dan van der Ster 26 Nov '24

26 Nov '24

Dear Fellow Cephalopods, Cephalocon 2024 hosted by CERN is just around the corner! In case you missed it, the full schedule is now live: https://events.linuxfoundation.org/cephalocon/ The Program Committee received a record-breaking 176 submissions, carefully selecting 60 speakers to present an exciting lineup. Attendees will have the chance to dive into the latest updates from Ceph Developers and engage with fellow users. We're thrilled to report that registration is approaching our target of 300 attendees, and interest is higher than anticipated! Our hosts are working hard to accommodate even more participants. If you haven't registered yet and plan to join us, don't delay---secure your spot today! Additionally, two pre-Cephalocon events are open for attendees: - New to Ceph? Join the Ceph New Users Workshop - https://indico.cern.ch/e/ceph-new-users-workshop It’s a great opportunity for beginners to get up to speed. - Experienced Ceph User? Participate in the Power Users afternoon session at the Developers Summit - https://indico.cern.ch/e/ceph-developer-summit These pre-events are free but require a valid Cephalocon registration. We look forward to seeing you there! Warm regards, Dan, on behalf of the Cephalocon Organizers -- Dan van der Ster CTO @ CLYSO

4 5

How to synchronize pools with the same name in multiple clusters to multiple pools in one cluster
by David Yang 26 Nov '24

26 Nov '24

We have ceph clusters in multiple regions to provide rbd services. We are currently preparing a remote backup plan, which is to synchronize pools with the same name in each region to different pools in one cluster. For example: Cluster A Pool synchronized to backup cluster poolA Cluster B Pool synchronized to backup cluster poolB I read the rbd-mirror documentation, but I don't understand how to start synchronization using bootstrap mode.

1 0

Ceph Steering Committee 2024-11-25
by Gregory Farnum 25 Nov '24

25 Nov '24

Another light meeting (we're appreciating it after our heavy governance discussions!): * Cancel next week due to Cephalocon travel * We were blocked on the quincy release due to some build issues with Ganesha (apparently we were pointed at the wrong kind of CentOS repo, and then the team was asking why we are still shipping an old version — consensus was not to change mid-stream without a reason and targeted testing), but those have been sorted out. * Report on the Ceph Ambassadors monthly meeting -Greg

1 0

Encrypt OSDs on running System. A good Idea?
by Giovanna Ratini 25 Nov '24

25 Nov '24

Hello all :-), We use Ceph both as storage in Proxmox and as storage in K8S. I would like to encrypt the OSDs. I have backups of the Proxmox machines, but honestly, I would prefer not to have to use them, as it would take two days to rebuild everything from scratch. I ran some tests on a small Proxmox machine, and telling the OSDs to encrypt themselves doesn’t seem too difficult. However, nothing is running on that storage. What issues should I expect if I take an OSD (15TB) out one at a time, encrypt it, and put it back into the cluster? I would have a long period where some OSDs are encrypted and others are not. How dangerous is this? Has anyone done it before ? Best wishes, Gio

2 2

UPGRADE_REDEPLOY_DAEMON: Upgrading daemon failed
by Stolte, Felix 25 Nov '24

25 Nov '24

Hi folks, we did upgrade one of our clusters from pacific to Quincy. Everything worked fine, but cephadm complains about one osd not being upgraded: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.15 on host osd-dmz-k5-1 failed. Upgrade daemon: osd.15: cephadm exited with an error code: 1, stderr: Redeploy daemon osd.15 ... Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2df852c3fc\x2d05a0\x2d11e8\x2dbae7\x2d77689751e5e7.slice/ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service Non-zero exit code 1 from systemctl start ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15 systemctl: stderr Job for ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service failed because the control process exited with error code. systemctl: stderr See "systemctl status ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service" and "journalctl -xeu ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service" for details. Traceback (most recent call last): File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9679, in <module> main() File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9667, in main r = ctx.func(ctx) File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2168, in _default_image return func(ctx) File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 5992, in command_deploy deploy_daemon(ctx, ctx.fsid, daemon_type, daemon_id, c, uid, gid, File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 3301, in deploy_daemon deploy_daemon_units(ctx, fsid, uid, gid, daemon_type, daemon_id, File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 3558, in deploy_daemon_units call_throws(ctx, ['systemctl', 'start', unit_name]) File "/var/lib/ceph/f852c3fc-05a0-11e8-bae7-77689751e5e7/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 1806, in call_throws raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') RuntimeError: Failed command: systemctl start ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15: Job for ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service failed because the control process exited with error code. See "systemctl status ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service" and "journalctl -xeu ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service" for details. The osd in question seems to be running fine: systemctl status ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service ● ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service - Ceph osd.15 for f852c3fc-05a0-11e8-bae7-77689751e5e7 Loaded: loaded (/etc/systemd/system/ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7@.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2024-11-16 10:02:27 CET; 1 week 2 days ago Main PID: 24583 (conmon) Tasks: 67 (limit: 76281) Memory: 6.0G CPU: 9h 57min 20.017s CGroup: /system.slice/system-ceph\x2df852c3fc\x2d05a0\x2d11e8\x2dbae7\x2d77689751e5e7.slice/ceph-f852c3fc-05a0-11e8-bae7-77689751e5e7(a)osd.15.service ├─libpod-payload-3e6ba1f01ad8ca4c20c08a9984bdd983f43b9a15a0ec1b452b4d17c9f5ef519e │ ├─24586 /dev/init -- /usr/bin/ceph-osd -n osd.15 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false │ └─24588 /usr/bin/ceph-osd -n osd.15 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false └─supervisor └─24583 /usr/bin/conmon --api-version 1 -c 3e6ba1f01ad8ca4c20c08a9984bdd983f43b9a15a0ec1b452b4d17c9f5ef519e -u 3e6ba1f01ad8ca4c20c08a9984bdd983f43b9a15a0ec1b452b4d17c9f5ef519e -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/3e6ba1f01ad8ca4c20c08a9984bdd983f43b9a15a0ec1b452b4d17c9f5ef519e/userdata -p /run/containers/storage/over> Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: (Original Log Time 2024/11/25-10:23:14.904662) [db/memtable_list.cc:628] [default] Level-0 commit table #794120: memtable #1 done Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: (Original Log Time 2024/11/25-10:23:14.904710) EVENT_LOG_v1 {"time_micros": 1732530194904694, "job": 1660, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 1, 8, 44, 0, 0, 0], "immutable_memtables": 0} Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: (Original Log Time 2024/11/25-10:23:14.904789) [db/db_impl/db_impl_compaction_flush.cc:233] [default] Level summary: files[2 1 8 44 0 0 0] max score 0.78 Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: [db/db_impl/db_impl_files.cc:415] [JOB 1660] Try to delete WAL files size 255924988, prev total WAL file size 256244157, number of live WAL files 2. Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: [file/delete_scheduler.cc:69] Deleted file db/794117.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 Nov 25 11:23:14 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: (Original Log Time 2024/11/25-10:23:14.905401) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do Nov 25 11:32:34 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- Nov 25 11:32:34 osd-dmz-k5-1 ceph-osd[24588]: rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 783001.8 total, 600.0 interval Cumulative writes: 24M writes, 97M keys, 24M commit groups, 1.0 writes per commit group, ingest: 119.22 GB, 0.16 MB/s Cumulative WAL: 24M writes, 11M syncs, 2.03 writes per sync, written: 119.22 GB, 0.16 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 17K writes, 61K keys, 17K commit groups, 1.0 writes per commit group, ingest: 95.37 MB, 0.16 MB/s Interval WAL: 17K writes, 8473 syncs, 2.01 writes per sync, written: 0.09 MB, 0.16 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 9.86 MB 0.5 0.0 0.0 0.0 1.9 1.9 0.0 1.0 0.0 29.6 66.03 64.32 505 0.131 0 0 L1 1/0 66.88 MB 0.7 4.6 1.9 2.7 3.3 0.7 0.0 1.8 71.1 52.1 65.56 60.72 126 0.520 100M 4156K L2 8/0 450.76 MB 0.8 7.1 0.7 6.4 6.8 0.4 0.0 10.3 53.7 51.6 135.52 118.93 16 8.470 190M 1298K L3 44/0 2.65 GB 0.1 0.7 0.3 0.4 0.4 -0.0 0.0 1.3 79.5 43.3 8.89 7.81 4 2.223 28M 17M Sum 55/0 3.17 GB 0.0 12.3 2.9 9.5 12.4 3.0 0.0 6.5 45.8 46.2 276.01 251.78 651 0.424 318M 22M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 44.0 0.12 0.12 1 0.124 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.3 2.9 9.5 10.5 1.0 0.0 0.0 60.2 51.4 209.98 187.46 146 1.438 318M 22M High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 29.5 66.00 64.32 504 0.131 0 0 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 102.0 0.02 0.00 1 0.025 0 0 Uptime(secs): 783001.9 total, 600.0 interval Flush(GB): cumulative 1.906, interval 0.005 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 12.45 GB write, 0.02 MB/s write, 12.35 GB read, 0.02 MB/s read, 276.0 seconds Interval compaction: 0.01 GB write, 0.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.1 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 9.86 MB 0.5 0.0 0.0 0.0 1.9 1.9 0.0 1.0 0.0 29.6 66.03 64.32 505 0.131 0 0 L1 1/0 66.88 MB 0.7 4.6 1.9 2.7 3.3 0.7 0.0 1.8 71.1 52.1 65.56 60.72 126 0.520 100M 4156K L2 8/0 450.76 MB 0.8 7.1 0.7 6.4 6.8 0.4 0.0 10.3 53.7 51.6 135.52 118.93 16 8.470 190M 1298K L3 44/0 2.65 GB 0.1 0.7 0.3 0.4 0.4 -0.0 0.0 1.3 79.5 43.3 8.89 7.81 4 2.223 28M 17M Sum 55/0 3.17 GB 0.0 12.3 2.9 9.5 12.4 3.0 0.0 6.5 45.8 46.2 276.01 251.78 651 0.424 318M 22M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.3 2.9 9.5 10.5 1.0 0.0 0.0 60.2 51.4 209.98 187.46 146 1.438 318M 22M High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 29.5 66.00 64.32 504 0.131 0 0 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 102.0 0.02 0.00 1 0.025 0 0 Uptime(secs): 783001.9 total, 0.0 interval Flush(GB): cumulative 1.906, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 12.45 GB write, 0.02 MB/s write, 12.35 GB read, 0.02 MB/s read, 276.0 seconds How do i fix this? We tried redeploying the osd but to no success. Best regards Felix ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

2 1

Ceph OSD perf metrics missing
by Kilian Ries 25 Nov '24

25 Nov '24

Hi, i'm running a Ceph v18.2.4 cluster. I'm trying to build some latency monitoring with the ceph daemon osd.4 perf dump cli command. On most of the OSDs i get all the metrics i need. On some OSDs i only get zero values: osd.op_latency.avgcount: 0 I already tried restarting the OSD process which didn't help. I also tried to reset the metrics via ceph daemon osd.4 perf reset all but that didn't help either. How can that be that some OSDs don't show values here? How can i fix that? Thanks Regards, Kilian

3 4

Separate gateway for bucket lifecycle
by Szabo, Istvan (Agoda) 25 Nov '24

25 Nov '24

Hi, I remember some discussion that someone is using separated gateways for bucket lifecycles but I couldn't find it. How is that can be possible? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

How to speed up OSD deployment process
by YuFan Chen 23 Nov '24

23 Nov '24

Hi, I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices). I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete. However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next. Is there a way to speed up the OSD deployment process? Thanks in advance for your help! Best regards, Yufan Chen

4 6

multisite sync issue with bucket sync
by Christopher Durham 22 Nov '24

22 Nov '24

I have a 2-site multisite configuration on cdnh 18.2.4 on EL9. After system updates, we discovered that a particular bucket had several thousand objects missing, which the other side had. Newly created objects were being replicated just fine. I decided to 'restart' syncing that bucket. Here is what I did On the side with misisng objects: > radosgw-admin bucket sync init --bucket <bucketname> --src-zone <zone> I restarted the radosgw set up to do the sync thread on the same zone as I ran the radosgw-admin command. Logs on the radosgw src-zone side show GETs with http code 200 for objects that do not exist on the side with missing objects, and GETs with http 304 for objects that already exist on the side with missing objects. So far, so good. As I said, the bucket is active. So on the src-zone side, data is continually being written to /prefixA/../../ There is also data being written to /prefixB/../../ prefixA/ comes lexographically before prefixB/ What happens is that all the 304s happen as it scans the bucket, then starts pulling with GETs and http 200s for the objects the side doing the sync doesnt have. This is on /prefixA. When it 'caches up' with alldata in /prefixA at the moment, the sync seems to START OVER with /prefixA, giving 304s for everything that existed in the bucket up to the moment it caught up, then doing GETs with 200s for the remainingnewer objects. This happens over and over again. It NEVER gets to /prefixB. So it seems to be periodically catching up to /prefixA, but never going on to /prefixB that is also being written to There are 1.2 million objects in this bucket, with about 35 TiB in the bucket. There is a lifecycle expiration happening of 60 days. Any thoughts would be appreciated. -Chris

2 7

please unsubscribe
by Debian 108 22 Nov '24

22 Nov '24

Hi this is the tenth unsubscribe mail I send and after few minutes I receive another email. please, could some admin delete my email from the mailing list ? thanks

2 1

Crush rule examples
by Andre Tann 22 Nov '24

22 Nov '24

Hi all, I'm trying to understand how crush rules need to be set up, and much to my surprise I cannot find examples and/or good explanations (or I'm too stupid to understand them ;) ) My setup looks like this: DC1 node01 node02 node03 node04 node05 DC2 node01 node01 node01 node01 -- Andre Tann

4 10

MDS blocklist/evict clients during network maintenance
by Eugen Block 22 Nov '24

22 Nov '24

Hi, can anyone share some experience with these two configs? ceph config get mds mds_session_blocklist_on_timeout true ceph config get mds mds_session_blocklist_on_evict true If there's some network maintenance going on and the client connection is interrupted, could it help to disable evicting and blocklisting MDS clients? And what risks should we be aware of if we tried that? We're not entirely sure yet if this could be a reasonable approach, but we're trying to figure out how to make network maintenance less painful for clients. I'm also looking at some other possible configs, but let's start with these two first. Any comments would be appreciated! Thanks! Eugen

2 2

2024-11-21 Perf meeting cancelled!
by Matt Vandermeulen 21 Nov '24

21 Nov '24

Hi folks, the perf meeting will be cancelled today, Mark is flying from a conference! Thanks, Matt

1 0

Lifetime for ceph
by Steve Brasier 21 Nov '24

21 Nov '24

The octopus repos disappeared a couple of days ago - no argument with that given its marked as out of support. However I see from https://docs.ceph.com/en/latest/releases/ that quincy is also marked as out of support, but currently the repos are still there. Is there any guesstimate of when the quincy repos might disappear please? many thanks Steve Brasier http://stackhpc.com/ Please note I work Tuesday to Friday.

1 0

CephFS subvolumes not inheriting ephemeral distributed pin
by Rajmohan Ramamoorthy 21 Nov '24

21 Nov '24

CephFS subvolumes not inheriting ephemeral distributed pin configured on the subvolumegroups. References: https://github.com/ceph/ceph/pull/35759 https://github.com/ceph/ceph/pull/43896 # Environmet setup Ceph version: 18.2.4 and 19.2.0 CephFS with multiple MDS (2 active + 1 standby MDS) RHEL 9.4 Linux myserver 5.14.0-427.37.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 13 12:41:50 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux # Ceph FS configuration ``` ceph config set mds mds_export_ephemeral_distributed true ceph fs set midline-a balance_automate false ``` # Subvolume group (csi) creation ``` ceph fs subvolumegroup create midline-a csi ``` # Subvolume group (csi) pinning (tried both methods) ``` ceph fs subvolumegroup pin midline-a csi distributed 1 setfattr -n ceph.dir.pin.distributed -v 1 setfattr -n ceph.dir.pin.distributed -v 1 /mnt/cephfs/volumes/csi/ ``` # Subvolume creation ``` ceph fs subvolume create midline-a subvol1 csi ceph fs subvolume create midline-a subvol2 csi ``` # Get attribute (subvolume group) ``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi ceph.dir.pin.distributed="1" ``` # Get attribute (subvolume) - Subvolumes not inheriting the distributed pin attribute ``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi/subvol1 getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi/subvol1 ceph.dir.pin.distributed="0" ``` ``` getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi/subvol2 getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/volumes/csi/subvol2 ceph.dir.pin.distributed="0" ``` # MDS logs (subtree information) ``` ceph tell mds.abc123 get subtrees | jq '.[] | [.dir.path, .auth_first, .export_pin, .export_pin_target]' [ "~mds1", 1, -1, -1 ] [ "", 0, -1, -1 ] [ "/volumes/csi", 1, -1, 1 ] ``` ``` ceph tell mds.xyz123 get subtrees | jq '.[] | [.dir.path, .auth_first, .export_pin, .export_pin_target]' [ "/volumes/csi", 1, -1, 1 ] [ "", 0, -1, -1 ] [ "~mds0", 0, -1, -1 ] [ "/volumes/csi", 0, -1, 0 ] ``` Regards, Rajmohan R

2 5

Squid: regression in rgw multisite replication from Quincy/Reef clusters
by Casey Bodley 20 Nov '24

20 Nov '24

Recent multisite testing has uncovered a regression on Squid that happens when secondary zones are upgraded to Squid before the metadata master zone. User metadata replicates incorrectly in this configuration, such that their access keys are "inactive". As a result, these users are denied access to API requests on that secondary zone. This is tracked in https://tracker.ceph.com/issues/68985, and we'll prioritize the backport for 19.2.1. In the meantime, we strongly recommend upgrading the metadata master zone before others.

1 0

Join us for today's User + Developer Monthly Meetup!
by Laura Flores 20 Nov '24

20 Nov '24

Hi all, Please join us for today's User + Developer Monthly Meetup at 10:00 AM ET. RSVP here! https://www.meetup.com/ceph-user-group/events/304636936 Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

CephFS maximum filename length
by Naumann, Thomas 20 Nov '24

20 Nov '24

Hi at all, we use proxmoxcluster (v8.2.8) with ceph (v18.2.4) an EC-pools (all ceph options on default). One pool is exported with CephFS as backend storage for nextcloud servers. At the moment there is data migration from old S3 storage to CephFS pool. There are many files with huge filename length. In Wikipedia (see: https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits) there is CephFS max filename length to 255 characters and no limit for pathname length. Is this true? What is maximum filename length for CephFS (reef)? Which CephFS options could be altered to increase filnename length? What is the best practice to handle files with too long filename? Has anyone experience about this setting? best regards -- Thomas Naumann

1 0

Centos 9 updates break Reef MGR
by Chris Palmer 19 Nov '24

19 Nov '24

I've just applied routine Centos 9 updates to one node of a Reef 18.2.4 system (package install). They include some python3 updates that break the MGR in at least two ways. When the MGR starts in standby, it immediately logs the following two crashes. Only the dashboard and restful services are enabled (prometheus is not enabled). We have mgr/dashboard/FEATURE_TOGGLE_DASHBOARD = false. { "archived": "2024-11-19 10:30:46.031489", "backtrace": [ " File \"/usr/share/ceph/mgr/dashboard/__init__.py\", line 60, in <module>\n from .module import Module, StandbyModule # noqa: F401", " File \"/usr/share/ceph/mgr/dashboard/module.py\", line 51, in <module>\n patch_cherrypy(cherrypy.__version__)", " File \"/usr/share/ceph/mgr/dashboard/cherrypy_backports.py\", line 197, in patch_cherrypy\n accept_exceptions_from_builtin_ssl(ver)", " File \"/usr/share/ceph/mgr/dashboard/cherrypy_backports.py\", line 113, in accept_exceptions_from_builtin_ssl\n patch_builtin_ssl_wrap(v, accept_ssl_errors)", " File \"/usr/share/ceph/mgr/dashboard/cherrypy_backports.py\", line 75, in patch_builtin_ssl_wrap\n from cherrypy.wsgiserver.ssl_builtin import BuiltinSSLAdapter as builtin_ssl", "ModuleNotFoundError: No module named 'cherrypy.wsgiserver'" ], "ceph_version": "18.2.4", "crash_id": "2024-11-19T09:19:40.427015Z_f96f4d70-2112-47b5-96f7-5e9bd463e8eb", "entity_name": "mgr.ceph1", "mgr_module": "dashboard", "mgr_module_caller": "PyModule::load_subclass_of", "mgr_python_exception": "ModuleNotFoundError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mgr", "stack_sig": "9c98ebf1b6831bfca2823f54c9e6be01306090c2a7749def2ed8a15167fc527a", "timestamp": "2024-11-19T09:19:40.427015Z", "utsname_hostname": "ceph1.xxxxxx", "utsname_machine": "x86_64", "utsname_release": "6.1.112-1.el9.elrepo.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Sep 30 13:59:36 EDT 2024" } { "archived": "2024-11-19 10:30:46.056007", "backtrace": [ " File \"/usr/share/ceph/mgr/prometheus/__init__.py\", line 2, in <module>\n from .module import Module, StandbyModule", " File \"/usr/share/ceph/mgr/prometheus/module.py\", line 38, in <module>\n v = Version(cherrypy.__version__)", " File \"/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py\", line 277, in __init__\n raise InvalidVersion(\"Invalid version: '{0}'\".format(version))", "pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: 'unknown'" ], "ceph_version": "18.2.4", "crash_id": "2024-11-19T09:19:42.188291Z_c112de10-cdd5-4ed3-86b5-04dffd660cb8", "entity_name": "mgr.ceph1", "mgr_module": "prometheus", "mgr_module_caller": "PyModule::load_subclass_of", "mgr_python_exception": "InvalidVersion", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mgr", "stack_sig": "7fb0c6c17573887e8772c311aac1f3547c8366e50d6f2f8d8bd38deb6e0e9405", "timestamp": "2024-11-19T09:19:42.188291Z", "utsname_hostname": "ceph1.xxxxxx", "utsname_machine": "x86_64", "utsname_release": "6.1.112-1.el9.elrepo.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Sep 30 13:59:36 EDT 2024" } The MGR can be made active, but attempts to use the dashboard generate more of these. The culprit turns out to be python3-jaraco-text from the epel repository, which upgraded from python3-jaraco-text-3.2.0-6.el9.noarch to python3-jaraco-text-4.0.0-2.el9.noarch. (It also installed python3-jaraco-context-6.0.1-3.el9.noarch as a new dependency). Reverting to python3-jaraco-text-3.2.0 has avoided the problem for now, but that's not a sustainable long-term fix.

2 3

Re: Question about speeding hdd based cluster
by quaglio＠bol.com.br 19 Nov '24

19 Nov '24

1 2

Something like RAID0 with Ceph
by Christoph Pleger 19 Nov '24

19 Nov '24

Hello, Is it possible to have something like RAID0 with Ceph? That is, when the cluster configuration file contains osd pool default size = 4 and I have four hosts with one osd drive (all the same size, let's call it size s) per host, is it somehow possible to add four other hosts with one osd drive (again with size s) per host, so that the resulting Ceph block device is of size 2 * s? Regards Christoph

3 2

What is the Best stable option for production env in Q4/24 Quincy or Reef?
by Özkan Göksu 18 Nov '24

18 Nov '24

Hello! I always track Ceph releases 1-2 versions behind and I used Quincy for all my deployments this year. I’m planning to deploy a new cluster on Ubuntu 22.04 and would like to know if Reef v18.2.4 is stable and ready for production environments in 2025. My use case involves RBD, CephFS, and RGW with multisite S3. Would upgrading to Reef provide significant benefits over Quincy and is it safe & stable enough? Looking forward to any insights and recommendations. Best regards,

1 0

Ceph Octopus packages missing at download.ceph.com
by bzieglmeier＠gmail.com 18 Nov '24

18 Nov '24

Today I noticed that all ceph octopus packages are missing from download.ceph.com. Is this intentional? Was this an accident? I'm unable to find any announcement or existing issue tracking this...

6 6

Stray monitor
by Jakub Daniel 18 Nov '24

18 Nov '24

Hello, I'm pretty new to ceph deployment. I have setup my first cephfs cluster using cephadm. Initially, I deployed ceph in 3 virtualbox instances that I called cephfs-cluster-node-{0, 1, 2} just to test things. Later, I added 5 more real hardware nodes. Later I decided I'd remove the virtualboxes, so I drained the osds and removed the hosts. Suddenly, ceph status detail started reporting HEALTH_WARN 1 stray host(s) with 1 daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by cephadm stray host cephfs-cluster-node-2 has 1 stray daemons: ['mon.X'] The cephfs-cluster-node-2 is no longer listed among hosts, it is (and has been for tens of hours) offline (powered down). The mon.X doesn't even belong to that node, it is one of the real hardware nodes. I am unaware of mon.X ever running on cephfs-cluster-node-2 (never noticed it among systemd units). Where does cephadm shell -- ceph status detail come to the conclusion there is something stray? How can I address this? Thank you for any insights Jakub

3 5

done, waiting for purge
by Torkil Svensgaard 18 Nov '24

18 Nov '24

Hi 18.2.4 We had some hard drives going AWOL due to a failing SAS expander so I initiated "ceph orch host drain host". After a couple days I'm now looking at this: " OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 528 gimpy done, waiting for purge 0 False False False 529 gimpy done, waiting for purge 0 False False False 530 gimpy done, waiting for purge 0 False False False 531 gimpy done, waiting for purge 0 False False False 532 gimpy done, waiting for purge 0 False False False 533 gimpy done, waiting for purge 0 False False False 534 gimpy done, waiting for purge 0 False False False 535 gimpy done, waiting for purge 0 False False False 536 gimpy done, waiting for purge 0 False False False 537 gimpy done, waiting for purge 0 False False False 538 gimpy done, waiting for purge 0 False False False 539 gimpy done, waiting for purge 0 False False False " It removed the drives that was still working just fine but the missing drives seem stuck like this. How do I get these to finish? Force rm? Mvh. Torkil

2 1

Re: The effect of changing an osd's class
by Roland Giesler 18 Nov '24

18 Nov '24

On 2024/11/16 18:38, Anthony D'Atri wrote: > Disabling mclock as described here > https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/ might > help I cannot see any option that allows me to disable mclock... > > Also, you have a small cluster with a bunch of small OSDs. Please > send `ceph health detail` # ceph health detail HEALTH_WARN 1 pool(s) have no replicas configured [WRN] POOL_NO_REDUNDANCY: 1 pool(s) have no replicas configured pool 'backups' has no replicas configured The backups pool is on spinners and 1 copy is sufficient since we're also replicating this offsite in real time. Otherwise too many copies take too much space. > > Please also send `ceph osd df` and `ceph osd crush dump` NodeC:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 hdd 1.86029 1.00000 1.9 TiB 1.1 TiB 1.1 TiB 16 MiB 2.7 GiB 746 GiB 60.97 1.23 68 up 3 hdd 1.86029 1.00000 1.9 TiB 949 GiB 902 GiB 12 MiB 2.6 GiB 961 GiB 49.68 1.00 57 up 4 hdd 1.86029 1.00000 1.9 TiB 848 GiB 800 GiB 10 MiB 2.3 GiB 1.0 TiB 44.37 0.89 51 up 5 hdd 1.86589 1.00000 1.9 TiB 1.0 TiB 982 GiB 15 MiB 2.4 GiB 881 GiB 53.87 1.08 66 up 0 ssd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 1 ssd 0.28319 1.00000 290 GiB 123 GiB 122 GiB 2.3 MiB 1.2 GiB 167 GiB 42.38 0.85 46 up 36 ssd 0 1.00000 290 GiB 1.3 GiB 23 MiB 2.8 MiB 1.2 GiB 289 GiB 0.44 0.01 13 up 37 ssd 0 1.00000 290 GiB 1.1 GiB 23 MiB 2.5 MiB 1.1 GiB 289 GiB 0.38 0.01 18 up 38 ssd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 41 up 39 ssd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 86 up 40 ssd 3.30690 1.00000 3.3 TiB 1.7 TiB 1.7 TiB 23 MiB 4.8 GiB 1.6 TiB 50.49 1.02 424 up 10 hdd 1.86029 1.00000 1.9 TiB 1.0 TiB 990 GiB 13 MiB 2.8 GiB 873 GiB 54.19 1.09 64 up 11 hdd 1.86029 1.00000 1.9 TiB 1.3 TiB 1.2 TiB 15 MiB 3.2 GiB 625 GiB 67.20 1.35 72 up 26 hdd 1.86029 1.00000 1.9 TiB 843 GiB 801 GiB 12 MiB 2.3 GiB 1.0 TiB 44.25 0.89 50 up 27 hdd 1.86029 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 16 MiB 2.7 GiB 793 GiB 58.38 1.18 70 up 6 ssd 0.28319 1.00000 290 GiB 115 GiB 115 GiB 2.3 MiB 743 MiB 175 GiB 39.82 0.80 28 up 7 ssd 0.28319 1.00000 290 GiB 94 GiB 93 GiB 2.0 MiB 921 MiB 196 GiB 32.51 0.65 23 up 8 ssd 0.28319 1.00000 290 GiB 103 GiB 102 GiB 2.1 MiB 963 MiB 187 GiB 35.46 0.71 27 up 9 ssd 0.28319 1.00000 290 GiB 131 GiB 130 GiB 2.5 MiB 1.1 GiB 159 GiB 45.28 0.91 33 up 24 ssd 0.28319 1.00000 290 GiB 102 GiB 101 GiB 2.1 MiB 719 MiB 188 GiB 35.10 0.71 26 up 25 ssd 0.28319 1.00000 290 GiB 152 GiB 151 GiB 2.8 MiB 1020 MiB 138 GiB 52.52 1.06 39 up 41 ssd 3.49309 1.00000 3.5 TiB 1.6 TiB 1.6 TiB 22 MiB 5.4 GiB 1.9 TiB 44.58 0.90 411 up 14 hdd 1.59999 1.00000 1.9 TiB 1.0 TiB 1.0 TiB 13 MiB 2.4 GiB 837 GiB 56.04 1.13 63 up 15 hdd 1.86029 1.00000 1.9 TiB 1017 GiB 975 GiB 13 MiB 2.1 GiB 888 GiB 53.36 1.07 60 up 16 hdd 1.86029 1.00000 1.9 TiB 990 GiB 948 GiB 13 MiB 2.2 GiB 915 GiB 51.97 1.05 59 up 17 hdd 1.86029 1.00000 1.9 TiB 1.0 TiB 992 GiB 14 MiB 2.5 GiB 871 GiB 54.28 1.09 64 up 12 ssd 0.28319 1.00000 290 GiB 122 GiB 121 GiB 2.3 MiB 1.1 GiB 168 GiB 41.96 0.84 31 up 13 ssd 0.28319 1.00000 290 GiB 145 GiB 144 GiB 2.6 MiB 1.2 GiB 145 GiB 49.95 1.01 37 up 28 ssd 0.28319 1.00000 290 GiB 142 GiB 141 GiB 2.6 MiB 1.1 GiB 148 GiB 48.89 0.98 35 up 29 ssd 0.28319 1.00000 290 GiB 109 GiB 108 GiB 2.2 MiB 1.3 GiB 181 GiB 37.58 0.76 29 up 30 ssd 0.28319 1.00000 290 GiB 146 GiB 145 GiB 2.7 MiB 1.3 GiB 144 GiB 50.37 1.01 37 up 31 ssd 0.28319 1.00000 290 GiB 131 GiB 130 GiB 2.5 MiB 1.3 GiB 159 GiB 45.24 0.91 34 up 43 ssd 3.30690 1.00000 3.3 TiB 1.4 TiB 1.4 TiB 19 MiB 4.0 GiB 1.9 TiB 42.50 0.86 353 up 20 hdd 1.86029 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 12 MiB 2.8 GiB 659 GiB 65.38 1.32 67 up 21 hdd 1.86029 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 15 MiB 2.8 GiB 815 GiB 57.20 1.15 68 up 22 hdd 1.86029 1.00000 1.9 TiB 878 GiB 836 GiB 13 MiB 2.4 GiB 1.0 TiB 46.10 0.93 54 up 23 hdd 1.86029 1.00000 1.9 TiB 1018 GiB 977 GiB 14 MiB 2.8 GiB 886 GiB 53.46 1.08 59 up 18 ssd 0.28319 1.00000 290 GiB 115 GiB 114 GiB 2.3 MiB 1.3 GiB 175 GiB 39.74 0.80 29 up 19 ssd 0.28319 1.00000 290 GiB 115 GiB 114 GiB 2.3 MiB 961 MiB 175 GiB 39.59 0.80 28 up 32 ssd 0.28319 1.00000 290 GiB 103 GiB 102 GiB 2.1 MiB 1.0 GiB 187 GiB 35.65 0.72 26 up 33 ssd 0.28319 1.00000 290 GiB 98 GiB 97 GiB 2.1 MiB 1.4 GiB 192 GiB 33.92 0.68 26 up 34 ssd 0.28319 1.00000 290 GiB 141 GiB 140 GiB 2.6 MiB 987 MiB 149 GiB 48.49 0.98 35 up 35 ssd 0.28319 1.00000 290 GiB 116 GiB 114 GiB 2.3 MiB 1.3 GiB 174 GiB 39.92 0.80 31 up 42 ssd 3.49309 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 20 MiB 4.5 GiB 2.1 TiB 40.45 0.81 372 up TOTAL 49 TiB 25 TiB 24 TiB 351 MiB 83 GiB 25 TiB 49.67 MIN/MAX VAR: 0.01/1.35 STDDEV: 13.87 NodeC:~# ceph osd crush dump { "devices": [ { "id": 0, "name": "osd.0", "class": "ssd" }, { "id": 1, "name": "osd.1", "class": "ssd" }, { "id": 2, "name": "osd.2", "class": "hdd" }, { "id": 3, "name": "osd.3", "class": "hdd" }, { "id": 4, "name": "osd.4", "class": "hdd" }, { "id": 5, "name": "osd.5", "class": "hdd" }, { "id": 6, "name": "osd.6", "class": "ssd" }, { "id": 7, "name": "osd.7", "class": "ssd" }, { "id": 8, "name": "osd.8", "class": "ssd" }, { "id": 9, "name": "osd.9", "class": "ssd" }, { "id": 10, "name": "osd.10", "class": "hdd" }, { "id": 11, "name": "osd.11", "class": "hdd" }, { "id": 12, "name": "osd.12", "class": "ssd" }, { "id": 13, "name": "osd.13", "class": "ssd" }, { "id": 14, "name": "osd.14", "class": "hdd" }, { "id": 15, "name": "osd.15", "class": "hdd" }, { "id": 16, "name": "osd.16", "class": "hdd" }, { "id": 17, "name": "osd.17", "class": "hdd" }, { "id": 18, "name": "osd.18", "class": "ssd" }, { "id": 19, "name": "osd.19", "class": "ssd" }, { "id": 20, "name": "osd.20", "class": "hdd" }, { "id": 21, "name": "osd.21", "class": "hdd" }, { "id": 22, "name": "osd.22", "class": "hdd" }, { "id": 23, "name": "osd.23", "class": "hdd" }, { "id": 24, "name": "osd.24", "class": "ssd" }, { "id": 25, "name": "osd.25", "class": "ssd" }, { "id": 26, "name": "osd.26", "class": "hdd" }, { "id": 27, "name": "osd.27", "class": "hdd" }, { "id": 28, "name": "osd.28", "class": "ssd" }, { "id": 29, "name": "osd.29", "class": "ssd" }, { "id": 30, "name": "osd.30", "class": "ssd" }, { "id": 31, "name": "osd.31", "class": "ssd" }, { "id": 32, "name": "osd.32", "class": "ssd" }, { "id": 33, "name": "osd.33", "class": "ssd" }, { "id": 34, "name": "osd.34", "class": "ssd" }, { "id": 35, "name": "osd.35", "class": "ssd" }, { "id": 36, "name": "osd.36", "class": "ssd" }, { "id": 37, "name": "osd.37", "class": "ssd" }, { "id": 38, "name": "osd.38", "class": "ssd" }, { "id": 39, "name": "osd.39", "class": "ssd" }, { "id": 40, "name": "osd.40", "class": "ssd" }, { "id": 41, "name": "osd.41", "class": "ssd" }, { "id": 42, "name": "osd.42", "class": "ssd" }, { "id": 43, "name": "osd.43", "class": "ssd" } ], "types": [ { "type_id": 0, "name": "osd" }, { "type_id": 1, "name": "host" }, { "type_id": 2, "name": "chassis" }, { "type_id": 3, "name": "rack" }, { "type_id": 4, "name": "row" }, { "type_id": 5, "name": "pdu" }, { "type_id": 6, "name": "pod" }, { "type_id": 7, "name": "room" }, { "type_id": 8, "name": "datacenter" }, { "type_id": 9, "name": "zone" }, { "type_id": 10, "name": "region" }, { "type_id": 11, "name": "root" } ], "buckets": [ { "id": -1, "name": "default", "type_id": 11, "type_name": "root", "weight": 3177873, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -3, "weight": 723311, "pos": 0 }, { "id": -7, "weight": 827941, "pos": 1 }, { "id": -10, "weight": 798680, "pos": 2 }, { "id": -13, "weight": 827941, "pos": 3 } ] }, { "id": -2, "name": "default~ssd", "type_id": 11, "type_name": "root", "weight": 1243909, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -4, "weight": 235280, "pos": 0 }, { "id": -8, "weight": 340277, "pos": 1 }, { "id": -11, "weight": 328075, "pos": 2 }, { "id": -14, "weight": 340277, "pos": 3 } ] }, { "id": -3, "name": "FT1-NodeA", "type_id": 1, "type_name": "host", "weight": 723311, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 1, "weight": 18559, "pos": 0 }, { "id": 36, "weight": 0, "pos": 1 }, { "id": 37, "weight": 0, "pos": 2 }, { "id": 0, "weight": 0, "pos": 3 }, { "id": 38, "weight": 0, "pos": 4 }, { "id": 39, "weight": 0, "pos": 5 }, { "id": 2, "weight": 121916, "pos": 6 }, { "id": 3, "weight": 121916, "pos": 7 }, { "id": 4, "weight": 121916, "pos": 8 }, { "id": 40, "weight": 216721, "pos": 9 }, { "id": 5, "weight": 122283, "pos": 10 } ] }, { "id": -4, "name": "FT1-NodeA~ssd", "type_id": 1, "type_name": "host", "weight": 235280, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 1, "weight": 18559, "pos": 0 }, { "id": 36, "weight": 0, "pos": 1 }, { "id": 37, "weight": 0, "pos": 2 }, { "id": 0, "weight": 0, "pos": 3 }, { "id": 38, "weight": 0, "pos": 4 }, { "id": 39, "weight": 0, "pos": 5 }, { "id": 40, "weight": 216721, "pos": 6 } ] }, { "id": -5, "name": "FT1-NodeA~hdd", "type_id": 1, "type_name": "host", "weight": 488031, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 121916, "pos": 0 }, { "id": 3, "weight": 121916, "pos": 1 }, { "id": 4, "weight": 121916, "pos": 2 }, { "id": 5, "weight": 122283, "pos": 3 } ] }, { "id": -6, "name": "default~hdd", "type_id": 11, "type_name": "root", "weight": 1933964, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -5, "weight": 488031, "pos": 0 }, { "id": -9, "weight": 487664, "pos": 1 }, { "id": -12, "weight": 470605, "pos": 2 }, { "id": -15, "weight": 487664, "pos": 3 } ] }, { "id": -7, "name": "FT1-NodeB", "type_id": 1, "type_name": "host", "weight": 827941, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 6, "weight": 18559, "pos": 0 }, { "id": 24, "weight": 18559, "pos": 1 }, { "id": 25, "weight": 18559, "pos": 2 }, { "id": 11, "weight": 121916, "pos": 3 }, { "id": 10, "weight": 121916, "pos": 4 }, { "id": 7, "weight": 18559, "pos": 5 }, { "id": 8, "weight": 18559, "pos": 6 }, { "id": 9, "weight": 18559, "pos": 7 }, { "id": 26, "weight": 121916, "pos": 8 }, { "id": 27, "weight": 121916, "pos": 9 }, { "id": 41, "weight": 228923, "pos": 10 } ] }, { "id": -8, "name": "FT1-NodeB~ssd", "type_id": 1, "type_name": "host", "weight": 340277, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 6, "weight": 18559, "pos": 0 }, { "id": 24, "weight": 18559, "pos": 1 }, { "id": 25, "weight": 18559, "pos": 2 }, { "id": 7, "weight": 18559, "pos": 3 }, { "id": 8, "weight": 18559, "pos": 4 }, { "id": 9, "weight": 18559, "pos": 5 }, { "id": 41, "weight": 228923, "pos": 6 } ] }, { "id": -9, "name": "FT1-NodeB~hdd", "type_id": 1, "type_name": "host", "weight": 487664, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 11, "weight": 121916, "pos": 0 }, { "id": 10, "weight": 121916, "pos": 1 }, { "id": 26, "weight": 121916, "pos": 2 }, { "id": 27, "weight": 121916, "pos": 3 } ] }, { "id": -10, "name": "FT1-NodeC", "type_id": 1, "type_name": "host", "weight": 798680, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 13, "weight": 18559, "pos": 0 }, { "id": 28, "weight": 18559, "pos": 1 }, { "id": 29, "weight": 18559, "pos": 2 }, { "id": 14, "weight": 104857, "pos": 3 }, { "id": 12, "weight": 18559, "pos": 4 }, { "id": 30, "weight": 18559, "pos": 5 }, { "id": 31, "weight": 18559, "pos": 6 }, { "id": 15, "weight": 121916, "pos": 7 }, { "id": 16, "weight": 121916, "pos": 8 }, { "id": 17, "weight": 121916, "pos": 9 }, { "id": 43, "weight": 216721, "pos": 10 } ] }, { "id": -11, "name": "FT1-NodeC~ssd", "type_id": 1, "type_name": "host", "weight": 328075, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 13, "weight": 18559, "pos": 0 }, { "id": 28, "weight": 18559, "pos": 1 }, { "id": 29, "weight": 18559, "pos": 2 }, { "id": 12, "weight": 18559, "pos": 3 }, { "id": 30, "weight": 18559, "pos": 4 }, { "id": 31, "weight": 18559, "pos": 5 }, { "id": 43, "weight": 216721, "pos": 6 } ] }, { "id": -12, "name": "FT1-NodeC~hdd", "type_id": 1, "type_name": "host", "weight": 470605, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 14, "weight": 104857, "pos": 0 }, { "id": 15, "weight": 121916, "pos": 1 }, { "id": 16, "weight": 121916, "pos": 2 }, { "id": 17, "weight": 121916, "pos": 3 } ] }, { "id": -13, "name": "FT1-NodeD", "type_id": 1, "type_name": "host", "weight": 827941, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 18, "weight": 18559, "pos": 0 }, { "id": 32, "weight": 18559, "pos": 1 }, { "id": 33, "weight": 18559, "pos": 2 }, { "id": 22, "weight": 121916, "pos": 3 }, { "id": 19, "weight": 18559, "pos": 4 }, { "id": 34, "weight": 18559, "pos": 5 }, { "id": 35, "weight": 18559, "pos": 6 }, { "id": 23, "weight": 121916, "pos": 7 }, { "id": 21, "weight": 121916, "pos": 8 }, { "id": 20, "weight": 121916, "pos": 9 }, { "id": 42, "weight": 228923, "pos": 10 } ] }, { "id": -14, "name": "FT1-NodeD~ssd", "type_id": 1, "type_name": "host", "weight": 340277, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 18, "weight": 18559, "pos": 0 }, { "id": 32, "weight": 18559, "pos": 1 }, { "id": 33, "weight": 18559, "pos": 2 }, { "id": 19, "weight": 18559, "pos": 3 }, { "id": 34, "weight": 18559, "pos": 4 }, { "id": 35, "weight": 18559, "pos": 5 }, { "id": 42, "weight": 228923, "pos": 6 } ] }, { "id": -15, "name": "FT1-NodeD~hdd", "type_id": 1, "type_name": "host", "weight": 487664, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 22, "weight": 121916, "pos": 0 }, { "id": 23, "weight": 121916, "pos": 1 }, { "id": 21, "weight": 121916, "pos": 2 }, { "id": 20, "weight": 121916, "pos": 3 } ] }, { "id": -16, "name": "FT1-NodeA~old-ssd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, { "id": -17, "name": "FT1-NodeB~old-ssd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, { "id": -18, "name": "FT1-NodeC~old-ssd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, { "id": -19, "name": "FT1-NodeD~old-ssd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, { "id": -20, "name": "default~old-ssd", "type_id": 11, "type_name": "root", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -16, "weight": 0, "pos": 0 }, { "id": -17, "weight": 0, "pos": 1 }, { "id": -18, "weight": 0, "pos": 2 }, { "id": -19, "weight": 0, "pos": 3 } ] } ], "rules": [ { "rule_id": 0, "rule_name": "ssd_rule", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "hdd_rule", "type": 1, "steps": [ { "op": "take", "item": -6, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "old_nvme", "type": 1, "steps": [ { "op": "take", "item": -20, "item_name": "default~old-ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ], "tunables": { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 }, "choose_args": {} } > > It may help to manually edit the CRUSH map as detailed here: > > https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/ > > > At the end of the decompiled CRUSH map is a “tunables” section. Look > for the “c|hoose_total_tries|” > line, which should have the default value of 50. Change that to 100. > > > >> On Nov 16, 2024, at 11:13 AM, Roland Giesler <roland(a)giesler.za.net> >> wrote: >> >> On 2024/11/15 16:37, Anthony D'Atri wrote: >>> Only 2 OSDs in the acting set. This isn’t a size=2 pool is it? Did >>> you restart osd.39? >> >> Restarted both repeatedly. Has no effect. >> >> >>> >>>> On Nov 15, 2024, at 9:32 AM, Roland Giesler <roland(a)giesler.za.net> >>>> wrote: >>>> >>>> "acting": [ >>>> 39, >>>> 1 >>> >

3 5

Re: Pacific: mgr loses osd removal queue
by Eugen Block 18 Nov '24

18 Nov '24

I have a feeling that this could be related to the drives, but I have no real proof. I drained the SSD OSDs yesterday, hours later I wanted to remove the OSDs (no PGs were on them anymore) via for loop with the orchestrator (ceph orch osd rm ID --force --zap). The first one got removed quite quickly, but the others disappeared from the queue. I tried again and looked at the iostat output on the node where the queued OSD was running on. The drive had no IO at all but was utilized 100% for several minutes until it was eventually removed. I find that very weird, especially since we’re currently helping a customer rebuild OSDs on Pacific as well, and I haven’t seen such a behavior yet. And we already redeployed 132 OSDs so far. Zitat von Eugen Block <eblock(a)nde.ag>: > Hi, > > I'm not sure if this has been asked before, or if there's an > existing tracker issue already. It's difficult to reproduce it on my > lab clusters. > I'm testing some new SSD OSDs on a Pacific cluster (16.2.15) and > noticed that if we instruct the orchestrator to remove two or three > OSDs (issuing the command 'ceph orch osd rm {ID}' a couple of > times), it eventually only removes the first in the queue. I've been > watching 'ceph orch osd rm status' to see the progress, and then the > rest of the queued OSDs suddenly vanish from the status and never > get removed. Then I have to issue the command again. If I remove one > OSD by one, so no others in the queue, they are all successfully > removed. Why is this happening? Is this a bug in Pacific? Has > someone seen this in newer releases? > > Thanks! > Eugen

2 2

RGW names disappeared in quincy
by Szabo, Istvan (Agoda) 18 Nov '24

18 Nov '24

Hi, After updated from octopus to quincy the rados-gateways are showing like: rgw: 35 daemons active (20 hosts, 1 zones) which wouldn't be an issue, however the metrics that are exposed are coming with instance_id only (3642302922) which is pretty difficult (impossible) to match where it is running. This id can't be seen neither in the logs, nor as a socket. (Our deployment is not containerized) Is there a way to somehow make it easier to identify or somehow match with the servers where they are running? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Question about speeding hdd based cluster
by Kyriazis, George 16 Nov '24

16 Nov '24

Hello ceph-users, I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools. What would increase the speed of the hdd-based cephfs more, and in what usage scenarios: 1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links. It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck? Thank you! George

6 15

The effect of changing an osd's class
by Roland Giesler 15 Nov '24

15 Nov '24

I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it. When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd. What happened when I did this?

6 15

Ceph Reef 16 pgs not deep scrub and scrub
by Saint Kid 15 Nov '24

15 Nov '24

Hello, I have exactly 16 pgs with these conditions. Is there anyway i could do? I have tried to initiate the scrubing both deep and normal but they remain the same. HEALTH_WARN: 16 pgs not deep-scrubbed in time pg 8.14 not deep-scrubbed since 2024-09-27T19:42:47.463766+0700 pg 7.1b not deep-scrubbed since 2024-09-30T21:54:16.584697+0700 pg 8.1b not deep-scrubbed since 2024-09-29T23:22:37.812579+0700 pg 8.1d not deep-scrubbed since 2024-10-01T03:51:01.714780+0700 pg 6.13 not deep-scrubbed since 2024-09-26T07:16:31.387575+0700 pg 6.12 not deep-scrubbed since 2024-09-27T04:55:34.699188+0700 pg 8.3 not deep-scrubbed since 2024-09-29T20:57:40.973759+0700 pg 7.c not deep-scrubbed since 2024-10-01T22:57:28.955052+0700 pg 8.b not deep-scrubbed since 2024-10-01T09:15:55.223268+0700 pg 7.3 not deep-scrubbed since 2024-10-02T05:12:50.743544+0700 pg 7.18 not deep-scrubbed since 2024-09-30T17:57:34.550318+0700 pg 8.a not deep-scrubbed since 2024-09-30T10:37:06.429820+0700 pg 8.5 not deep-scrubbed since 2024-09-25T19:23:57.422280+0700 pg 8.9 not deep-scrubbed since 2024-09-30T17:57:33.493494+0700 pg 8.4 not deep-scrubbed since 2024-09-30T21:54:13.122287+0700 pg 8.10 not deep-scrubbed since 2024-10-02T20:07:42.793192+0700 HEALTH_WARN: 16 pgs not scrubbed in time pg 8.14 not scrubbed since 2024-10-02T08:04:26.363903+0700 pg 7.1b not scrubbed since 2024-10-02T03:14:53.854041+0700 pg 8.1b not scrubbed since 2024-10-02T15:06:12.351794+0700 pg 8.1d not scrubbed since 2024-10-02T11:57:57.600594+0700 pg 6.13 not scrubbed since 2024-10-02T01:48:49.382133+0700 pg 6.12 not scrubbed since 2024-10-02T08:19:53.026422+0700 pg 8.3 not scrubbed since 2024-10-02T04:29:55.426251+0700 pg 7.c not scrubbed since 2024-10-01T22:57:28.955052+0700 pg 8.b not scrubbed since 2024-10-02T09:36:03.568661+0700 pg 7.3 not scrubbed since 2024-10-02T05:12:50.743544+0700 pg 7.18 not scrubbed since 2024-10-01T22:17:53.504919+0700 pg 8.a not scrubbed since 2024-10-01T19:42:53.953996+0700 pg 8.5 not scrubbed since 2024-10-02T10:04:33.412894+0700 pg 8.9 not scrubbed since 2024-10-01T20:22:40.181880+0700 pg 8.4 not scrubbed since 2024-10-01T21:56:08.596293+0700 pg 8.10 not scrubbed since 2024-10-02T20:07:42.793192+0700

3 2

cephadm node failure (re-use OSDs instead of reprovisioning)
by Stefan Kooman 13 Nov '24

13 Nov '24

Hi list, In case of an OS disk failure on a cephadm managed storage node, is there a way to redeploy ceph on the (reinstalled) node leaving the data (OSDs) intact? So instead of removing the storage node, have the cluster recover, redeploy the storage node, and let the cluster recover, I would like to skip both recovery steps (when all that is broken is an OS disk) and only have a little bit of recovery while the OSDs where down. With a package based install this is achieved pretty easily: reinstall OS / ceph, make sure ceph keyrings are in right place / right owner (ceph) and run: ceph-volume lvm activate --all ... and you are back in business. Thanks, Stefan

2 2

Cephadm Drive upgrade process
by brentk＠cfl.rr.com 13 Nov '24

13 Nov '24

We are contemplating an upgrade of 4TB HDD drives to 20TB HDD drives (cluster info below, size 3 ), but as part of that discussion, we were trying to see if there was a more efficient way to do so. Our current process is as follows for failed drives: 1. Pulled failed drive ( after troubleshooting of course ) 2. Cephadm gui - find OSD, purge osd 3. Wait for rebalance 4. Insert new drive ( let cluster rebalance after it automatically adds the drive as an OSD ) ( yes, we have auto-add on in the clusters ) I imagine with an existing good drive, we would use delete instead of purge, but the process would be the similar, except the drive swap would happen after the data was moved. Would the replace flag ( or keep OSD option in gui ) allow us to avoid the initial rebalance by backfilling the new drive with the old drives content? I would be nice if we could just copy the content to the new drive and go from there. We would like to avoid lots of read/write cluster recovery activity if possible sine we could be replacing 40+ drives in each cluster. Open to trying a few methods of course, but we thought the group here might have some insight into the most optimal way of doing these swaps. For a history, last time we did this, we just drained an entire host, upgraded the drives and then added it back. This caused temporary imbalance that we would like to avoid this time around. Regards, -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2mds US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds

4 3

Strange container restarts?
by Jan Marek 13 Nov '24

13 Nov '24

Hello, we have ceph cluster which consists of 12 host, on every host we have 12 NVMe "disks". On most of these host (9 of 12) we have in logs errors, see attached file. We tried to check this problem, and we have these points: 1) On every host there is only one OSD. Thus it's not problem in version 18.2.2 generally, because there will be on another OSD, not only one of host? 2) Sometimes one of this OSD crashed :-( It seems, that crashed OSD are from set of OSDs, which have this problem. 3) ceph cluster goes OK and it "doesn't know" about any problem with these OSD. It seem's, that this new instance of ceph-osd daemon tried to start either podman or conmon itself. We've tried to control PID files for conman, but they're seems to be OK? 4) We tried to check 'ceph orch' command, but it does not try to start these containers, because it know, that they exists and run ('ceph orch ps' list these containers as running). 5) I've tried to pause ochestrator, but I've still found in syslog these entries... :-( Please, is there any possibility to find out, where is problem and stop this? We have all of the ceph host prepared by ansible, thus there is the same environment. On every machine we have podman version 4.3.1+ds1-8+deb12u1 and conmon version 2.1.6+ds1-1. OS is Debian bookworm. Attached logs was prepared by: grep exec_died /var/log/syslog Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

3 3

9 out of 11 missing shards of shadow object in ERC 8:3 pool.
by Robert Kihlberg 12 Nov '24

12 Nov '24

After an upgrade from Nautilus to Pacific the scrub has found an inconsistent object and reports that 9 out of 11 shards are missing. (However, we're not sure this has to do with the upgrade). We have been able to trace it to a S3 bucket, but not to a specific S3 object. # radosgw-admin object stat --bucket=$BUCKET --object=$OBJECT ERROR: failed to stat object, returned error: (2) No such file or directory By design, we have a complete mirror of the bucket in another Ceph cluster and the amount of objects in the buckets match between the clusters. We are therefore somewhat confident that we are not missing any objects. Could this be a failed garbage collection where perhaps the primary OSD failed during gc? The garbage collector does not show anything that seems relevant though... radosgw-admin gc list --include-all | grep "eaa6801e-3967-4541-9b8ca98aa5c2.791015596" Any suggestions on how we can trace and/or fix this inconsistent object? # rados list-inconsistent-obj 11.3ff | jq { "epoch": 177981, "inconsistents": [ { "object": { "name": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "nspace": "", "locator": "", "snap": "head", "version": 109853 }, "errors": [], "union_shard_errors": [ "missing" ], "selected_object_info": { "oid": { "oid": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "key": "", "snapid": -2, "hash": 4294967295, "max": 0, "pool": 11, "namespace": "" }, "version": "17636'109853", "prior_version": "0'0", "last_reqid": "client.791015590.0:449317175", "user_version": 109853, "size": 8388608, "mtime": "2022-01-24T03:33:42.457722+0000", "local_mtime": "2022-01-24T03:33:42.471042+0000", "lost": 0, "flags": [ "dirty", "data_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xe588978d", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 14, "primary": true, "shard": 0, "errors": [], "size": 1048576 }, { "osd": 67, "primary": false, "shard": 1, "errors": [ "missing" ] }, { "osd": 77, "primary": false, "shard": 4, "errors": [], "size": 1048576 }, { "osd": 225, "primary": false, "shard": 9, "errors": [ "missing" ] }, { "osd": 253, "primary": false, "shard": 8, "errors": [ "missing" ] }, { "osd": 327, "primary": false, "shard": 6, "errors": [ "missing" ] }, { "osd": 568, "primary": false, "shard": 2, "errors": [ "missing" ] }, { "osd": 610, "primary": false, "shard": 7, "errors": [ "missing" ] }, { "osd": 700, "primary": false, "shard": 3, "errors": [ "missing" ] }, { "osd": 736, "primary": false, "shard": 10, "errors": [ "missing" ] }, { "osd": 764, "primary": false, "shard": 5, "errors": [ "missing" ] } ] } ] }

3 4

Error ENOENT: Module not found - ceph orch commands stoppd working
by Torkil Svensgaard 12 Nov '24

12 Nov '24

Hi 18.2.4. After failing over the active manager ceph orch commands seems to have stopped working. There's this in the mgr log: " 2024-11-12T08:16:30.136+0000 7f1b2d887640 0 log_channel(audit) log [DBG] : from='client.2088861125 -' entity='client.admin' cmd=[{"prefix": "orch osd rm status", "target": ["mon-mgr", ""]}]: dispatch 2024-11-12T08:16:30.136+0000 7f1b23cf4640 -1 no module 'cephadm' 2024-11-12T08:16:30.136+0000 7f1b23cf4640 -1 no module 'cephadm' 2024-11-12T08:16:30.136+0000 7f1b23cf4640 -1 mgr.server reply reply (2) No such file or directory Module not found " The module is still enabled: " [root@ceph-flash1 ~]# ceph mgr module ls MODULE balancer on (always on) crash on (always on) devicehealth on (always on) orchestrator on (always on) pg_autoscaler on (always on) progress on (always on) rbd_support on (always on) status on (always on) telemetry on (always on) volumes on (always on) alerts on cephadm on dashboard on insights on iostat on nfs on prometheus on stats on diskprediction_local - influx - k8sevents - localpool - mds_autoscaler - mirroring - osd_perf_query - osd_support - restful - rgw - rook - selftest - snap_schedule - telegraf - test_orchestrator - zabbix - " Cluster is working: " [root@ceph-flash1 ~]# ceph -s cluster: id: 8ee2d228-ed21-4580-8bbf-0649f229e21d health: HEALTH_WARN noout flag(s) set 5 nearfull osd(s) Degraded data redundancy: 22742466/3557557778 objects degraded (0.639%), 559 pgs degraded, 559 pgs undersized 4 pool(s) nearfull services: mon: 5 daemons, quorum ceph-flash1,ceph-flash2,ceph-flash3,grouchy,klutzy (age 8d) mgr: ceph-flash2.utlhuz(active, since 10m), standbys: ceph-flash3.ciudre, ceph-flash1.erhakb mds: 1/1 daemons up, 2 standby osd: 567 osds: 555 up (since 14h), 555 in (since 4d); 2689 remapped pgs flags noout data: volumes: 1/1 healthy pools: 17 pools, 15521 pgs objects: 619.72M objects, 1.3 PiB usage: 2.3 PiB used, 2.0 PiB / 4.3 PiB avail pgs: 22742466/3557557778 objects degraded (0.639%) 135987731/3557557778 objects misplaced (3.823%) 12832 active+clean 2111 active+remapped+backfill_wait 479 active+undersized+degraded+remapped+backfill_wait 80 active+undersized+degraded+remapped+backfilling 19 active+remapped+backfilling io: client: 73 MiB/s rd, 5.4 MiB/s wr, 574 op/s rd, 169 op/s wr recovery: 2.9 GiB/s, 1.01k objects/s " Suggestions? I tried failing over the manager again which didn't help. Mvh. Torkil

2 4

Ceph pacific error when add new host
by nguyenvandiep＠baoviet.com.vn 12 Nov '24

12 Nov '24

We have a Ceph cluster run with podman2.0.5 - ceph 16.2.4 Our public network is 10.29.1.* and private network is 172.28.1.* We need deploy new host But when run ceph orch host add newhost ip ceph return added successfully, BUT: No daemon were deployed on newhost podman on newhost pulled the right Ceph image when we try ceph orch daemon add crash newhost on newhost, crash container started and running BUT on master node, crash.newhost is STARTING Could anyone help us. Thank you all

2 3

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

quincy v17.2.8 QE Validation status
by Yuri Weinstein 11 Nov '24

11 Nov '24

Details of this release are summarized here: https://tracker.ceph.com/issues/68643#note-2 Release Notes - TBD Seeking approvals/reviews for: rados - Laura, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x (quincy) - Josh pls approve upgrade/pacific-x (quincy) - Josh pls approve upgrade/quincy-p2p - Known issue IIRC, Laura, Casey pls confirm/approve client-upgrade-quincy-reef - in progress powercycle - Brad pls approve ceph-volume - Guillaume pls take a look Please reply to this email with approval and/or trackers of known issues/PRs to address them. gibba and LRC upgrades -- N/A for quincy now after reef release. Thx YuriW

9 19

Ceph Steering Committee 2024-11-11
by Gregory Farnum 11 Nov '24

11 Nov '24

We had a short meeting today, with no pre-set topics. The main topic that came up is a bug detected on upgrades to squid when using the balancer in large clusters: https://tracker.ceph.com/issues/68657. The RADOS team would like to do a squid point release once our quincy release is out the door (imminent). Guarav from the Ceph ambassadors program is doing planning for FOSDEM. He's looking for anybody in Ceph technical leadership who is interested and available for co-presenting a Ceph talk. Also, the election closes for our executive council next week. We'll see the results then. -Greg

1 0

NFS and Service Dependencies
by Alex Buie 10 Nov '24

10 Nov '24

Hello all, Facing an issue with my first ceph with cephfs deployment. I have 2 admin (mgr) hosts, 3 mon/mds hosts, and 3 OSD hosts with 3 OSDs each. I also have been using the OSDs to run the nfs service that my hypervisor connects to. I have these service definitions I've applied: service_type: nfs service_id: nfs service_name: nfs.nfs placement: count: 1 label: nfs spec: port: 2049 virtual_ip: 172.19.19.165 service_type: ingress service_id: nfs.nfs service_name: ingress.nfs.nfs placement: count: 1 label: nfs spec: backend_service: nfs.nfs monitor_port: 9049 virtual_ip: 172.19.19.165 keepalive_only: true and I have tagged the 3 OSD nodes with the `nfs` label. Generally, both the ingress and the nfs service start on the same node (osd01, for example). But sometimes, errors occur with the nfs service and it gets rescheduled and ends up on a different node (osd03, for example) - and because the ingress service isn't running on this node, it breaks my NFS mount and the hypervisor gets very unhappy. Is there a way I can specify some kind of scheduling dependency between the nfs.nfs and ingress.nfs.nfs services so that they will get scheduled to run on the same node so that the virtual IP is present for nfs to bind to? Or, am I doing something wrong here? Thanks a bunch! *Alex*

3 2

OSD META Capacity issue of rgw ceph cluster
by Jaemin Joo 08 Nov '24

08 Nov '24

Hi All, I have RADOS gw(rgw) ceph cluster (v18.2.1) applied Erasure Coding and 10009 shards to save a lot of objects in a bucket root:/home/ceph# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.00490 1.00000 5.0 GiB 318 MiB 58 MiB 1 KiB 260 MiB 4.7 GiB 6.21 0.94 143 up 1 hdd 0.00490 1.00000 5.0 GiB 330 MiB 56 MiB 2 KiB 274 MiB 4.7 GiB 6.45 0.98 155 up 2 hdd 0.00490 1.00000 5.0 GiB 372 MiB 60 MiB 1 KiB 313 MiB 4.6 GiB 7.28 1.10 148 up (The osds are just data pool, it's not index pool for rgw..I think a lot of shards are not impact data pool META size) the OSD META occupied same as data size. I know that OSD META is rocksdb. I think oversized even though rocksdb META is needed. I can use osd compact to decrease META size. but I want to know why OSD META size is big than I expect, Is there way to decrease the META size by changing parameter instead of osd compact? I guess that some of "bluestore_rocksdb_options" can impact the META size. And let me know if you need the other parameter of ceph cluster to figure out the symptom.

2 3

1 stray daemon(s) not managed by cephadm
by Michel Niyoyita 08 Nov '24

08 Nov '24

Hello team, I have run a ceph cluster using cephadm Reef , on ubuntu 22.04 , the cluster is composed by 3 mons and 3 hosts running OSDs, after deployment my cluster is in warning state with : [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm . I have run ceph heath detail and the out put is : HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon mon.ceph-osd3 on host ceph-osd3 not managed by cephadm could you please help to solve this problem? Best regards Michel

3 2

Fwd: Call for participation: Software Defined Storage devroom at FOSDEM 2025
by Jan Fajerski 08 Nov '24

08 Nov '24

----- Forwarded message from Jan Fajerski <jan(a)fajerski.name> ----- Date: Mon, 28 Oct 2024 17:31:49 +0100 From: Jan Fajerski <jan(a)fajerski.name> To: fosdem(a)lists.fosdem.org Cc: Subject: [devroom-managers] Call for participation: Software Defined Storage devroom at FOSDEM 2025 FOSDEM is a free software event that offers open source communities a place to meet, share ideas and collaborate. It is well known for being highly developer-oriented and in the past brought together 8000+ participants from all over the world. Its home is in the city of Brussels (Belgium). FOSDEM 2025 will take place as an in-person event during the weekend of February 1/2 2025. More details about the event can be found at http://fosdem.org/ ** Call For Participation The Software Defined Storage devroom will go into its 9. iteration for talks around Open Source Software Defined Storage projects, management tools and real world deployments. Presentation topics could include but are not limited too: - Your work on a SDS project like Ceph, Gluster, Garage, OpenEBS, CORTX or Longhorn - Your work on or with SDS related projects like Samba, OpenStack SWIFT or Container Storage Interface - Management tools for SDS deployments - Monitoring tools for SDS clusters ** Important dates: - Dec 1, 2024: submission deadline for talk proposals - Dec 8, 2024: announcement of the final schedule - Feb 1, 2025: Software Defined Storage dev room Talk proposals will be reviewed by a steering committee: - Niels de Vos (IBM) - Jan Fajerski (Red Hat) - Hugo Gonzalez Labrador (CERN) - Abhishek Lekshmanan (CERN) - Richard Bachmann (CERN) We also welcome additional volunteers to help with making this devroom a success. Please submit talks at https://fosdem.org/submit. - Important! Select the "Software Defined Storage devroom" track. Hope to hear from you soon! And please forward this announcement. If you have any further questions, please write to the mailing list at storage-devroom(a)lists.fosdem.org and we will try to answer as soon as possible. Thanks! _______________________________________________ devroom-managers mailing list devroom-managers(a)lists.fosdem.org https://lists.fosdem.org/listinfo/devroom-managers ----- End forwarded message -----

1 0

Re: osd removal leaves 'stray daemon'
by Frédéric Nass 08 Nov '24

08 Nov '24

Hi, We're encountering this unexpected behavior as well. This tracker [1] was created 4 months ago. Regards, Frédéric. [1] https://tracker.ceph.com/issues/67018 ----- Le 6 Déc 22, à 8:41, Holger Naundorf naundorf(a)rz.uni-kiel.de a écrit : > Hello, > a mgr failover did not change the situation - the osd still shows up in > the 'ceph node ls' - I assume that this is more or less 'working as > intended' as I did ask for the OSD to be kept in the CRUSH map to be > replacd later - but as we are still not so experienced with Ceph here I > wanted to get some input from other sites. > > Regards, > Holger > > On 30.11.22 16:28, Adam King wrote: >> I typically don't see this when I do OSD replacement. If you do a mgr >> failover ("ceph mgr fail") and wait a few minutes does this still show >> up? The stray daemon/host warning is roughly equivalent to comparing the >> daemons in `ceph node ls` and `ceph orch ps` and seeing if there's >> anything in the former but not the latter. Sometimes I have seen the mgr >> will have some out of data info in the node ls and a failover will >> refresh it. >> >> On Fri, Nov 25, 2022 at 6:07 AM Holger Naundorf <naundorf(a)rz.uni-kiel.de >> <mailto:naundorf@rz.uni-kiel.de>> wrote: >> >> Hello, >> I have a question about osd removal/replacement: >> >> I just removed an osd where the disk was still running but had read >> errors, leading to failed deep scrubs - as the intent is to replace >> this >> as soon as we manage to get a spare I removed it with the >> '--replace' flag: >> >> # ceph orch osd rm 224 --replace >> >> After all placement groups are evacuated I now have 1 osd down/out >> and showing as 'destroyed': >> >> # ceph osd tree >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> (...) >> 214 hdd 14.55269 osd.214 up 1.00000 1.00000 >> 224 hdd 14.55269 osd.224 destroyed 0 1.00000 >> 234 hdd 14.55269 osd.234 up 1.00000 1.00000 >> (...) >> >> All as expected - but now the health check complains that the >> (destroyed) osd is not managed: >> >> # ceph health detail >> HEALTH_WARN 1 stray daemon(s) not managed by cephadm >> [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm >> stray daemon osd.224 on host ceph19 not managed by cephadm >> >> Is this expected behaviour and I have to live with the yellow check >> until we get a replacement disk and recreate the osd or did something >> not finish correctly? >> >> Regards, >> Holger >> >> -- >> Dr. Holger Naundorf >> Christian-Albrechts-Universität zu Kiel >> Rechenzentrum / HPC / Server und Storage >> Tel: +49 431 880-1990 >> Fax: +49 431 880-1523 >> naundorf(a)rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> <mailto:ceph-users-leave@ceph.io> >> > > -- > Dr. Holger Naundorf > Christian-Albrechts-Universität zu Kiel > Rechenzentrum / HPC / Server und Storage > Tel: +49 431 880-1990 > Fax: +49 431 880-1523 > naundorf(a)rz.uni-kiel.de > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 4

Squid 19.2.0 balancer causes restful requests to be lost
by Chris Palmer 07 Nov '24

07 Nov '24

I've just upgraded a test cluster from 18.2.4 to 19.2.0. Package install on centos 9 stream. Very smooth upgrade. Only one problem so far... The MGR restful api calls work fine. EXCEPT whenever the balancer kicks in to find any new plans. During the few seconds that the balancer takes to run, all REST calls seem to be completely dropped. The MGR log file normally logs the POST requests, but the ones during these few seconds don't appear at all. This causes our monitoring to keep raising alarms. The cluster is in a completely stable state, HEALTH_OK, very little activity, just the occasional scrubs. We use the restful API for monitoring (using the Ceph for Zabbix Agent 2 plugin, as Zabbix is the over-arching monitoring platform in the data centre). I haven't yet checked the memory leak problems that we (like many) were having, because I have been chasing this new problem. The problem is quite repeatable. To diagnose I use the zabbix_get utility to query every second. Whenever the MGR log file shows the balancer kick in the REST requests time out (after 3 seconds - not sure whether the utility or the MGR is timing them out - I suspect the utility). They normally complete after a small fraction of a second. With the balancer disabled the REST interface works reliably again. The problem does not occur pre-squid. Anyone any ideas, or shall I raise a bug? Thanks, Chris

6 9

Recreate Destroyed OSD
by Dave Hall 07 Nov '24

07 Nov '24

Hello. Sorry if it appears that I am reposting the same issue under a different topic. However, I feel that the problem has moved and I now have different questions. At this point I have, I believe, removed all traces of OSD.12 from my cluster - based on steps in the Reef docs at https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#. I have further located and removed the WAL/DB LV on an associated NVMe drive (shared with 3 other OSDs). I don't believe the instructions for replacing an OSD (ceph-volume lvm prepare) still apply, so I have been trying to work with the instructions under ADDING AN OSD (MANUAL). However, since my installation is containerized (Podman), it is unclear which steps should be issued on the host and which within 'cephadm shell'. There is also another ambiguity: In step 3 the instruction is to 'mkfs -t {fstype}' and then to 'mount -o user_xattr'. However, which fs type? After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets throws errors about the keyring file. So, are these the right instructions to be using in a containerized installation? Are there, in general, alternate documents for containerized installations? Lastly, the above cited instructions don't say anything about the separate WAL/DB LV. Please advise. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

5 17

[RGW] Enable per user/bucket performance counters
by Nathan MALO 07 Nov '24

07 Nov '24

Hi, I come to you with a quite simple question. I am running a REEF cluster in 18.2.2, deployed in k3s via rook 1.14.8. I want to enable per user and per bucket performance counters on my RGW. According to this doc https://docs.ceph.com/en/latest/radosgw/metrics/#user-bucket-counter-caches I need to run ceph config set client.rgw rgw_{user,bucket}_counters_cache true. But when I run the commands in the ceph toolbox, I end up with the following errors: bash-4.4$ ceph config set client.rgw rgw_user_counters_cache true Error EINVAL: unrecognized config option 'rgw_user_counters_cache' bash-4.4$ ceph config set client.rgw rgw_bucket_counters_cache true Error EINVAL: unrecognized config option 'rgw_bucket_counters_cache' This doc https://www.ibm.com/docs/en/storage-ceph/7?topic=counters-viewing-performan… mentions a different command regarding the client.rgw. So I did the following: bash-4.4$ ceph auth list | grep rgw client.bootstrap-rgw caps: [mon] allow profile bootstrap-rgw client.rgw.ceph.s3.a bash-4.4$ ceph config set client.rgw.ceph.s3.a rgw_user_counters_cache true Error EINVAL: unrecognized config option 'rgw_user_counters_cache' bash-4.4$ ceph config set client.rgw.ceph.s3.a rgw_bucket_counters_cache true Error EINVAL: unrecognized config option 'rgw_bucket_counters_cache' When I run all this commands with --force, the keys are created. I restarted my RGWs pods but the keys do not seem to be taken into account as I do not have any ceph_rgw_op* metric. Did anyone successfully managed to enable this ? What am I missing ? Thank your for your help !

2 1