- ceph-users - lists.ceph.io

Re: MDS crashes to damaged metadata
by Stefan Kooman 24 May '23

24 May '23

On 12/15/22 15:31, Stolte, Felix wrote: > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): Are you willing to share this script? I would like to use it to scan our CephFS before upgrading to 16.2.13. Do you run this script when the filesystem is online / active? Thanks, Gr. Stefan

2 3

cephfs max_file_size
by Dietmar Rieder 24 May '23

24 May '23

Hi, can the cephfs "max_file_size" setting be changed at any point in the lifetime of a cephfs? Or is it critical for existing data if it is changed after some time? Is there anything to consider when changing, let's say, from 1TB (default) to 4TB ? We are running the latest Nautilus release, BTW. Thanks in advance Dietmar

3 5

Slow recovery on Quincy
by Sake Paulusma 24 May '23

24 May '23

We noticed extremely slow performance when remapping is necessary. We didn't do anything special other than assigning the correct device_class (to ssd). When checking ceph status, we notice the number of objects recovering is around 17-25 (with watch -n 1 -c ceph status). How can we increase the recovery process? There isn't any client load, because we're going to migrate to this cluster in the future, so only a rsync once a while is being executed. [ceph: root@pwsoel12998 /]# ceph status cluster: id: da3ca2e4-ee5b-11ed-8096-0050569e8c3b health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set services: mon: 5 daemons, quorum pqsoel12997,pqsoel12996,pwsoel12994,pwsoel12998,prghygpl03 (age 3h) mgr: pwsoel12998.ylvjcb(active, since 3h), standbys: pqsoel12997.gagpbt mds: 4/4 daemons up, 2 standby osd: 32 osds: 32 up (since 73m), 32 in (since 6d); 10 remapped pgs flags noscrub,nodeep-scrub data: volumes: 2/2 healthy pools: 5 pools, 193 pgs objects: 13.97M objects, 853 GiB usage: 3.5 TiB used, 12 TiB / 16 TiB avail pgs: 755092/55882956 objects misplaced (1.351%) 183 active+clean 10 active+remapped+backfilling io: recovery: 2.3 MiB/s, 20 objects/s

8 17

[cephfs-data-scan] Estimate time for scanning extents and inodes
by Justin Li 24 May '23

24 May '23

Dear All, I'm using metadata repair tools to repair a damaged MDS following below document. My storage has about 276TB data. Cephfs-data-scan is using 32 workers. How long will it take to finish scanning extents? What about scanning inodes? It has run 6 hours and metadata pool dropped 1G. Is this normal? Thanks. ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.3 PiB 454 TiB 835 TiB 835 TiB 64.79 ssd 7.1 TiB 6.2 TiB 878 GiB 878 GiB 12.15 TOTAL 1.3 PiB 460 TiB 836 TiB 836 TiB 64.50 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 4096 276 TiB 222.96M 828 TiB 71.28 111 TiB cephfs_metadata 2 32 101 GiB 5.83M 302 GiB 0.09 111 TiB device_health_metrics 10 1 239 MiB 251 716 MiB 0 112 TiB cephfs_ssd 11 32 117 GiB 1.49M 361 GiB 5.83 1.9 TiB https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment Request for assistance can be lodged to the SIT Technical Team using this form<https://deakinesmprod.service-now.com/esc?id=sc_cat_item&sys_id=7afa8fa5db6…> Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 justin.li(a)deakin.edu.au<mailto:justin.li@deakin.edu.au> http://www.deakin.edu.au<http://www.deakin.edu.au/> Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

1 0

RGW: How to restore the index and metadata of a bucket
by huy nguyen 24 May '23

24 May '23

There is a test bucket, I have removed its index and metadata: radosgw-admin bi purge --bucket abccc --yes-i-really-mean-it radosgw-admin metadata rm bucket.instance:abccc:17a4ce99-009e-40f2-a2d2-2afc218ebd9b.425824299.4 Now the index and metadata is gone, but how to clean its data? Or is there any way to restore the index and metadata? I have tried to fix the index, but it doesn't work: radosgw-admin bucket check --check-objects --fix --bucket abccc 2023-05-24 16:46:41.896 7f32d5890a80 -1 ERROR: get_bucket_instance_from_oid failed: -2 2023-05-24 16:46:41.896 7f32d5890a80 0 could not get bucket info for bucket=abccc Same result with bucket rm: radosgw-admin bucket rm --bucket abccc --purge-objects 2023-05-24 16:47:49.439 7fafb7939a80 -1 ERROR: get_bucket_instance_from_oid failed: -2 2023-05-24 16:47:49.439 7fafb7939a80 0 could not get bucket info for bucket=abccc

1 0

Timeout in Dashboard
by mailing-lists 24 May '23

24 May '23

Hey all, im facing a "minor" problem. I do not always get results when going to the dashboard, under Block->Images in the tab Images or Namespaces. The little refresh button will keep spinning and sometimes after several minutes it will finally show something. That is odd, because from the shell I am not seeing any problems like that. rbd namespace ls as well as: rbd ls --namespace someNS ...will give immediately their results. Does anyone know whats causing this? I did not yet find anything in the logs. Best Ken

1 0

[Help appreciated] ceph mds damaged
by Justin Li 23 May '23

23 May '23

Dear All, After a unsuccessful upgrade to pacific, MDS were offline and could not get back on. Checked the MDS log and found below. See cluster info from below as well. Appreciate it if anyone can point me to the right direction. Thanks. MDS log: 2023-05-24T06:21:36.831+1000 7efe56e7d700 1 mds.0.cache.den(0x600 1005480d3b2) loaded already corrupt dentry: [dentry #0x100/stray0/1005480d3b2 [19ce,head] rep(a)0,-2.0<mailto:rep@0,-2.0> NULL (dversion lock) pv=0 v=2154265030 ino=(nil) state=0 0x556433addb80] -5> 2023-05-24T06:21:36.831+1000 7efe56e7d700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x600is fatal because it is a system directory for this rank -4> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco set_want_state: up:active -> down:damaged -3> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco Sending beacon down:damaged seq 5339 -2> 2023-05-24T06:21:36.831+1000 7efe56e7d700 10 monclient: _send_mon_message to mon.ceph-3 at v2:10.120.0.146:3300/0 -1> 2023-05-24T06:21:37.659+1000 7efe60690700 5 mds.beacon.posco received beacon reply down:damaged seq 5339 rtt 0.827966 0> 2023-05-24T06:21:37.659+1000 7efe56e7d700 1 mds.posco respawn! Cluster info: root@ceph-1:~# ceph -s cluster: id: e2b93a76-2f97-4b34-8670-727d6ac72a64 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged services: mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 26h) mgr: ceph-3(active, since 15h), standbys: ceph-1, ceph-2 mds: 0/1 daemons up, 3 standby osd: 135 osds: 133 up (since 10h), 133 in (since 2w) data: volumes: 0/1 healthy, 1 recovering; 1 damaged pools: 4 pools, 4161 pgs objects: 230.30M objects, 276 TiB usage: 836 TiB used, 460 TiB / 1.3 PiB avail pgs: 4138 active+clean 13 active+clean+scrubbing 10 active+clean+scrubbing+deep root@ceph-1:~# ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [ERR] MDS_ALL_DOWN: 1 filesystem is offline fs cephfs is offline because no MDS is active for it. [ERR] MDS_DAMAGE: 1 mds daemon damaged fs cephfs mds.0 is damaged Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment Request for assistance can be lodged to the SIT Technical Team using this form<https://deakinesmprod.service-now.com/esc?id=sc_cat_item&sys_id=7afa8fa5db6…> Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 justin.li(a)deakin.edu.au<mailto:justin.li@deakin.edu.au> http://www.deakin.edu.au<http://www.deakin.edu.au/> Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

3 4

Unable to change port range for OSDs in Pacific
by Renata Callado Borges 23 May '23

23 May '23

Dear all, How are you? I have a Pacific 3 nodes cluster, and the machines do double-duty as Ceph nodes and as Slurm clients. (I am well aware that this is not desirable, but my client wants it like this anyway). Our Slurm install uses the port 6818 for slurmd everywhere. In one of our Ceph/Slurm nodes, Ceph decided that port 6818 is great for an OSD. This prevents slurmd from running properly. Changing the slurmd port causes the Slurm master, slurmctld, to misread the OSD communication as Slurm "Insane length messages". I have tried unsuccessfully to change this port in Slurm and in Ceph. I wonder if someone here can help me limit the ports Ceph uses for its OSDs. I have tried this with no success: [root@darkside1 ~]# cat /etc/ceph/ceph.conf~ # minimal ceph.conf for 1902a026-496d-11ed-b43e-08c0eb320ec2 [global] fsid = 1902a026-496d-11ed-b43e-08c0eb320ec2 mon_host = [v2:172.22.132.188:3300/0,v1:172.22.132.188:6789/0] [osd] ms_bind_port_min = 6830 ms_bind_port_max = 7300 Then I restarted with "systemctl restart ceph.target" but the OSD keeps being re-bound to 6818. I also tried the same config but with the options under the [global] section. No luck there either. Tried reboot the Ceph/Slurm machine, the OSD is re-bound in 6818 also. Could someone help? Thanks in advance! Cordially, Renata.

2 2

Grafana service fails to start due to bad directory name after Quincy upgrade
by Adiga, Anantha 23 May '23

23 May '23

Hi Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. Check if path exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory Check if grafana.fl31ca104ja0201 directory exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4 -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4 Thank you, Anantha

3 5

Deleting a CephFS volume
by Conrad Hoffmann 23 May '23

23 May '23

Hi all, I'm having difficulties removing a CephFS volume that I set up for testing. I've been through this with RBDs, so I do know about `mon_allow_pool_delete`. However, it doesn't help in this case. It is a cluster with 3 monitors. You can find a console log of me verifying that `mon_allow_pool_delete` is indeed true on all monitors but still fail to remove the volume here: https://paste.sr.ht/~bitfehler/4b317e5b8c5823583c1b26b42e8495f54cbd5e2a I am happy to post config or logs as needed, I wasn't sure which ones might be relevant. I scanned monitor and MDS logs, but nothing stuck out. Using the volume worked fine in tests, so I don't think it is broken or anything. Any ideas would be greatly appreciated! Thanks a bunch, Conrad

3 4