- ceph-users - lists.ceph.io

external multipath disk not mounted after power off/on the server
by Iban Cabrillo 27 Feb '25

27 Feb '25

Dear cephers, We have a series of devices that mount several SATA disks via an external cabinet. These disks have 4 paths managed by multipath. (local disks about 20 work perfectly) cephosd23:~ # multipath -ll mpathe (35000c500d88657e3) dm-30 LENOVO-X,ST14000NM004J size=13T features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 2:0:9:0 sdae 65:224 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:21:0 sdap 66:144 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:33:0 sdba 67:64 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 2:0:45:0 sdbl 67:240 active ready running mpathd (35000c500d886ac87) dm-23 LENOVO-X,ST14000NM004J size=13T features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 2:0:2:0 sdx 65:112 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:14:0 sdai 66:32 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 2:0:26:0 sdat 66:208 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 2:0:38:0 sdbe 67:128 active ready running mpathc (35000c500d87a5c5f) dm-20 LENOVO-X,ST14000NM004J The disks are accessible from the server, but Ceph has not been able to mount them and they are not accessible from Ceph. ... tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-110 tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-113 tmpfs 126G 0 126G 0% /var/lib/ceph/osd/ceph-112 the directory is empty and of course: /usr/bin/ceph-osd -f --cluster ceph --id 110 --setuser ceph --setgroup ceph 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-110/keyring: (2) No such file or directory 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 AuthRegistry(0x55841ec96940) no keyring found at /var/lib/ceph/osd/ceph-110/keyring, disabling cephx 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-110/keyring: (2) No such file or directory 2025-02-27T15:22:25.833+0100 7f799db60d80 -1 AuthRegistry(0x7ffd668863a0) no keyring found at /var/lib/ceph/osd/ceph-110/keyring, disabling cephx Is there any way to fix this error and mount them again? The data should be intact. Best regards -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

7 10

Re: Free space
by quaglio＠bol.com.br 27 Feb '25

27 Feb '25

1 0

Re: Free space
by quaglio＠bol.com.br 27 Feb '25

27 Feb '25

1 0

Statistics?
by Jan Marek 27 Feb '25

27 Feb '25

Hello, I have a new created ceph cluster, I had some issues with disks, and now I have this 'ceph -s' list: health: HEALTH_WARN Degraded data redundancy: 103798/888 objects degraded (11688.964%), 188 pgs degraded, 192 pgs undersized 24 daemons have recently crashed services: mon: 3 daemons, quorum ceph0,ceph1,ceph2 (age 22m) mgr: ceph1.xzkgcv(active, since 40m), standbys: ceph0.qseeqr mds: 1/1 daemons up, 1 standby osd: 64 osds: 64 up (since 22m), 64 in (since 22m); 198 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 2065 pgs objects: 296 objects, 965 MiB usage: 63 GiB used, 1.4 PiB / 1.4 PiB avail pgs: 103798/888 objects degraded (11688.964%) 3770/888 objects misplaced (424.550%) 1865 active+clean 188 active+recovery_wait+undersized+degraded+remapped 7 active+recovery_wait+remapped 4 active+recovering+undersized+remapped 1 active+remapped+backfill_wait io: client: 11 KiB/s wr, 0 op/s rd, 2 op/s wr recovery: 0 B/s, 198 objects/s Please, pay attention to percentage. It seems, that there is swapped numerator and denominator of fraction, isn't it? Sincerely Jan Marek --. Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

2 2

Discussion on issues encountered while creating osds
by hera sami 26 Feb '25

26 Feb '25

Hi team, glad to connect with you all. I am seeing an issue with ceph deployed using cephadm. I am using the "reef" version of ceph. I had written an ansible script for bringing up ceph as described in the ceph official documentation. My script works fine most of the time but on some vms I am seeing an issue. I am creating a cluster of 3 nodes. The bootstrapping went well and other 2 nodes got added to the cluster. [image: image.png] While my mon and mgr are added successfully, the osd on 2 nodes are not coming up. While the device for osd creation is AVAILABLE as seen from below, when i run the ceph orch add osd command, the ceph orch command hangs indefinitely. [image: image.png] [image: image.png] [image: image.png] I am trying to create osd on sftp-ntp-1 and ceph-non-sftp-1 with device /dev/sdb. Althought the device is AVAILABLE, but when I run the ceph och add osd command, the ceph orch command hangs indefinitely. Neither the osd gets added nor any ceph orch or ceph osd commands accessible. Sometimes ceph orch commands resumes when executed "sudo systemctl restart ceph.target", but no luck on osd creation. Also in these nodes, I dont see the logs of Reconfig daemon osd in /var/log/ceph/cephadm.log unlike other daemons like mon, mgr root@sftp-ntp-1:/var/log# cat /var/log/ceph/cephadm.log | grep "Reconfig daemon" 2025-02-27 06:49:09,426 7f37b62ce1c0 INFO Reconfig daemon mgr.sftp-ntp-1.hkvpch ... 2025-02-27 06:49:09,807 7fbbcc9d21c0 INFO Reconfig daemon mon.sftp-ntp-1 ... 2025-02-27 06:49:10,203 7f9cd5a691c0 INFO Reconfig daemon crash.sftp-ntp-1 ... 2025-02-27 06:49:10,581 7fbe2e4e11c0 INFO Reconfig daemon ceph-exporter.sftp-ntp-1 ... This is seen on only few vms and I am not able to understand what is the root cause behind the issue. It would be much appreciated if I can get any sort of idea on this. Thanks, Hera

1 0

Looking for 'rados df' output command explanation for some columns
by jbareapa＠redhat.com 26 Feb '25

26 Feb '25

Hello folks I'm trying to understand some columns of 'rados df' output command, and i appreciate some feedback around. Sorry if was posted before. This cluster is pretty little in HW specs # rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR .mgr 1.8 MiB 2 0 6 0 0 0 526 469 KiB 715 12 MiB 0 B 0 B cephfs1.data 32 KiB 4 0 8 0 0 0 21 20 KiB 47 13 KiB 0 B 0 B cephfs1.metadata 902 MiB 190 0 380 0 0 0 6941 5.3 GiB 909148 2.5 GiB 0 B 0 B cephfs2.data 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B cephfs2.metadata 151 KiB 22 0 66 0 0 0 40 40 KiB 47 13 KiB 0 B 0 B firstpoolrbd 30 KiB 5 0 15 0 0 0 3561 2.8 MiB 58 27 KiB 0 B 0 B I have been 'punishing' on purpose 'cephfs1.metadata' on some clients creating and deleting files/dirs. My question are for the columns below , the values showed are 'accumulatve' stats figures, right? I mean, from what i did, that makes sense. It cannot be values per second, right? Is there a way to reset such columns values? RD_OPS Total number of read operations requested for this pool. RD Total number of bytes read from this pool. WR_OPS Total number of write operations requested for this pool. WR Total number of bytes written to the pool. Note that , writing the same object many times will result is that the pool's usage will remain the same but the number of bytes written to the pool will grow. Thanks in advance for any comment

1 0

backfill_toofull not clearing on Reef
by Deep Dish 26 Feb '25

26 Feb '25

Hello, I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was replaced (same capacity). For the past week or so the cluster has been recovering, slowly, and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity. I tried setting the following without any result: ceph osd set-backfillfull-ratio 0.90 # ceph -s ... health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull 11 pgs not deep-scrubbed in time ... data: volumes: 1/1 healthy pools: 13 pools, 1665 pgs objects: 27.81M objects, 79 TiB usage: 197 TiB used, 413 TiB / 610 TiB avail pgs: 16341449/128092605 objects misplaced (12.758%) 1371 active+clean 150 active+remapped+backfill_wait 121 active+remapped+backfill_wait+backfill_toofull 16 active+remapped+backfill_toofull 6 active+clean+scrubbing+deep 1 active+remapped+backfilling io: client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr recovery: 16 MiB/s, 4 objects/s # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up 9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up 16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up 27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up 35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6 TiB 35.51 1.10 113 up 43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5 TiB 37.92 1.17 106 up 51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8 TiB 33.63 1.04 108 up 59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9 TiB 32.70 1.01 101 up 67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9 TiB 31.99 0.99 112 up 74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2 TiB 28.68 0.89 110 up 3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5 TiB 28.45 0.88 112 up 11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1 TiB 21.50 0.67 115 up 19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6 TiB 36.51 1.13 115 up 25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2 TiB 29.08 0.90 91 up 33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2 TiB 27.27 0.84 109 up 41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5 TiB 24.36 0.75 105 up 49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1 TiB 29.98 0.93 107 up 56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0 TiB 31.77 0.98 116 up 62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5 TiB 38.47 1.19 108 up 69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1 TiB 30.05 0.93 109 up 5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2 TiB 31.40 0.97 109 up 13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3 TiB 31.12 0.96 116 up 22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8 TiB 34.72 1.07 114 up 31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0 TiB 31.08 0.96 101 up 39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7 TiB 35.60 1.10 110 up 47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0 TiB 31.89 0.99 108 up 55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4 TiB 39.43 1.22 116 up 65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9 TiB 32.62 1.01 116 up 73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6 TiB 36.13 1.12 111 up 78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7 TiB 35.50 1.10 118 up 4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6 TiB 27.11 0.84 131 up 12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9 TiB 23.80 0.74 106 up 20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4 TiB 26.11 0.81 106 up 28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5 TiB 38.59 1.19 121 up 36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9 TiB 31.98 0.99 108 up 44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8 TiB 33.68 1.04 90 up 52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1 TiB 29.76 0.92 108 up 60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0 TiB 31.04 0.96 104 up 68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8 TiB 33.83 1.05 119 up 76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7 TiB 35.77 1.11 107 up 6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1 TiB 33.01 1.02 111 up 14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3 TiB 30.96 0.96 84 up 23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8 TiB 33.96 1.05 95 up 30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5 TiB 38.67 1.20 93 up 38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0 TiB 31.72 0.98 101 up 46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7 TiB 36.05 1.12 81 up 53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6 TiB 36.12 1.12 88 up 63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2 TiB 42.94 1.33 87 up 75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up 79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6 TiB 36.90 1.14 93 up 7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1 TiB 22.22 0.69 98 up 15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5 TiB 28.50 0.88 114 up 21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6 TiB 22.60 0.70 88 up 29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8 TiB 34.20 1.06 114 up 37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9 TiB 33.02 1.02 128 up 45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6 TiB 36.26 1.12 109 up 54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7 TiB 35.04 1.08 111 up 61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2 TiB 27.87 0.86 103 up 70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0 TiB 31.86 0.99 109 up 77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8 TiB 32.35 1.00 112 up 2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2 TiB 32.27 1.00 115 up 10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9 TiB 24.28 0.75 97 up 18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3 TiB 26.63 0.82 107 up 26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6 TiB 35.60 1.10 111 up 34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6 TiB 36.17 1.12 106 up 42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1 TiB 30.17 0.93 111 up 50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6 TiB 36.05 1.12 101 up 58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4 TiB 39.11 1.21 122 up 66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3 TiB 26.73 0.83 98 up 72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0 TiB 29.57 0.92 110 up 0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3 TiB 30.63 0.95 119 up 8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7 TiB 26.77 0.83 112 up 17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5 TiB 38.79 1.20 107 up 24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9 TiB 32.22 1.00 116 up 32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7 TiB 35.56 1.10 94 up 40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3 TiB 39.44 1.22 106 up 48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9 TiB 32.22 1.00 114 up 57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2 TiB 28.40 0.88 95 up 64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6 TiB 37.42 1.16 130 up 71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9 TiB 32.01 0.99 105 up TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413 TiB 32.31 MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 86.15359 root default -19 10.00000 host 01stor28 1 hdd 1.00000 osd.1 up 1.00000 1.00000 9 hdd 1.00000 osd.9 up 1.00000 1.00000 16 hdd 1.00000 osd.16 up 1.00000 1.00000 27 hdd 1.00000 osd.27 up 1.00000 1.00000 35 hdd 1.00000 osd.35 up 1.00000 1.00000 43 hdd 1.00000 osd.43 up 1.00000 1.00000 51 hdd 1.00000 osd.51 up 1.00000 1.00000 59 hdd 1.00000 osd.59 up 1.00000 1.00000 67 hdd 1.00000 osd.67 up 1.00000 1.00000 74 hdd 1.00000 osd.74 up 1.00000 1.00000 -21 10.00000 host 01stor32 3 hdd 1.00000 osd.3 up 1.00000 1.00000 11 hdd 1.00000 osd.11 up 1.00000 1.00000 19 hdd 1.00000 osd.19 up 1.00000 1.00000 25 hdd 1.00000 osd.25 up 1.00000 1.00000 33 hdd 1.00000 osd.33 up 1.00000 1.00000 41 hdd 1.00000 osd.41 up 1.00000 1.00000 49 hdd 1.00000 osd.49 up 1.00000 1.00000 56 hdd 1.00000 osd.56 up 1.00000 1.00000 62 hdd 1.00000 osd.62 up 1.00000 1.00000 69 hdd 1.00000 osd.69 up 1.00000 1.00000 -15 10.00000 host 02stor23 5 hdd 1.00000 osd.5 up 1.00000 1.00000 13 hdd 1.00000 osd.13 up 1.00000 1.00000 22 hdd 1.00000 osd.22 up 1.00000 1.00000 31 hdd 1.00000 osd.31 up 1.00000 1.00000 39 hdd 1.00000 osd.39 up 1.00000 1.00000 47 hdd 1.00000 osd.47 up 1.00000 1.00000 55 hdd 1.00000 osd.55 up 1.00000 1.00000 65 hdd 1.00000 osd.65 up 1.00000 1.00000 73 hdd 1.00000 osd.73 up 1.00000 1.00000 78 hdd 1.00000 osd.78 up 1.00000 1.00000 -11 10.00000 host 02stor27 4 hdd 1.00000 osd.4 up 1.00000 1.00000 12 hdd 1.00000 osd.12 up 1.00000 1.00000 20 hdd 1.00000 osd.20 up 1.00000 1.00000 28 hdd 1.00000 osd.28 up 1.00000 1.00000 36 hdd 1.00000 osd.36 up 1.00000 1.00000 44 hdd 1.00000 osd.44 up 1.00000 1.00000 52 hdd 1.00000 osd.52 up 1.00000 1.00000 60 hdd 1.00000 osd.60 up 1.00000 1.00000 68 hdd 1.00000 osd.68 up 1.00000 1.00000 76 hdd 1.00000 osd.76 up 1.00000 1.00000 -17 16.15359 host 03stor23 6 hdd 1.00000 osd.6 up 1.00000 1.00000 14 hdd 1.00000 osd.14 up 1.00000 1.00000 23 hdd 1.00000 osd.23 up 1.00000 1.00000 30 hdd 1.00000 osd.30 up 1.00000 1.00000 38 hdd 1.00000 osd.38 up 1.00000 1.00000 46 hdd 1.00000 osd.46 up 1.00000 1.00000 53 hdd 1.00000 osd.53 up 1.00000 1.00000 63 hdd 1.00000 osd.63 up 1.00000 1.00000 75 hdd 7.15359 osd.75 up 1.00000 1.00000 79 hdd 1.00000 osd.79 up 1.00000 1.00000 -13 10.00000 host 03stor27 7 hdd 1.00000 osd.7 up 1.00000 1.00000 15 hdd 1.00000 osd.15 up 1.00000 1.00000 21 hdd 1.00000 osd.21 up 1.00000 1.00000 29 hdd 1.00000 osd.29 up 1.00000 1.00000 37 hdd 1.00000 osd.37 up 1.00000 1.00000 45 hdd 1.00000 osd.45 up 1.00000 1.00000 54 hdd 1.00000 osd.54 up 1.00000 1.00000 61 hdd 1.00000 osd.61 up 1.00000 1.00000 70 hdd 1.00000 osd.70 up 1.00000 1.00000 77 hdd 1.00000 osd.77 up 1.00000 1.00000 -3 10.00000 host 04stor28 2 hdd 1.00000 osd.2 up 1.00000 1.00000 10 hdd 1.00000 osd.10 up 1.00000 1.00000 18 hdd 1.00000 osd.18 up 1.00000 1.00000 26 hdd 1.00000 osd.26 up 1.00000 1.00000 34 hdd 1.00000 osd.34 up 1.00000 1.00000 42 hdd 1.00000 osd.42 up 1.00000 1.00000 50 hdd 1.00000 osd.50 up 1.00000 1.00000 58 hdd 1.00000 osd.58 up 1.00000 1.00000 66 hdd 1.00000 osd.66 up 1.00000 1.00000 72 hdd 1.00000 osd.72 up 1.00000 1.00000 -7 10.00000 host 04stor32 0 hdd 1.00000 osd.0 up 1.00000 1.00000 8 hdd 1.00000 osd.8 up 1.00000 1.00000 17 hdd 1.00000 osd.17 up 1.00000 1.00000 24 hdd 1.00000 osd.24 up 1.00000 1.00000 32 hdd 1.00000 osd.32 up 1.00000 1.00000 40 hdd 1.00000 osd.40 up 1.00000 1.00000 48 hdd 1.00000 osd.48 up 1.00000 1.00000 57 hdd 1.00000 osd.57 up 1.00000 1.00000 64 hdd 1.00000 osd.64 up 1.00000 1.00000 71 hdd 1.00000 osd.71 up 1.00000 1.00000 # ceph health detail HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting [0,67,4] pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting [71,44,22] pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting [58,51,23] pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting [69,45,22] pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting [24,3,63] pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting [35,50,14] pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting [28,16,69] pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6] pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting [78,26,29] pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30] pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting [43,11,17] pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting [68,33,2] pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66] pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting [78,17,6] pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting [57,67,13] pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38] pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting [1,0,28] pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting [47,14,66] pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14] pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting [27,45,38] pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72] pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting [52,69,46] pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting [55,26,32] pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting [34,59,11] pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2] pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting [55,70,46] pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23] pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting [37,25,59] pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23] pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting [11,26,24] pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting [59,60,46] pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6] pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting [35,57,46] pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting [28,9,72] pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting [5,43,72] pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting [70,72,14] pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting [78,10,23] pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting [66,7,53] pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13] pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting [25,15,63] pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting [78,76,34] pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting [24,76,54,16,25,79,22,2] pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting [10,9,56,4,47,61,32,79] pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting [70,41,36,48,23,31,1,2] pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting [10,29,78,25,1,44,64,79] pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting [28,26,15,23,57,16,25,78] pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting [59,46,19,7,72,68,55,8] pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting [76,54,25,31,0,46,58,59] pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting [31,76,1,66,8,56,79,21] pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting [74,0,70,44,39,34,33,6] pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting [44,21,43,18,69,64,38,31] [WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000 pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000 pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000 pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000 pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000 pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000 pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000 pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000 pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000 pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000 pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000 pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000 pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000 pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000 pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000 pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000 pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000 pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000 pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000 pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000 pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000 pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000 pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000 pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000 pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000

7 9

Problem with S3 presigned URLs & CORS & Object tagging
by Haarländer, Markus 26 Feb '25

26 Feb '25

Hi list, We encountered a problem with presigned URLs for putting objects in connection with CORS and S3 Object tagging in Ceph v17.2.7 It works fine with v 16.2.15 and it works fine if the tagging is disabled. Here are the steps to reproduce: 1. Create a CORS rule for a bucket called "my-bucket": { "CORSRules": [ { "AllowedOrigins": ["*"], "AllowedHeaders": ["*"], "AllowedMethods": ["PUT", "GET"], "ExposeHeaders": ["ETag", "Accept-Ranges", "Content-Encoding", "Content-Range"] } ] } 2. Create a presigned URL to upload data to the key "test.txt". A tag called "test" should be applied to the resulting object. The following presigned url is returned eg. by the Java SDK. It automatically contains the "x-amz-tagging" in the X-Amz-Signed-Headers parameter. https://my-bucket.my-s3-server/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-… 3. Simulate a preflight OPTIONS request with Origin and Access-Control-Request-Method headers, as the browser would do when trying to PUT to the presigned URL curl --request OPTIONS 'https://my-bucket.my-s3-server/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-…' \ --header 'Origin: https://example.org' \ --header 'Access-Control-Request-Method: PUT' The server replies with 403, no CORS headers and the following body: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000006b5686df4e08cfcd-0067bdc7f3-218e3740-default</RequestId><HostId>218e3740-default-default</HostId></Error> Expectation: The server should reply with 200 and with CORS headers Additional observations: - The exact same workflow works on Ceph v16.2.15, with tagging - It works with Ceph v17.2.7 if we do not use the tagging (then no x-amz-tagging is returned in the X-Amz-Signed-Headers parameter) - It works with Ceph v17.2.7 for non-CORS environments (then no OPTIONS request is done, the PUT request itself works with tagging) - It works with Ceph v17.2.7 if we add the "x-amz-tagging" header to the OPTIONS request. But this does not lead anywhere, as for CORS, every browser creates the OPTIONS request itself for a preflight request and removes all custom headers. There's no possibility to change that. Any ideas or hints are very welcome. Thank you. Markus

2 2

S3 Bucket Upload - Boto3 - Disable or Enable Checksum on bucket/rgw
by Devender Singh 25 Feb '25

25 Feb '25

Hello Using boto3 trying to upload files to s3 bucket, when I am using boto version 1.35 it works fine, when upgrading it to 1.36 doesn’t work, facing below error, python3 s3_upload.py Error testing the new bucket: An error occurred (XAmzContentSHA256Mismatch) when calling the PutObject operation: None But works with AWS s3 Bucket. “https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/“ Is there a way to enable/disable checksum ? Regards Dev

3 2

is upgrade from quincy to squid supported?
by Simon Oosthoek 25 Feb '25

25 Feb '25

Hi list, we have a ceph cluster running ubuntu 22.04 and quincy packages from ubuntu. To keep up with the releases of Ceph, we want to upgrade to a more recent stable release... We could upgrade to ubuntu 24.04 LTS (re-install actually) but the ceph version then skips 18.2.x and goes directly to 19.2.x, if this is a supported upgrade, that would be the least amount of work, since upgrade of the OS will be needed at some point anyway. We are not planning a move to cephadm yet. I suppose if we need to install 18 before 19, we'd need to add the ceph repo for ubuntu package to upgrade and then upgrade to ubuntu 24.04's ceph 19.2.x after reinstalling all the ceph servers... Has anyone successfully upgraded from 17.2.x to 19.2.x in one upgrade step? (And is this supported?) Cheers /Simon -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

2 1