I was running into that as well. Setting `osd_mclock_override_recovery_settings` [1] to true allowed me to manage osd_max_backfills again and get recovery to start happening again. It's on my todo list to understand mclock profiles, but resizing PGs was a nightmare with it. Changing to override the recovery settings saved me. [1] https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/#confval... On Fri, Oct 25, 2024 at 11:13 AM Joffrey <joff.au@gmail.com> wrote:
HI,
This is my cluster:
cluster: id: c300532c-51fa-11ec-9a41-0050569c3b55 health: HEALTH_WARN Degraded data redundancy: 2062374/1331064781 objects degraded (0.155%), 278 pgs degraded, 40 pgs undersized 2497 pgs not deep-scrubbed in time 2497 pgs not scrubbed in time
services: mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 (age 9d) mgr: hbgt-ceph1-mon3.gmfzqm(active, since 10d), standbys: hbgt-ceph1-mon2.nteihj, hbgt-ceph1-mon1.thrnnu osd: 96 osds: 96 up (since 9d), 96 in (since 45h); 1588 remapped pgs rgw: 3 daemons active (3 hosts, 2 zones)
data: pools: 16 pools, 2497 pgs objects: 266.22M objects, 518 TiB usage: 976 TiB used, 808 TiB / 1.7 PiB avail pgs: 2062374/1331064781 objects degraded (0.155%) 349917519/1331064781 objects misplaced (26.289%) 1312 active+remapped+backfill_wait 864 active+clean 199 active+recovery_wait+degraded+remapped 38 active+recovery_wait+degraded 33 active+undersized+degraded+remapped+backfill_wait 33 active+recovery_wait+remapped 7 active+recovery_wait 6 active+undersized+degraded+remapped+backfilling 2 active+recovering+remapped 1 active+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+recovery_wait+undersized+degraded+remapped
io: client: 683 KiB/s rd, 2.2 KiB/s wr, 51 op/s rd, 2 op/s wr
No recovery is running and I don't understand why. I have free space:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1784.12231 - 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1 TiB 808 TiB 54.72 1.00 - root default -5 208.09680 - 208 TiB 142 TiB 130 TiB 51 GiB 605 GiB 66 TiB 68.14 1.25 - host hbgt-ceph1-osd01 1 hdd 17.34140 1.00000 17 TiB 11 TiB 11 TiB 33 KiB 49 GiB 5.9 TiB 66.16 1.21 136 up osd.1 3 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 23 GiB 49 GiB 6.3 TiB 63.80 1.17 139 up osd.3 5 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 139 MiB 53 GiB 4.8 TiB 72.31 1.32 142 up osd.5 7 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 51 GiB 5.6 TiB 67.97 1.24 145 up osd.7 9 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 2.2 GiB 49 GiB 6.0 TiB 65.67 1.20 140 up osd.9 11 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 329 MiB 50 GiB 5.5 TiB 68.42 1.25 145 up osd.11 13 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52 GiB 5.1 TiB 70.45 1.29 153 up osd.13 15 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 61 KiB 48 GiB 5.7 TiB 66.85 1.22 144 up osd.15 17 hdd 17.34140 1.00000 17 TiB 11 TiB 9.5 TiB 272 MiB 45 GiB 6.8 TiB 60.63 1.11 120 up osd.17 19 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 12 GiB 50 GiB 5.9 TiB 65.90 1.20 134 up osd.19 21 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.6 GiB 57 GiB 4.1 TiB 76.49 1.40 152 up osd.21 23 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 31 KiB 54 GiB 4.7 TiB 73.10 1.34 124 up osd.23 -3 208.09680 - 208 TiB 146 TiB 134 TiB 64 GiB 629 GiB 62 TiB 70.05 1.28 - host hbgt-ceph1-osd02 0 hdd 17.34140 1.00000 17 TiB 11 TiB 9.8 TiB 22 GiB 49 GiB 6.6 TiB 62.07 1.13 124 up osd.0 2 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.7 GiB 52 GiB 5.2 TiB 70.14 1.28 150 up osd.2 4 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 48 GiB 5.8 TiB 66.83 1.22 152 up osd.4 6 hdd 17.34140 0.85004 17 TiB 13 TiB 12 TiB 11 GiB 58 GiB 4.0 TiB 76.85 1.40 153 up osd.6 8 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 54 GiB 4.9 TiB 71.58 1.31 152 up osd.8 10 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 6.3 MiB 47 GiB 6.1 TiB 64.91 1.19 133 up osd.10 12 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 109 MiB 51 GiB 5.6 TiB 67.72 1.24 137 up osd.12 14 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 GiB 53 GiB 5.1 TiB 70.37 1.29 148 up osd.14 16 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 2.9 GiB 54 GiB 4.9 TiB 71.65 1.31 145 up osd.16 18 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 88 MiB 55 GiB 4.8 TiB 72.45 1.32 154 up osd.18 20 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 108 MiB 55 GiB 4.6 TiB 73.39 1.34 166 up osd.20 22 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 209 MiB 54 GiB 4.7 TiB 72.68 1.33 138 up osd.22 -7 208.09680 - 208 TiB 148 TiB 136 TiB 29 GiB 635 GiB 60 TiB 71.03 1.30 - host hbgt-ceph1-osd03 24 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.4 GiB 56 GiB 4.5 TiB 74.27 1.36 150 up osd.24 25 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 32 KiB 57 GiB 4.1 TiB 76.27 1.39 162 up osd.25 26 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 1.4 GiB 49 GiB 6.0 TiB 65.30 1.19 134 up osd.26 27 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.7 GiB 53 GiB 4.7 TiB 73.16 1.34 152 up osd.27 28 hdd 17.34140 1.00000 17 TiB 12 TiB 12 TiB 2.3 MiB 52 GiB 4.9 TiB 71.97 1.32 158 up osd.28 29 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 10 GiB 52 GiB 5.3 TiB 69.72 1.27 141 up osd.29 30 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 266 MiB 54 GiB 4.6 TiB 73.38 1.34 142 up osd.30 31 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 93 MiB 50 GiB 5.7 TiB 67.36 1.23 145 up osd.31 32 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 13 GiB 50 GiB 6.1 TiB 64.77 1.18 131 up osd.32 33 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 MiB 53 GiB 5.2 TiB 69.77 1.28 132 up osd.33 34 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 242 MiB 50 GiB 5.5 TiB 68.17 1.25 139 up osd.34 35 hdd 17.34140 1.00000 17 TiB 14 TiB 13 TiB 135 MiB 58 GiB 3.8 TiB 78.26 1.43 157 up osd.35 -9 208.09680 - 208 TiB 146 TiB 134 TiB 38 GiB 627 GiB 62 TiB 70.19 1.28 - host hbgt-ceph1-osd04 36 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 210 MiB 47 GiB 6.4 TiB 63.16 1.15 139 up osd.36 37 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 222 MiB 53 GiB 5.0 TiB 71.34 1.30 147 up osd.37 38 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 13 MiB 55 GiB 4.5 TiB 74.33 1.36 155 up osd.38 39 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 135 MiB 52 GiB 5.2 TiB 70.22 1.28 140 up osd.39 40 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 25 MiB 53 GiB 4.9 TiB 71.46 1.31 150 up osd.40 41 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 54 GiB 5.0 TiB 71.36 1.30 150 up osd.41 42 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 823 MiB 50 GiB 5.8 TiB 66.49 1.22 156 up osd.42 43 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52 GiB 5.2 TiB 69.97 1.28 160 up osd.43 44 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52 GiB 5.3 TiB 69.35 1.27 133 up osd.44 45 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 12 GiB 55 GiB 4.8 TiB 72.12 1.32 148 up osd.45 46 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52 GiB 5.1 TiB 70.74 1.29 148 up osd.46 47 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 188 MiB 54 GiB 4.9 TiB 71.74 1.31 151 up osd.47 -11 208.09680 - 208 TiB 147 TiB 135 TiB 27 GiB 627 GiB 62 TiB 70.45 1.29 - host hbgt-ceph1-osd05 48 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 49 GiB 5.7 TiB 67.11 1.23 152 up osd.48 49 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 52 GiB 5.1 TiB 70.87 1.30 155 up osd.49 50 hdd 17.34140 0.90002 17 TiB 12 TiB 11 TiB 11 GiB 54 GiB 5.2 TiB 70.01 1.28 147 up osd.50 51 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 84 MiB 54 GiB 4.6 TiB 73.24 1.34 147 up osd.51 52 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 337 MiB 53 GiB 4.8 TiB 72.26 1.32 146 up osd.52 53 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.6 GiB 54 GiB 5.1 TiB 70.56 1.29 132 up osd.53 54 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.3 GiB 56 GiB 4.5 TiB 73.77 1.35 138 up osd.54 55 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 10 GiB 53 GiB 4.8 TiB 72.59 1.33 148 up osd.55 56 hdd 17.34140 1.00000 17 TiB 11 TiB 9.6 TiB 1.8 GiB 46 GiB 6.7 TiB 61.16 1.12 127 up osd.56 57 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 3.1 MiB 50 GiB 5.6 TiB 67.96 1.24 148 up osd.57 58 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 687 KiB 52 GiB 5.0 TiB 70.93 1.30 152 up osd.58 59 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 22 KiB 55 GiB 4.4 TiB 74.88 1.37 147 up osd.59 -13 245.93866 - 246 TiB 112 TiB 106 TiB 28 GiB 498 GiB 134 TiB 45.56 0.83 - host hbgt-ceph1-osd06 60 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 10 GiB 51 GiB 9.3 TiB 54.43 0.99 134 up osd.60 61 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 216 MiB 43 GiB 11 TiB 47.06 0.86 120 up osd.61 62 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 330 MiB 41 GiB 11 TiB 44.53 0.81 116 up osd.62 63 hdd 20.49489 1.00000 20 TiB 8.9 TiB 8.4 TiB 354 MiB 39 GiB 12 TiB 43.29 0.79 113 up osd.63 64 hdd 20.49489 1.00000 20 TiB 8.1 TiB 7.7 TiB 148 MiB 36 GiB 12 TiB 39.72 0.73 117 up osd.64 65 hdd 20.49489 1.00000 20 TiB 12 TiB 12 TiB 14 KiB 53 GiB 8.4 TiB 58.90 1.08 159 up osd.65 66 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 18 MiB 49 GiB 9.8 TiB 52.33 0.96 129 up osd.66 67 hdd 20.49489 1.00000 20 TiB 498 GiB 1.5 GiB 1 KiB 745 MiB 20 TiB 2.37 0.04 6 up osd.67 68 hdd 20.49489 1.00000 20 TiB 10 TiB 9.8 TiB 13 GiB 47 GiB 10 TiB 50.12 0.92 138 up osd.68 69 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 2.0 GiB 48 GiB 9.7 TiB 52.77 0.96 135 up osd.69 70 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 1.6 GiB 50 GiB 9.2 TiB 55.24 1.01 150 up osd.70 71 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 86 MiB 42 GiB 11 TiB 45.96 0.84 130 up osd.71 -15 245.93866 - 246 TiB 114 TiB 108 TiB 62 GiB 505 GiB 132 TiB 46.18 0.84 - host hbgt-ceph1-osd07 72 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 12 GiB 49 GiB 9.6 TiB 53.39 0.98 146 up osd.72 73 hdd 20.49489 1.00000 20 TiB 8.5 TiB 8.0 TiB 194 MiB 38 GiB 12 TiB 41.31 0.75 112 up osd.73 74 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 10 GiB 42 GiB 11 TiB 46.12 0.84 124 up osd.74 75 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 12 GiB 41 GiB 11 TiB 44.16 0.81 125 up osd.75 76 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 114 MiB 41 GiB 11 TiB 45.91 0.84 111 up osd.76 77 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 1.5 GiB 42 GiB 11 TiB 47.03 0.86 127 up osd.77 78 hdd 20.49489 1.00000 20 TiB 7.7 TiB 7.3 TiB 12 GiB 36 GiB 13 TiB 37.77 0.69 105 up osd.78 79 hdd 20.49489 1.00000 20 TiB 8.2 TiB 7.7 TiB 1 KiB 36 GiB 12 TiB 39.92 0.73 91 up osd.79 80 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 371 MiB 41 GiB 11 TiB 46.13 0.84 132 up osd.80 81 hdd 20.49489 1.00000 20 TiB 9.8 TiB 9.3 TiB 353 MiB 45 GiB 11 TiB 47.95 0.88 121 up osd.81 82 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 12 GiB 51 GiB 9.2 TiB 55.23 1.01 142 up osd.82 83 hdd 20.49489 1.00000 20 TiB 10 TiB 9.6 TiB 1.9 GiB 44 GiB 10 TiB 49.28 0.90 125 up osd.83 -17 251.76105 - 252 TiB 23 TiB 11 TiB 43 MiB 64 GiB 229 TiB 8.99 0.16 - host hbgt-ceph1-osd08 84 hdd 20.98009 1.00000 21 TiB 1.0 TiB 65 GiB 1 KiB 1.4 GiB 20 TiB 4.93 0.09 5 up osd.84 85 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 813 MiB 20 TiB 4.63 0.08 7 up osd.85 86 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.0 GiB 20 TiB 4.63 0.08 2 up osd.86 87 hdd 20.98009 1.00000 21 TiB 1008 GiB 14 GiB 1 KiB 1.0 GiB 20 TiB 4.69 0.09 7 up osd.87 88 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.6 GiB 20 TiB 4.63 0.08 6 up osd.88 89 hdd 20.98009 1.00000 21 TiB 4.9 TiB 4.0 TiB 17 MiB 20 GiB 16 TiB 23.56 0.43 51 up osd.89 90 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.5 GiB 20 TiB 4.63 0.08 5 up osd.90 91 hdd 20.98009 1.00000 21 TiB 3.3 TiB 2.3 TiB 1 KiB 11 GiB 18 TiB 15.70 0.29 40 up osd.91 92 hdd 20.98009 1.00000 21 TiB 5.1 TiB 4.1 TiB 26 MiB 20 GiB 16 TiB 24.34 0.44 60 up osd.92 93 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.3 GiB 20 TiB 4.63 0.08 9 up osd.93 94 hdd 20.98009 1.00000 21 TiB 1.4 TiB 489 GiB 1 KiB 3.4 GiB 20 TiB 6.90 0.13 14 up osd.94 95 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1016 MiB 20 TiB 4.63 0.08 4 up osd.95 TOTAL 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1 TiB 808 TiB 54.72
I tried: - update backfill and recovery settings - change osd mclock profile - change pg/pgp_num
The cluster does not recovery. Clients are working well.
What can I do ?
Thanks for your help _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io