backfill_toofull not clearing on Reef
Hello, I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was replaced (same capacity). For the past week or so the cluster has been recovering, slowly, and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity. I tried setting the following without any result: ceph osd set-backfillfull-ratio 0.90 # ceph -s ... health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull 11 pgs not deep-scrubbed in time ... data: volumes: 1/1 healthy pools: 13 pools, 1665 pgs objects: 27.81M objects, 79 TiB usage: 197 TiB used, 413 TiB / 610 TiB avail pgs: 16341449/128092605 objects misplaced (12.758%) 1371 active+clean 150 active+remapped+backfill_wait 121 active+remapped+backfill_wait+backfill_toofull 16 active+remapped+backfill_toofull 6 active+clean+scrubbing+deep 1 active+remapped+backfilling io: client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr recovery: 16 MiB/s, 4 objects/s # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up 9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up 16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up 27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up 35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6 TiB 35.51 1.10 113 up 43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5 TiB 37.92 1.17 106 up 51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8 TiB 33.63 1.04 108 up 59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9 TiB 32.70 1.01 101 up 67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9 TiB 31.99 0.99 112 up 74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2 TiB 28.68 0.89 110 up 3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5 TiB 28.45 0.88 112 up 11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1 TiB 21.50 0.67 115 up 19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6 TiB 36.51 1.13 115 up 25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2 TiB 29.08 0.90 91 up 33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2 TiB 27.27 0.84 109 up 41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5 TiB 24.36 0.75 105 up 49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1 TiB 29.98 0.93 107 up 56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0 TiB 31.77 0.98 116 up 62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5 TiB 38.47 1.19 108 up 69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1 TiB 30.05 0.93 109 up 5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2 TiB 31.40 0.97 109 up 13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3 TiB 31.12 0.96 116 up 22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8 TiB 34.72 1.07 114 up 31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0 TiB 31.08 0.96 101 up 39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7 TiB 35.60 1.10 110 up 47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0 TiB 31.89 0.99 108 up 55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4 TiB 39.43 1.22 116 up 65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9 TiB 32.62 1.01 116 up 73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6 TiB 36.13 1.12 111 up 78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7 TiB 35.50 1.10 118 up 4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6 TiB 27.11 0.84 131 up 12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9 TiB 23.80 0.74 106 up 20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4 TiB 26.11 0.81 106 up 28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5 TiB 38.59 1.19 121 up 36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9 TiB 31.98 0.99 108 up 44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8 TiB 33.68 1.04 90 up 52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1 TiB 29.76 0.92 108 up 60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0 TiB 31.04 0.96 104 up 68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8 TiB 33.83 1.05 119 up 76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7 TiB 35.77 1.11 107 up 6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1 TiB 33.01 1.02 111 up 14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3 TiB 30.96 0.96 84 up 23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8 TiB 33.96 1.05 95 up 30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5 TiB 38.67 1.20 93 up 38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0 TiB 31.72 0.98 101 up 46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7 TiB 36.05 1.12 81 up 53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6 TiB 36.12 1.12 88 up 63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2 TiB 42.94 1.33 87 up 75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up 79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6 TiB 36.90 1.14 93 up 7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1 TiB 22.22 0.69 98 up 15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5 TiB 28.50 0.88 114 up 21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6 TiB 22.60 0.70 88 up 29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8 TiB 34.20 1.06 114 up 37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9 TiB 33.02 1.02 128 up 45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6 TiB 36.26 1.12 109 up 54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7 TiB 35.04 1.08 111 up 61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2 TiB 27.87 0.86 103 up 70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0 TiB 31.86 0.99 109 up 77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8 TiB 32.35 1.00 112 up 2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2 TiB 32.27 1.00 115 up 10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9 TiB 24.28 0.75 97 up 18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3 TiB 26.63 0.82 107 up 26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6 TiB 35.60 1.10 111 up 34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6 TiB 36.17 1.12 106 up 42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1 TiB 30.17 0.93 111 up 50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6 TiB 36.05 1.12 101 up 58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4 TiB 39.11 1.21 122 up 66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3 TiB 26.73 0.83 98 up 72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0 TiB 29.57 0.92 110 up 0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3 TiB 30.63 0.95 119 up 8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7 TiB 26.77 0.83 112 up 17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5 TiB 38.79 1.20 107 up 24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9 TiB 32.22 1.00 116 up 32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7 TiB 35.56 1.10 94 up 40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3 TiB 39.44 1.22 106 up 48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9 TiB 32.22 1.00 114 up 57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2 TiB 28.40 0.88 95 up 64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6 TiB 37.42 1.16 130 up 71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9 TiB 32.01 0.99 105 up TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413 TiB 32.31 MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 86.15359 root default -19 10.00000 host 01stor28 1 hdd 1.00000 osd.1 up 1.00000 1.00000 9 hdd 1.00000 osd.9 up 1.00000 1.00000 16 hdd 1.00000 osd.16 up 1.00000 1.00000 27 hdd 1.00000 osd.27 up 1.00000 1.00000 35 hdd 1.00000 osd.35 up 1.00000 1.00000 43 hdd 1.00000 osd.43 up 1.00000 1.00000 51 hdd 1.00000 osd.51 up 1.00000 1.00000 59 hdd 1.00000 osd.59 up 1.00000 1.00000 67 hdd 1.00000 osd.67 up 1.00000 1.00000 74 hdd 1.00000 osd.74 up 1.00000 1.00000 -21 10.00000 host 01stor32 3 hdd 1.00000 osd.3 up 1.00000 1.00000 11 hdd 1.00000 osd.11 up 1.00000 1.00000 19 hdd 1.00000 osd.19 up 1.00000 1.00000 25 hdd 1.00000 osd.25 up 1.00000 1.00000 33 hdd 1.00000 osd.33 up 1.00000 1.00000 41 hdd 1.00000 osd.41 up 1.00000 1.00000 49 hdd 1.00000 osd.49 up 1.00000 1.00000 56 hdd 1.00000 osd.56 up 1.00000 1.00000 62 hdd 1.00000 osd.62 up 1.00000 1.00000 69 hdd 1.00000 osd.69 up 1.00000 1.00000 -15 10.00000 host 02stor23 5 hdd 1.00000 osd.5 up 1.00000 1.00000 13 hdd 1.00000 osd.13 up 1.00000 1.00000 22 hdd 1.00000 osd.22 up 1.00000 1.00000 31 hdd 1.00000 osd.31 up 1.00000 1.00000 39 hdd 1.00000 osd.39 up 1.00000 1.00000 47 hdd 1.00000 osd.47 up 1.00000 1.00000 55 hdd 1.00000 osd.55 up 1.00000 1.00000 65 hdd 1.00000 osd.65 up 1.00000 1.00000 73 hdd 1.00000 osd.73 up 1.00000 1.00000 78 hdd 1.00000 osd.78 up 1.00000 1.00000 -11 10.00000 host 02stor27 4 hdd 1.00000 osd.4 up 1.00000 1.00000 12 hdd 1.00000 osd.12 up 1.00000 1.00000 20 hdd 1.00000 osd.20 up 1.00000 1.00000 28 hdd 1.00000 osd.28 up 1.00000 1.00000 36 hdd 1.00000 osd.36 up 1.00000 1.00000 44 hdd 1.00000 osd.44 up 1.00000 1.00000 52 hdd 1.00000 osd.52 up 1.00000 1.00000 60 hdd 1.00000 osd.60 up 1.00000 1.00000 68 hdd 1.00000 osd.68 up 1.00000 1.00000 76 hdd 1.00000 osd.76 up 1.00000 1.00000 -17 16.15359 host 03stor23 6 hdd 1.00000 osd.6 up 1.00000 1.00000 14 hdd 1.00000 osd.14 up 1.00000 1.00000 23 hdd 1.00000 osd.23 up 1.00000 1.00000 30 hdd 1.00000 osd.30 up 1.00000 1.00000 38 hdd 1.00000 osd.38 up 1.00000 1.00000 46 hdd 1.00000 osd.46 up 1.00000 1.00000 53 hdd 1.00000 osd.53 up 1.00000 1.00000 63 hdd 1.00000 osd.63 up 1.00000 1.00000 75 hdd 7.15359 osd.75 up 1.00000 1.00000 79 hdd 1.00000 osd.79 up 1.00000 1.00000 -13 10.00000 host 03stor27 7 hdd 1.00000 osd.7 up 1.00000 1.00000 15 hdd 1.00000 osd.15 up 1.00000 1.00000 21 hdd 1.00000 osd.21 up 1.00000 1.00000 29 hdd 1.00000 osd.29 up 1.00000 1.00000 37 hdd 1.00000 osd.37 up 1.00000 1.00000 45 hdd 1.00000 osd.45 up 1.00000 1.00000 54 hdd 1.00000 osd.54 up 1.00000 1.00000 61 hdd 1.00000 osd.61 up 1.00000 1.00000 70 hdd 1.00000 osd.70 up 1.00000 1.00000 77 hdd 1.00000 osd.77 up 1.00000 1.00000 -3 10.00000 host 04stor28 2 hdd 1.00000 osd.2 up 1.00000 1.00000 10 hdd 1.00000 osd.10 up 1.00000 1.00000 18 hdd 1.00000 osd.18 up 1.00000 1.00000 26 hdd 1.00000 osd.26 up 1.00000 1.00000 34 hdd 1.00000 osd.34 up 1.00000 1.00000 42 hdd 1.00000 osd.42 up 1.00000 1.00000 50 hdd 1.00000 osd.50 up 1.00000 1.00000 58 hdd 1.00000 osd.58 up 1.00000 1.00000 66 hdd 1.00000 osd.66 up 1.00000 1.00000 72 hdd 1.00000 osd.72 up 1.00000 1.00000 -7 10.00000 host 04stor32 0 hdd 1.00000 osd.0 up 1.00000 1.00000 8 hdd 1.00000 osd.8 up 1.00000 1.00000 17 hdd 1.00000 osd.17 up 1.00000 1.00000 24 hdd 1.00000 osd.24 up 1.00000 1.00000 32 hdd 1.00000 osd.32 up 1.00000 1.00000 40 hdd 1.00000 osd.40 up 1.00000 1.00000 48 hdd 1.00000 osd.48 up 1.00000 1.00000 57 hdd 1.00000 osd.57 up 1.00000 1.00000 64 hdd 1.00000 osd.64 up 1.00000 1.00000 71 hdd 1.00000 osd.71 up 1.00000 1.00000 # ceph health detail HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting [0,67,4] pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting [71,44,22] pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting [58,51,23] pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting [69,45,22] pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting [24,3,63] pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting [35,50,14] pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting [28,16,69] pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6] pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting [78,26,29] pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30] pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting [43,11,17] pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting [68,33,2] pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66] pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting [78,17,6] pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting [57,67,13] pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38] pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting [1,0,28] pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting [47,14,66] pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14] pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting [27,45,38] pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72] pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting [52,69,46] pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting [55,26,32] pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting [34,59,11] pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2] pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting [55,70,46] pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23] pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting [37,25,59] pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23] pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting [11,26,24] pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting [59,60,46] pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6] pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting [35,57,46] pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting [28,9,72] pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting [5,43,72] pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting [70,72,14] pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting [78,10,23] pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting [66,7,53] pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13] pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting [25,15,63] pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting [78,76,34] pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting [24,76,54,16,25,79,22,2] pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting [10,9,56,4,47,61,32,79] pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting [70,41,36,48,23,31,1,2] pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting [10,29,78,25,1,44,64,79] pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting [28,26,15,23,57,16,25,78] pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting [59,46,19,7,72,68,55,8] pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting [76,54,25,31,0,46,58,59] pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting [31,76,1,66,8,56,79,21] pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting [74,0,70,44,39,34,33,6] pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting [44,21,43,18,69,64,38,31] [WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000 pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000 pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000 pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000 pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000 pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000 pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000 pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000 pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000 pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000 pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000 pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000 pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000 pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000 pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000 pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000 pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000 pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000 pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000 pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000 pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000 pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000 pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000 pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000 pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000
So the one thing that sticks out straight away is OSD.75 and it having a different weight to all the other devices. Why is this? Is this the drive that was replaced? Darren
On 26 Feb 2025, at 12:47, Deep Dish <deeepdish@gmail.com> wrote:
Hello,
I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was replaced (same capacity). For the past week or so the cluster has been recovering, slowly, and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity.
I tried setting the following without any result:
ceph osd set-backfillfull-ratio 0.90
# ceph -s
...
health: HEALTH_WARN
Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull
11 pgs not deep-scrubbed in time
...
data:
volumes: 1/1 healthy
pools: 13 pools, 1665 pgs
objects: 27.81M objects, 79 TiB
usage: 197 TiB used, 413 TiB / 610 TiB avail
pgs: 16341449/128092605 objects misplaced (12.758%)
1371 active+clean
150 active+remapped+backfill_wait
121 active+remapped+backfill_wait+backfill_toofull
16 active+remapped+backfill_toofull
6 active+clean+scrubbing+deep
1 active+remapped+backfilling
io:
client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr
recovery: 16 MiB/s, 4 objects/s
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up
9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up
16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up
27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up
35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6 TiB 35.51 1.10 113 up
43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5 TiB 37.92 1.17 106 up
51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8 TiB 33.63 1.04 108 up
59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9 TiB 32.70 1.01 101 up
67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9 TiB 31.99 0.99 112 up
74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2 TiB 28.68 0.89 110 up
3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5 TiB 28.45 0.88 112 up
11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1 TiB 21.50 0.67 115 up
19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6 TiB 36.51 1.13 115 up
25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2 TiB 29.08 0.90 91 up
33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2 TiB 27.27 0.84 109 up
41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5 TiB 24.36 0.75 105 up
49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1 TiB 29.98 0.93 107 up
56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0 TiB 31.77 0.98 116 up
62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5 TiB 38.47 1.19 108 up
69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1 TiB 30.05 0.93 109 up
5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2 TiB 31.40 0.97 109 up
13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3 TiB 31.12 0.96 116 up
22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8 TiB 34.72 1.07 114 up
31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0 TiB 31.08 0.96 101 up
39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7 TiB 35.60 1.10 110 up
47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0 TiB 31.89 0.99 108 up
55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4 TiB 39.43 1.22 116 up
65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9 TiB 32.62 1.01 116 up
73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6 TiB 36.13 1.12 111 up
78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7 TiB 35.50 1.10 118 up
4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6 TiB 27.11 0.84 131 up
12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9 TiB 23.80 0.74 106 up
20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4 TiB 26.11 0.81 106 up
28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5 TiB 38.59 1.19 121 up
36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9 TiB 31.98 0.99 108 up
44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8 TiB 33.68 1.04 90 up
52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1 TiB 29.76 0.92 108 up
60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0 TiB 31.04 0.96 104 up
68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8 TiB 33.83 1.05 119 up
76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7 TiB 35.77 1.11 107 up
6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1 TiB 33.01 1.02 111 up
14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3 TiB 30.96 0.96 84 up
23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8 TiB 33.96 1.05 95 up
30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5 TiB 38.67 1.20 93 up
38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0 TiB 31.72 0.98 101 up
46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7 TiB 36.05 1.12 81 up
53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6 TiB 36.12 1.12 88 up
63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2 TiB 42.94 1.33 87 up
75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up
79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6 TiB 36.90 1.14 93 up
7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1 TiB 22.22 0.69 98 up
15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5 TiB 28.50 0.88 114 up
21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6 TiB 22.60 0.70 88 up
29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8 TiB 34.20 1.06 114 up
37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9 TiB 33.02 1.02 128 up
45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6 TiB 36.26 1.12 109 up
54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7 TiB 35.04 1.08 111 up
61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2 TiB 27.87 0.86 103 up
70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0 TiB 31.86 0.99 109 up
77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8 TiB 32.35 1.00 112 up
2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2 TiB 32.27 1.00 115 up
10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9 TiB 24.28 0.75 97 up
18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3 TiB 26.63 0.82 107 up
26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6 TiB 35.60 1.10 111 up
34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6 TiB 36.17 1.12 106 up
42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1 TiB 30.17 0.93 111 up
50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6 TiB 36.05 1.12 101 up
58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4 TiB 39.11 1.21 122 up
66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3 TiB 26.73 0.83 98 up
72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0 TiB 29.57 0.92 110 up
0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3 TiB 30.63 0.95 119 up
8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7 TiB 26.77 0.83 112 up
17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5 TiB 38.79 1.20 107 up
24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9 TiB 32.22 1.00 116 up
32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7 TiB 35.56 1.10 94 up
40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3 TiB 39.44 1.22 106 up
48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9 TiB 32.22 1.00 114 up
57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2 TiB 28.40 0.88 95 up
64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6 TiB 37.42 1.16 130 up
71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9 TiB 32.01 0.99 105 up
TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413 TiB 32.31
MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 86.15359 root default
-19 10.00000 host 01stor28
1 hdd 1.00000 osd.1 up 1.00000 1.00000
9 hdd 1.00000 osd.9 up 1.00000 1.00000
16 hdd 1.00000 osd.16 up 1.00000 1.00000
27 hdd 1.00000 osd.27 up 1.00000 1.00000
35 hdd 1.00000 osd.35 up 1.00000 1.00000
43 hdd 1.00000 osd.43 up 1.00000 1.00000
51 hdd 1.00000 osd.51 up 1.00000 1.00000
59 hdd 1.00000 osd.59 up 1.00000 1.00000
67 hdd 1.00000 osd.67 up 1.00000 1.00000
74 hdd 1.00000 osd.74 up 1.00000 1.00000
-21 10.00000 host 01stor32
3 hdd 1.00000 osd.3 up 1.00000 1.00000
11 hdd 1.00000 osd.11 up 1.00000 1.00000
19 hdd 1.00000 osd.19 up 1.00000 1.00000
25 hdd 1.00000 osd.25 up 1.00000 1.00000
33 hdd 1.00000 osd.33 up 1.00000 1.00000
41 hdd 1.00000 osd.41 up 1.00000 1.00000
49 hdd 1.00000 osd.49 up 1.00000 1.00000
56 hdd 1.00000 osd.56 up 1.00000 1.00000
62 hdd 1.00000 osd.62 up 1.00000 1.00000
69 hdd 1.00000 osd.69 up 1.00000 1.00000
-15 10.00000 host 02stor23
5 hdd 1.00000 osd.5 up 1.00000 1.00000
13 hdd 1.00000 osd.13 up 1.00000 1.00000
22 hdd 1.00000 osd.22 up 1.00000 1.00000
31 hdd 1.00000 osd.31 up 1.00000 1.00000
39 hdd 1.00000 osd.39 up 1.00000 1.00000
47 hdd 1.00000 osd.47 up 1.00000 1.00000
55 hdd 1.00000 osd.55 up 1.00000 1.00000
65 hdd 1.00000 osd.65 up 1.00000 1.00000
73 hdd 1.00000 osd.73 up 1.00000 1.00000
78 hdd 1.00000 osd.78 up 1.00000 1.00000
-11 10.00000 host 02stor27
4 hdd 1.00000 osd.4 up 1.00000 1.00000
12 hdd 1.00000 osd.12 up 1.00000 1.00000
20 hdd 1.00000 osd.20 up 1.00000 1.00000
28 hdd 1.00000 osd.28 up 1.00000 1.00000
36 hdd 1.00000 osd.36 up 1.00000 1.00000
44 hdd 1.00000 osd.44 up 1.00000 1.00000
52 hdd 1.00000 osd.52 up 1.00000 1.00000
60 hdd 1.00000 osd.60 up 1.00000 1.00000
68 hdd 1.00000 osd.68 up 1.00000 1.00000
76 hdd 1.00000 osd.76 up 1.00000 1.00000
-17 16.15359 host 03stor23
6 hdd 1.00000 osd.6 up 1.00000 1.00000
14 hdd 1.00000 osd.14 up 1.00000 1.00000
23 hdd 1.00000 osd.23 up 1.00000 1.00000
30 hdd 1.00000 osd.30 up 1.00000 1.00000
38 hdd 1.00000 osd.38 up 1.00000 1.00000
46 hdd 1.00000 osd.46 up 1.00000 1.00000
53 hdd 1.00000 osd.53 up 1.00000 1.00000
63 hdd 1.00000 osd.63 up 1.00000 1.00000
75 hdd 7.15359 osd.75 up 1.00000 1.00000
79 hdd 1.00000 osd.79 up 1.00000 1.00000
-13 10.00000 host 03stor27
7 hdd 1.00000 osd.7 up 1.00000 1.00000
15 hdd 1.00000 osd.15 up 1.00000 1.00000
21 hdd 1.00000 osd.21 up 1.00000 1.00000
29 hdd 1.00000 osd.29 up 1.00000 1.00000
37 hdd 1.00000 osd.37 up 1.00000 1.00000
45 hdd 1.00000 osd.45 up 1.00000 1.00000
54 hdd 1.00000 osd.54 up 1.00000 1.00000
61 hdd 1.00000 osd.61 up 1.00000 1.00000
70 hdd 1.00000 osd.70 up 1.00000 1.00000
77 hdd 1.00000 osd.77 up 1.00000 1.00000
-3 10.00000 host 04stor28
2 hdd 1.00000 osd.2 up 1.00000 1.00000
10 hdd 1.00000 osd.10 up 1.00000 1.00000
18 hdd 1.00000 osd.18 up 1.00000 1.00000
26 hdd 1.00000 osd.26 up 1.00000 1.00000
34 hdd 1.00000 osd.34 up 1.00000 1.00000
42 hdd 1.00000 osd.42 up 1.00000 1.00000
50 hdd 1.00000 osd.50 up 1.00000 1.00000
58 hdd 1.00000 osd.58 up 1.00000 1.00000
66 hdd 1.00000 osd.66 up 1.00000 1.00000
72 hdd 1.00000 osd.72 up 1.00000 1.00000
-7 10.00000 host 04stor32
0 hdd 1.00000 osd.0 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
17 hdd 1.00000 osd.17 up 1.00000 1.00000
24 hdd 1.00000 osd.24 up 1.00000 1.00000
32 hdd 1.00000 osd.32 up 1.00000 1.00000
40 hdd 1.00000 osd.40 up 1.00000 1.00000
48 hdd 1.00000 osd.48 up 1.00000 1.00000
57 hdd 1.00000 osd.57 up 1.00000 1.00000
64 hdd 1.00000 osd.64 up 1.00000 1.00000
71 hdd 1.00000 osd.71 up 1.00000 1.00000
# ceph health detail
HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull
pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting [0,67,4]
pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting [71,44,22]
pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting [58,51,23]
pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting [69,45,22]
pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting [24,3,63]
pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting [35,50,14]
pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting [28,16,69]
pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6]
pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting [78,26,29]
pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30]
pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting [43,11,17]
pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting [68,33,2]
pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66]
pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting [78,17,6]
pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting [57,67,13]
pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38]
pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting [1,0,28]
pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting [47,14,66]
pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14]
pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting [27,45,38]
pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72]
pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting [52,69,46]
pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting [55,26,32]
pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting [34,59,11]
pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2]
pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting [55,70,46]
pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23]
pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting [37,25,59]
pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23]
pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting [11,26,24]
pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting [59,60,46]
pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6]
pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting [35,57,46]
pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting [28,9,72]
pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting [5,43,72]
pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting [70,72,14]
pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting [78,10,23]
pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting [66,7,53]
pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13]
pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting [25,15,63]
pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting [78,76,34]
pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting [24,76,54,16,25,79,22,2]
pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting [10,9,56,4,47,61,32,79]
pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting [70,41,36,48,23,31,1,2]
pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting [10,29,78,25,1,44,64,79]
pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting [28,26,15,23,57,16,25,78]
pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting [59,46,19,7,72,68,55,8]
pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting [76,54,25,31,0,46,58,59]
pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting [31,76,1,66,8,56,79,21]
pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting [74,0,70,44,39,34,33,6]
pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting [44,21,43,18,69,64,38,31]
[WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time
pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000
pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000
pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000
pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000
pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000
pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000
pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000
pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000
pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000
pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000
pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000
pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000
pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000
pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000
pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000
pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000
pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000
pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000
pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000
pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000
pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000
pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000
pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000
pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000
pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Did you change osd.75 weight on purpose? 75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up Setting it back to 1 with 'ceph osd reweight 75 1' may help. Regards, Frédéric. ----- Le 26 Fév 25, à 13:47, Deep Dish deeepdish@gmail.com a écrit :
Hello,
I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was replaced (same capacity). For the past week or so the cluster has been recovering, slowly, and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity.
I tried setting the following without any result:
ceph osd set-backfillfull-ratio 0.90
# ceph -s
...
health: HEALTH_WARN
Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull
11 pgs not deep-scrubbed in time
...
data:
volumes: 1/1 healthy
pools: 13 pools, 1665 pgs
objects: 27.81M objects, 79 TiB
usage: 197 TiB used, 413 TiB / 610 TiB avail
pgs: 16341449/128092605 objects misplaced (12.758%)
1371 active+clean
150 active+remapped+backfill_wait
121 active+remapped+backfill_wait+backfill_toofull
16 active+remapped+backfill_toofull
6 active+clean+scrubbing+deep
1 active+remapped+backfilling
io:
client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr
recovery: 16 MiB/s, 4 objects/s
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up
9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up
16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up
27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up
35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6 TiB 35.51 1.10 113 up
43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5 TiB 37.92 1.17 106 up
51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8 TiB 33.63 1.04 108 up
59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9 TiB 32.70 1.01 101 up
67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9 TiB 31.99 0.99 112 up
74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2 TiB 28.68 0.89 110 up
3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5 TiB 28.45 0.88 112 up
11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1 TiB 21.50 0.67 115 up
19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6 TiB 36.51 1.13 115 up
25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2 TiB 29.08 0.90 91 up
33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2 TiB 27.27 0.84 109 up
41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5 TiB 24.36 0.75 105 up
49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1 TiB 29.98 0.93 107 up
56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0 TiB 31.77 0.98 116 up
62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5 TiB 38.47 1.19 108 up
69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1 TiB 30.05 0.93 109 up
5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2 TiB 31.40 0.97 109 up
13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3 TiB 31.12 0.96 116 up
22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8 TiB 34.72 1.07 114 up
31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0 TiB 31.08 0.96 101 up
39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7 TiB 35.60 1.10 110 up
47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0 TiB 31.89 0.99 108 up
55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4 TiB 39.43 1.22 116 up
65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9 TiB 32.62 1.01 116 up
73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6 TiB 36.13 1.12 111 up
78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7 TiB 35.50 1.10 118 up
4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6 TiB 27.11 0.84 131 up
12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9 TiB 23.80 0.74 106 up
20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4 TiB 26.11 0.81 106 up
28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5 TiB 38.59 1.19 121 up
36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9 TiB 31.98 0.99 108 up
44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8 TiB 33.68 1.04 90 up
52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1 TiB 29.76 0.92 108 up
60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0 TiB 31.04 0.96 104 up
68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8 TiB 33.83 1.05 119 up
76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7 TiB 35.77 1.11 107 up
6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1 TiB 33.01 1.02 111 up
14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3 TiB 30.96 0.96 84 up
23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8 TiB 33.96 1.05 95 up
30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5 TiB 38.67 1.20 93 up
38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0 TiB 31.72 0.98 101 up
46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7 TiB 36.05 1.12 81 up
53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6 TiB 36.12 1.12 88 up
63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2 TiB 42.94 1.33 87 up
75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up
79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6 TiB 36.90 1.14 93 up
7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1 TiB 22.22 0.69 98 up
15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5 TiB 28.50 0.88 114 up
21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6 TiB 22.60 0.70 88 up
29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8 TiB 34.20 1.06 114 up
37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9 TiB 33.02 1.02 128 up
45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6 TiB 36.26 1.12 109 up
54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7 TiB 35.04 1.08 111 up
61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2 TiB 27.87 0.86 103 up
70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0 TiB 31.86 0.99 109 up
77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8 TiB 32.35 1.00 112 up
2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2 TiB 32.27 1.00 115 up
10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9 TiB 24.28 0.75 97 up
18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3 TiB 26.63 0.82 107 up
26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6 TiB 35.60 1.10 111 up
34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6 TiB 36.17 1.12 106 up
42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1 TiB 30.17 0.93 111 up
50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6 TiB 36.05 1.12 101 up
58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4 TiB 39.11 1.21 122 up
66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3 TiB 26.73 0.83 98 up
72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0 TiB 29.57 0.92 110 up
0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3 TiB 30.63 0.95 119 up
8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7 TiB 26.77 0.83 112 up
17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5 TiB 38.79 1.20 107 up
24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9 TiB 32.22 1.00 116 up
32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7 TiB 35.56 1.10 94 up
40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3 TiB 39.44 1.22 106 up
48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9 TiB 32.22 1.00 114 up
57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2 TiB 28.40 0.88 95 up
64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6 TiB 37.42 1.16 130 up
71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9 TiB 32.01 0.99 105 up
TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413 TiB 32.31
MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 86.15359 root default
-19 10.00000 host 01stor28
1 hdd 1.00000 osd.1 up 1.00000 1.00000
9 hdd 1.00000 osd.9 up 1.00000 1.00000
16 hdd 1.00000 osd.16 up 1.00000 1.00000
27 hdd 1.00000 osd.27 up 1.00000 1.00000
35 hdd 1.00000 osd.35 up 1.00000 1.00000
43 hdd 1.00000 osd.43 up 1.00000 1.00000
51 hdd 1.00000 osd.51 up 1.00000 1.00000
59 hdd 1.00000 osd.59 up 1.00000 1.00000
67 hdd 1.00000 osd.67 up 1.00000 1.00000
74 hdd 1.00000 osd.74 up 1.00000 1.00000
-21 10.00000 host 01stor32
3 hdd 1.00000 osd.3 up 1.00000 1.00000
11 hdd 1.00000 osd.11 up 1.00000 1.00000
19 hdd 1.00000 osd.19 up 1.00000 1.00000
25 hdd 1.00000 osd.25 up 1.00000 1.00000
33 hdd 1.00000 osd.33 up 1.00000 1.00000
41 hdd 1.00000 osd.41 up 1.00000 1.00000
49 hdd 1.00000 osd.49 up 1.00000 1.00000
56 hdd 1.00000 osd.56 up 1.00000 1.00000
62 hdd 1.00000 osd.62 up 1.00000 1.00000
69 hdd 1.00000 osd.69 up 1.00000 1.00000
-15 10.00000 host 02stor23
5 hdd 1.00000 osd.5 up 1.00000 1.00000
13 hdd 1.00000 osd.13 up 1.00000 1.00000
22 hdd 1.00000 osd.22 up 1.00000 1.00000
31 hdd 1.00000 osd.31 up 1.00000 1.00000
39 hdd 1.00000 osd.39 up 1.00000 1.00000
47 hdd 1.00000 osd.47 up 1.00000 1.00000
55 hdd 1.00000 osd.55 up 1.00000 1.00000
65 hdd 1.00000 osd.65 up 1.00000 1.00000
73 hdd 1.00000 osd.73 up 1.00000 1.00000
78 hdd 1.00000 osd.78 up 1.00000 1.00000
-11 10.00000 host 02stor27
4 hdd 1.00000 osd.4 up 1.00000 1.00000
12 hdd 1.00000 osd.12 up 1.00000 1.00000
20 hdd 1.00000 osd.20 up 1.00000 1.00000
28 hdd 1.00000 osd.28 up 1.00000 1.00000
36 hdd 1.00000 osd.36 up 1.00000 1.00000
44 hdd 1.00000 osd.44 up 1.00000 1.00000
52 hdd 1.00000 osd.52 up 1.00000 1.00000
60 hdd 1.00000 osd.60 up 1.00000 1.00000
68 hdd 1.00000 osd.68 up 1.00000 1.00000
76 hdd 1.00000 osd.76 up 1.00000 1.00000
-17 16.15359 host 03stor23
6 hdd 1.00000 osd.6 up 1.00000 1.00000
14 hdd 1.00000 osd.14 up 1.00000 1.00000
23 hdd 1.00000 osd.23 up 1.00000 1.00000
30 hdd 1.00000 osd.30 up 1.00000 1.00000
38 hdd 1.00000 osd.38 up 1.00000 1.00000
46 hdd 1.00000 osd.46 up 1.00000 1.00000
53 hdd 1.00000 osd.53 up 1.00000 1.00000
63 hdd 1.00000 osd.63 up 1.00000 1.00000
75 hdd 7.15359 osd.75 up 1.00000 1.00000
79 hdd 1.00000 osd.79 up 1.00000 1.00000
-13 10.00000 host 03stor27
7 hdd 1.00000 osd.7 up 1.00000 1.00000
15 hdd 1.00000 osd.15 up 1.00000 1.00000
21 hdd 1.00000 osd.21 up 1.00000 1.00000
29 hdd 1.00000 osd.29 up 1.00000 1.00000
37 hdd 1.00000 osd.37 up 1.00000 1.00000
45 hdd 1.00000 osd.45 up 1.00000 1.00000
54 hdd 1.00000 osd.54 up 1.00000 1.00000
61 hdd 1.00000 osd.61 up 1.00000 1.00000
70 hdd 1.00000 osd.70 up 1.00000 1.00000
77 hdd 1.00000 osd.77 up 1.00000 1.00000
-3 10.00000 host 04stor28
2 hdd 1.00000 osd.2 up 1.00000 1.00000
10 hdd 1.00000 osd.10 up 1.00000 1.00000
18 hdd 1.00000 osd.18 up 1.00000 1.00000
26 hdd 1.00000 osd.26 up 1.00000 1.00000
34 hdd 1.00000 osd.34 up 1.00000 1.00000
42 hdd 1.00000 osd.42 up 1.00000 1.00000
50 hdd 1.00000 osd.50 up 1.00000 1.00000
58 hdd 1.00000 osd.58 up 1.00000 1.00000
66 hdd 1.00000 osd.66 up 1.00000 1.00000
72 hdd 1.00000 osd.72 up 1.00000 1.00000
-7 10.00000 host 04stor32
0 hdd 1.00000 osd.0 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
17 hdd 1.00000 osd.17 up 1.00000 1.00000
24 hdd 1.00000 osd.24 up 1.00000 1.00000
32 hdd 1.00000 osd.32 up 1.00000 1.00000
40 hdd 1.00000 osd.40 up 1.00000 1.00000
48 hdd 1.00000 osd.48 up 1.00000 1.00000
57 hdd 1.00000 osd.57 up 1.00000 1.00000
64 hdd 1.00000 osd.64 up 1.00000 1.00000
71 hdd 1.00000 osd.71 up 1.00000 1.00000
# ceph health detail
HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull
pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting [0,67,4]
pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting [71,44,22]
pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting [58,51,23]
pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting [69,45,22]
pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting [24,3,63]
pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting [35,50,14]
pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting [28,16,69]
pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6]
pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting [78,26,29]
pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30]
pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting [43,11,17]
pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting [68,33,2]
pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66]
pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting [78,17,6]
pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting [57,67,13]
pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38]
pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting [1,0,28]
pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting [47,14,66]
pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14]
pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting [27,45,38]
pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72]
pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting [52,69,46]
pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting [55,26,32]
pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting [34,59,11]
pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2]
pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting [55,70,46]
pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23]
pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting [37,25,59]
pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23]
pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting [11,26,24]
pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting [59,60,46]
pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6]
pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting [35,57,46]
pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting [28,9,72]
pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting [5,43,72]
pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting [70,72,14]
pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting [78,10,23]
pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting [66,7,53]
pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13]
pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting [25,15,63]
pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting [78,76,34]
pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting [24,76,54,16,25,79,22,2]
pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting [10,9,56,4,47,61,32,79]
pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting [70,41,36,48,23,31,1,2]
pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting [10,29,78,25,1,44,64,79]
pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting [28,26,15,23,57,16,25,78]
pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting [59,46,19,7,72,68,55,8]
pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting [76,54,25,31,0,46,58,59]
pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting [31,76,1,66,8,56,79,21]
pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting [74,0,70,44,39,34,33,6]
pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting [44,21,43,18,69,64,38,31]
[WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time
pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000
pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000
pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000
pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000
pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000
pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000
pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000
pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000
pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000
pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000
pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000
pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000
pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000
pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000
pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000
pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000
pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000
pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000
pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000
pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000
pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000
pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000
pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000
pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000
pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, after one PG finish backfilling another PG will start to backfill. You can rise osd_max_backfills if you want to backfill more at the same time. backfill_toofull will decrease in time. Why you see toofull ? Ceph remove old data only then new are in place. While it haven`t done he calculate at full space requirement to move the data. ----- Original Message ----- From: Deep Dish <deeepdish@gmail.com> To: ceph-users@ceph.io Date: Wednesday, February 26, 2025, 2:47:20 PM Subject: [ceph-users] backfill_toofull not clearing on Reef
Hello,
I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was replaced (same capacity). For the past week or so the cluster has been recovering, slowly, and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity.
I tried setting the following without any result:
ceph osd set-backfillfull-ratio 0.90
# ceph -s
...
health: HEALTH_WARN
Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull
11 pgs not deep-scrubbed in time
...
data:
volumes: 1/1 healthy
pools: 13 pools, 1665 pgs
objects: 27.81M objects, 79 TiB
usage: 197 TiB used, 413 TiB / 610 TiB avail
pgs: 16341449/128092605 objects misplaced (12.758%)
1371 active+clean
150 active+remapped+backfill_wait
121 active+remapped+backfill_wait+backfill_toofull
16 active+remapped+backfill_toofull
6 active+clean+scrubbing+deep
1 active+remapped+backfilling
io:
client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr
recovery: 16 MiB/s, 4 objects/s
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up
9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up
16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up
27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up
35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6 TiB 35.51 1.10 113 up
43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5 TiB 37.92 1.17 106 up
51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8 TiB 33.63 1.04 108 up
59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9 TiB 32.70 1.01 101 up
67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9 TiB 31.99 0.99 112 up
74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2 TiB 28.68 0.89 110 up
3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5 TiB 28.45 0.88 112 up
11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1 TiB 21.50 0.67 115 up
19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6 TiB 36.51 1.13 115 up
25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2 TiB 29.08 0.90 91 up
33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2 TiB 27.27 0.84 109 up
41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5 TiB 24.36 0.75 105 up
49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1 TiB 29.98 0.93 107 up
56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0 TiB 31.77 0.98 116 up
62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5 TiB 38.47 1.19 108 up
69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1 TiB 30.05 0.93 109 up
5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2 TiB 31.40 0.97 109 up
13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3 TiB 31.12 0.96 116 up
22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8 TiB 34.72 1.07 114 up
31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0 TiB 31.08 0.96 101 up
39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7 TiB 35.60 1.10 110 up
47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0 TiB 31.89 0.99 108 up
55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4 TiB 39.43 1.22 116 up
65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9 TiB 32.62 1.01 116 up
73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6 TiB 36.13 1.12 111 up
78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7 TiB 35.50 1.10 118 up
4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6 TiB 27.11 0.84 131 up
12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9 TiB 23.80 0.74 106 up
20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4 TiB 26.11 0.81 106 up
28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5 TiB 38.59 1.19 121 up
36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9 TiB 31.98 0.99 108 up
44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8 TiB 33.68 1.04 90 up
52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1 TiB 29.76 0.92 108 up
60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0 TiB 31.04 0.96 104 up
68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8 TiB 33.83 1.05 119 up
76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7 TiB 35.77 1.11 107 up
6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1 TiB 33.01 1.02 111 up
14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3 TiB 30.96 0.96 84 up
23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8 TiB 33.96 1.05 95 up
30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5 TiB 38.67 1.20 93 up
38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0 TiB 31.72 0.98 101 up
46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7 TiB 36.05 1.12 81 up
53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6 TiB 36.12 1.12 88 up
63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2 TiB 42.94 1.33 87 up
75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up
79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6 TiB 36.90 1.14 93 up
7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1 TiB 22.22 0.69 98 up
15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5 TiB 28.50 0.88 114 up
21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6 TiB 22.60 0.70 88 up
29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8 TiB 34.20 1.06 114 up
37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9 TiB 33.02 1.02 128 up
45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6 TiB 36.26 1.12 109 up
54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7 TiB 35.04 1.08 111 up
61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2 TiB 27.87 0.86 103 up
70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0 TiB 31.86 0.99 109 up
77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8 TiB 32.35 1.00 112 up
2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2 TiB 32.27 1.00 115 up
10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9 TiB 24.28 0.75 97 up
18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3 TiB 26.63 0.82 107 up
26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6 TiB 35.60 1.10 111 up
34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6 TiB 36.17 1.12 106 up
42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1 TiB 30.17 0.93 111 up
50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6 TiB 36.05 1.12 101 up
58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4 TiB 39.11 1.21 122 up
66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3 TiB 26.73 0.83 98 up
72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0 TiB 29.57 0.92 110 up
0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3 TiB 30.63 0.95 119 up
8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7 TiB 26.77 0.83 112 up
17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5 TiB 38.79 1.20 107 up
24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9 TiB 32.22 1.00 116 up
32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7 TiB 35.56 1.10 94 up
40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3 TiB 39.44 1.22 106 up
48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9 TiB 32.22 1.00 114 up
57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2 TiB 28.40 0.88 95 up
64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6 TiB 37.42 1.16 130 up
71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9 TiB 32.01 0.99 105 up
TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413 TiB 32.31
MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 86.15359 root default
-19 10.00000 host 01stor28
1 hdd 1.00000 osd.1 up 1.00000 1.00000
9 hdd 1.00000 osd.9 up 1.00000 1.00000
16 hdd 1.00000 osd.16 up 1.00000 1.00000
27 hdd 1.00000 osd.27 up 1.00000 1.00000
35 hdd 1.00000 osd.35 up 1.00000 1.00000
43 hdd 1.00000 osd.43 up 1.00000 1.00000
51 hdd 1.00000 osd.51 up 1.00000 1.00000
59 hdd 1.00000 osd.59 up 1.00000 1.00000
67 hdd 1.00000 osd.67 up 1.00000 1.00000
74 hdd 1.00000 osd.74 up 1.00000 1.00000
-21 10.00000 host 01stor32
3 hdd 1.00000 osd.3 up 1.00000 1.00000
11 hdd 1.00000 osd.11 up 1.00000 1.00000
19 hdd 1.00000 osd.19 up 1.00000 1.00000
25 hdd 1.00000 osd.25 up 1.00000 1.00000
33 hdd 1.00000 osd.33 up 1.00000 1.00000
41 hdd 1.00000 osd.41 up 1.00000 1.00000
49 hdd 1.00000 osd.49 up 1.00000 1.00000
56 hdd 1.00000 osd.56 up 1.00000 1.00000
62 hdd 1.00000 osd.62 up 1.00000 1.00000
69 hdd 1.00000 osd.69 up 1.00000 1.00000
-15 10.00000 host 02stor23
5 hdd 1.00000 osd.5 up 1.00000 1.00000
13 hdd 1.00000 osd.13 up 1.00000 1.00000
22 hdd 1.00000 osd.22 up 1.00000 1.00000
31 hdd 1.00000 osd.31 up 1.00000 1.00000
39 hdd 1.00000 osd.39 up 1.00000 1.00000
47 hdd 1.00000 osd.47 up 1.00000 1.00000
55 hdd 1.00000 osd.55 up 1.00000 1.00000
65 hdd 1.00000 osd.65 up 1.00000 1.00000
73 hdd 1.00000 osd.73 up 1.00000 1.00000
78 hdd 1.00000 osd.78 up 1.00000 1.00000
-11 10.00000 host 02stor27
4 hdd 1.00000 osd.4 up 1.00000 1.00000
12 hdd 1.00000 osd.12 up 1.00000 1.00000
20 hdd 1.00000 osd.20 up 1.00000 1.00000
28 hdd 1.00000 osd.28 up 1.00000 1.00000
36 hdd 1.00000 osd.36 up 1.00000 1.00000
44 hdd 1.00000 osd.44 up 1.00000 1.00000
52 hdd 1.00000 osd.52 up 1.00000 1.00000
60 hdd 1.00000 osd.60 up 1.00000 1.00000
68 hdd 1.00000 osd.68 up 1.00000 1.00000
76 hdd 1.00000 osd.76 up 1.00000 1.00000
-17 16.15359 host 03stor23
6 hdd 1.00000 osd.6 up 1.00000 1.00000
14 hdd 1.00000 osd.14 up 1.00000 1.00000
23 hdd 1.00000 osd.23 up 1.00000 1.00000
30 hdd 1.00000 osd.30 up 1.00000 1.00000
38 hdd 1.00000 osd.38 up 1.00000 1.00000
46 hdd 1.00000 osd.46 up 1.00000 1.00000
53 hdd 1.00000 osd.53 up 1.00000 1.00000
63 hdd 1.00000 osd.63 up 1.00000 1.00000
75 hdd 7.15359 osd.75 up 1.00000 1.00000
79 hdd 1.00000 osd.79 up 1.00000 1.00000
-13 10.00000 host 03stor27
7 hdd 1.00000 osd.7 up 1.00000 1.00000
15 hdd 1.00000 osd.15 up 1.00000 1.00000
21 hdd 1.00000 osd.21 up 1.00000 1.00000
29 hdd 1.00000 osd.29 up 1.00000 1.00000
37 hdd 1.00000 osd.37 up 1.00000 1.00000
45 hdd 1.00000 osd.45 up 1.00000 1.00000
54 hdd 1.00000 osd.54 up 1.00000 1.00000
61 hdd 1.00000 osd.61 up 1.00000 1.00000
70 hdd 1.00000 osd.70 up 1.00000 1.00000
77 hdd 1.00000 osd.77 up 1.00000 1.00000
-3 10.00000 host 04stor28
2 hdd 1.00000 osd.2 up 1.00000 1.00000
10 hdd 1.00000 osd.10 up 1.00000 1.00000
18 hdd 1.00000 osd.18 up 1.00000 1.00000
26 hdd 1.00000 osd.26 up 1.00000 1.00000
34 hdd 1.00000 osd.34 up 1.00000 1.00000
42 hdd 1.00000 osd.42 up 1.00000 1.00000
50 hdd 1.00000 osd.50 up 1.00000 1.00000
58 hdd 1.00000 osd.58 up 1.00000 1.00000
66 hdd 1.00000 osd.66 up 1.00000 1.00000
72 hdd 1.00000 osd.72 up 1.00000 1.00000
-7 10.00000 host 04stor32
0 hdd 1.00000 osd.0 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
17 hdd 1.00000 osd.17 up 1.00000 1.00000
24 hdd 1.00000 osd.24 up 1.00000 1.00000
32 hdd 1.00000 osd.32 up 1.00000 1.00000
40 hdd 1.00000 osd.40 up 1.00000 1.00000
48 hdd 1.00000 osd.48 up 1.00000 1.00000
57 hdd 1.00000 osd.57 up 1.00000 1.00000
64 hdd 1.00000 osd.64 up 1.00000 1.00000
71 hdd 1.00000 osd.71 up 1.00000 1.00000
# ceph health detail
HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull
pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting [0,67,4]
pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting [71,44,22]
pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting [58,51,23]
pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting [69,45,22]
pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting [24,3,63]
pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting [35,50,14]
pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting [28,16,69]
pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6]
pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting [78,26,29]
pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30]
pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting [43,11,17]
pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting [68,33,2]
pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66]
pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting [78,17,6]
pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting [57,67,13]
pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38]
pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting [1,0,28]
pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting [47,14,66]
pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14]
pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting [27,45,38]
pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72]
pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting [52,69,46]
pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting [55,26,32]
pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting [34,59,11]
pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2]
pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting [55,70,46]
pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23]
pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting [37,25,59]
pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23]
pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting [11,26,24]
pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting [59,60,46]
pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6]
pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting [35,57,46]
pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting [28,9,72]
pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting [5,43,72]
pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting [70,72,14]
pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting [78,10,23]
pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting [66,7,53]
pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13]
pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting [25,15,63]
pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting [78,76,34]
pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting [24,76,54,16,25,79,22,2]
pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting [10,9,56,4,47,61,32,79]
pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting [70,41,36,48,23,31,1,2]
pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting [10,29,78,25,1,44,64,79]
pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting [28,26,15,23,57,16,25,78]
pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting [59,46,19,7,72,68,55,8]
pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting [76,54,25,31,0,46,58,59]
pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting [31,76,1,66,8,56,79,21]
pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting [74,0,70,44,39,34,33,6]
pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting [44,21,43,18,69,64,38,31]
[WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time
pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000
pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000
pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000
pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000
pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000
pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000
pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000
pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000
pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000
pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000
pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000
pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000
pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000
pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000
pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000
pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000
pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000
pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000
pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000
pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000
pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000
pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000
pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000
pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000
pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Feb 26, 2025, at 7:47 AM, Deep Dish <deeepdish@gmail.com> wrote: Your parents had quite the sense of humor.
Hello,
I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%.
Average isn’t what factors in here ...
Recently the cluster had a bad drive, and it was replaced (same capacity).
1TB HDDs? How old is this gear? Oh, looks like your CRUSH weights don’t align with OSD TBs. Tricky. I suspect your drives are …. 8TB?
So the one thing that sticks out straight away is OSD.75 and it having a different weight to all the other devices.
That sure doesn’t help. I suspect that for some reason the CRUSH weights of all OSDs in the cluster were set to 1.0000 in the past. Which in and of itself is … okay, as operationally CRUSH weights are *relative* to each other. The replaced drive wasn’t brought up with that custom CRUSH weight, so it has the default TiB CRUSH weight. As Frédéric suggests, do this NOW: ceph osd crush reweight osd.75 1.0000 This will back off your immediate problem.
ceph osd reweight 75 1
Without `crush` in there this would actually be a no-op ;) You could set osd_crush_initial_weight = 1.0 to force all new OSDs to have that 1.000 CRUSH weight, but that would bite you if you do legitimately add larger drives down the road. I suggest reweighting all of your drives to 7.15359 at the same time by decompiling and editing the CRUSH map to avoid future problems. Look at `dmesg` / `/var/log/messages` on each host, `smartctl -a` for each drive
For the past week or so the cluster has been recovering, slowly,
Look at `dmesg` / `/var/log/messages` on each host, `smartctl -a` for each drive, and `storcli64 /c0 show termlog`. See if there are any indications of one or more bad drives: lots of reallocated sectors, SATA downshifts, etc.
and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity.
Capacity and available capacity are different. Are you using EC? As wide as 8+2?
usage: 197 TiB used, 413 TiB / 610 TiB avail
recovery: 16 MiB/s, 4 objects/s
Small clusters recover more slowly, but that’s pretty slow for an 80 OSD cluster. Is this Reef or Squid with mclock?
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
Please set your MUA to not wrap
1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up
9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up
16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up
27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up 75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up ... TiB 32.01 0.99 105 up
TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413
TiB 32.31
MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72
You don’t have a balancer enabled, or it isn’t working. Your available space is a function not only of the *full ratios but of your replication strategies and is relative to the *most full* OSD. Send `ceph osd crush rule dump` and `ceph balancer status` and `ceph -v`
I appreciate all the tips! And thanks for the observation on weights. I don't know how it got to 1 for all OSDs. The custer has a mixture of 8 and 10T drives. Is there a way to automatically readjust them or this is done manually in the crush map (decompile/edit/compile)? I ran ceph osd crush reweight 75 1.0 and it started recovering right away 3-4 Gbit/s sustained throughput. I know this is a bandaid, waiting on your guidance on how to adjust the wrights above. Here is the requested additional output: # ceph -v ceph version 18.2.4 (..) reef (stable) NB: Once the cluster is stable and OK status, I plan to upgrade to 19.2.0 via ceph orch. # ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "fs01_data-ec", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "central.rgw.buckets.data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] # ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000350", "last_optimize_started": "Wed Feb 26 14:01:03 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Some objects (0.003469) are degraded; try again later", "plans": [] } On Wed, Feb 26, 2025 at 8:18 AM Anthony D'Atri <aad@dreamsnake.net> wrote:
On Feb 26, 2025, at 7:47 AM, Deep Dish <deeepdish@gmail.com> wrote:
Your parents had quite the sense of humor.
Hello,
I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%.
Average isn’t what factors in here ...
Recently the cluster had a bad drive, and it was replaced (same capacity).
1TB HDDs? How old is this gear? Oh, looks like your CRUSH weights don’t align with OSD TBs. Tricky. I suspect your drives are …. 8TB?
So the one thing that sticks out straight away is OSD.75 and it having a different weight to all the other devices.
That sure doesn’t help. I suspect that for some reason the CRUSH weights of all OSDs in the cluster were set to 1.0000 in the past. Which in and of itself is … okay, as operationally CRUSH weights are *relative* to each other. The replaced drive wasn’t brought up with that custom CRUSH weight, so it has the default TiB CRUSH weight.
As Frédéric suggests, do this NOW:
ceph osd crush reweight osd.75 1.0000
This will back off your immediate problem.
ceph osd reweight 75 1
Without `crush` in there this would actually be a no-op ;)
You could set osd_crush_initial_weight = 1.0 to force all new OSDs to have that 1.000 CRUSH weight, but that would bite you if you do legitimately add larger drives down the road.
I suggest reweighting all of your drives to 7.15359 at the same time by decompiling and editing the CRUSH map to avoid future problems.
Look at `dmesg` / `/var/log/messages` on each host, `smartctl -a` for each drive
For the past week or so the cluster has been recovering, slowly,
Look at `dmesg` / `/var/log/messages` on each host, `smartctl -a` for each drive, and `storcli64 /c0 show termlog`.
See if there are any indications of one or more bad drives: lots of reallocated sectors, SATA downshifts, etc.
and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity.
Capacity and available capacity are different.
Are you using EC? As wide as 8+2?
usage: 197 TiB used, 413 TiB / 610 TiB avail
recovery: 16 MiB/s, 4 objects/s
Small clusters recover more slowly, but that’s pretty slow for an 80 OSD cluster. Is this Reef or Squid with mclock?
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
Please set your MUA to not wrap
1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up
9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up
16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up
27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up
75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up
... TiB 32.01 0.99 105 up
TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413
TiB 32.31
MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72
You don’t have a balancer enabled, or it isn’t working. Your available space is a function not only of the *full ratios but of your replication strategies and is relative to the *most full* OSD.
Send `ceph osd crush rule dump` and `ceph balancer status` and `ceph -v`
On Feb 26, 2025, at 9:07 AM, Deep Dish <deeepdish@gmail.com> wrote:
I appreciate all the tips! And thanks for the observation on weights. I don't know how it got to 1 for all OSDs. The custer has a mixture of 8 and 10T drives. Is there a way to automatically readjust them or this is done manually in the crush map (decompile/edit/compile)?
You can do them individually with ceph osd crush reweight I didn’t suggest that mainly because when you have any OSDs that are currently marginal wrt fullness, issuing 80 of those commands serially is kind of a pain, and even when they’re run sequentially that’s 80 incremental changes, I don’t know that the cluster batches when it sends out map updates, maybe it does. But when any OSDs are close to full, increasing their CRUSH weights before the others *might* result in them getting too much data and going full. As other OSDs are reweighed, the PG mappings will change over and over. I would if you prefer put all the commands in a script and run the script so they’re executed in quick succession. Otherwise if you tell the cluster that some OSDs are 8x the size of others, you get what you have now. Which is why there’s a backfillfull guardrail. Your 8T drives should get 7.15359, assuming they’re all the same SKU, even if they are different SKUs they’re probably close in size. Note that Ceph is speaking TiB here while drive manufacturers rate in base-2 TB so they can claim a higher number. Weasels! Your 10T drives .. probably a value like 9.09495. If you want to be exact I would suggest setting them to that value, waiting for the dust to settle, then undeploying and redeploying one of them, it’ll come back with the exact CRUSH weight, which you can then retrofit to the others. I suspect that it won’t vary by more than 0.5 + / - 9.09495.
I ran ceph osd crush reweight 75 1.0 and it started recovering right away 3-4 Gbit/s sustained throughput. I know this is a bandaid, waiting on your guidance on how to adjust the wrights above.
Here is the requested additional output:
# ceph -v ceph version 18.2.4 (..) reef (stable)
NB: Once the cluster is stable and OK status, I plan to upgrade to 19.2.0 via ceph orch.
# ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "fs01_data-ec", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "central.rgw.buckets.data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000350", "last_optimize_started": "Wed Feb 26 14:01:03 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Some objects (0.003469) are degraded; try again later", "plans": [] }
On Wed, Feb 26, 2025 at 8:18 AM Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> wrote:
On Feb 26, 2025, at 7:47 AM, Deep Dish <deeepdish@gmail.com <mailto:deeepdish@gmail.com>> wrote:
Your parents had quite the sense of humor.
Hello,
I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%.
Average isn’t what factors in here ...
Recently the cluster had a bad drive, and it was replaced (same capacity).
1TB HDDs? How old is this gear? Oh, looks like your CRUSH weights don’t align with OSD TBs. Tricky. I suspect your drives are …. 8TB?
So the one thing that sticks out straight away is OSD.75 and it having a different weight to all the other devices.
That sure doesn’t help. I suspect that for some reason the CRUSH weights of all OSDs in the cluster were set to 1.0000 in the past. Which in and of itself is … okay, as operationally CRUSH weights are *relative* to each other. The replaced drive wasn’t brought up with that custom CRUSH weight, so it has the default TiB CRUSH weight.
As Frédéric suggests, do this NOW:
ceph osd crush reweight osd.75 1.0000
This will back off your immediate problem.
ceph osd reweight 75 1
Without `crush` in there this would actually be a no-op ;)
You could set osd_crush_initial_weight = 1.0 to force all new OSDs to have that 1.000 CRUSH weight, but that would bite you if you do legitimately add larger drives down the road.
I suggest reweighting all of your drives to 7.15359 at the same time by decompiling and editing the CRUSH map to avoid future problems.
Look at `dmesg` / `/var/log/messages` on each host, `smartctl -a` for each drive
For the past week or so the cluster has been recovering, slowly,
Look at `dmesg` / `/var/log/messages` on each host, `smartctl -a` for each drive, and `storcli64 /c0 show termlog`.
See if there are any indications of one or more bad drives: lots of reallocated sectors, SATA downshifts, etc.
and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity.
Capacity and available capacity are different.
Are you using EC? As wide as 8+2?
usage: 197 TiB used, 413 TiB / 610 TiB avail
recovery: 16 MiB/s, 4 objects/s
Small clusters recover more slowly, but that’s pretty slow for an 80 OSD cluster. Is this Reef or Squid with mclock?
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
Please set your MUA to not wrap
1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up
9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up
16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up
27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up 75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up ... TiB 32.01 0.99 105 up
TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413
TiB 32.31
MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72
You don’t have a balancer enabled, or it isn’t working. Your available space is a function not only of the *full ratios but of your replication strategies and is relative to the *most full* OSD.
Send `ceph osd crush rule dump` and `ceph balancer status` and `ceph -v`
On Feb 26, 2025, at 9:07 AM, Deep Dish <deeepdish@gmail.com> wrote:
I appreciate all the tips! And thanks for the observation on weights. I don't know how it got to 1 for all OSDs. The custer has a mixture of 8 and 10T drives. Is there a way to automatically readjust them or this is done manually in the crush map (decompile/edit/compile)?
I ran ceph osd crush reweight 75 1.0 and it started recovering right away 3-4 Gbit/s sustained throughput. I know this is a bandaid, waiting on your guidance on how to adjust the wrights above.
Here is the requested additional output:
# ceph -v
ceph version 18.2.4 (..) reef (stable)
NB: Once the cluster is stable and OK status, I plan to upgrade to 19.2.0 via ceph orch.
19.2.1 is the latest, if you want to go to Squid, go to 19.2.1, or wait for 19.2.2.
# ceph osd crush rule dump
Do you have any pools using rule 0? Note that it does not specify a device class, but the others do. Most likely you do have multiple pools using rule 0, and that’s confusing the balancer. I would suggest recompiling the CRUSH map and adding ~hdd to `item_name` for rule 0 OR Create a new replicated CRUSH rule that specifies the hdd device class and change your pools to use that instead of rule 0 I suspect that would unblock your balancer.
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"type": 1,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "fs01_data-ec",
"type": 3,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -2,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "central.rgw.buckets.data",
"type": 3,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -2,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
# ceph balancer status
{
"active": true,
"last_optimize_duration": "0:00:00.000350",
"last_optimize_started": "Wed Feb 26 14:01:03 2025",
"mode": "upmap",
"no_optimization_needed": true,
"optimize_result": "Some objects (0.003469) are degraded; try again later",
"plans": []
}
On Wed, Feb 26, 2025 at 8:18 AM Anthony D'Atri <aad@dreamsnake.net> wrote:
On Feb 26, 2025, at 7:47 AM, Deep Dish <deeepdish@gmail.com> wrote:
Your parents had quite the sense of humor.
Hello,
I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%.
Average isn’t what factors in here ...
Recently the cluster had a bad drive, and it was replaced (same capacity).
1TB HDDs? How old is this gear? Oh, looks like your CRUSH weights don’t align with OSD TBs. Tricky. I suspect your drives are …. 8TB?
So the one thing that sticks out straight away is OSD.75 and it having a different weight to all the other devices.
That sure doesn’t help. I suspect that for some reason the CRUSH weights of all OSDs in the cluster were set to 1.0000 in the past. Which in and of itself is … okay, as operationally CRUSH weights are *relative* to each other. The replaced drive wasn’t brought up with that custom CRUSH weight, so it has the default TiB CRUSH weight.
As Frédéric suggests, do this NOW:
ceph osd crush reweight osd.75 1.0000
This will back off your immediate problem.
ceph osd reweight 75 1
Without `crush` in there this would actually be a no-op ;)
You could set osd_crush_initial_weight = 1.0 to force all new OSDs to have that 1.000 CRUSH weight, but that would bite you if you do legitimately add larger drives down the road.
I suggest reweighting all of your drives to 7.15359 at the same time by decompiling and editing the CRUSH map to avoid future problems.
Look at `dmesg` / `/var/log/messages` on each host, `smartctl -a` for each drive
For the past week or so the cluster has been recovering, slowly,
Look at `dmesg` / `/var/log/messages` on each host, `smartctl -a` for each drive, and `storcli64 /c0 show termlog`.
See if there are any indications of one or more bad drives: lots of reallocated sectors, SATA downshifts, etc.
and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity.
Capacity and available capacity are different.
Are you using EC? As wide as 8+2?
usage: 197 TiB used, 413 TiB / 610 TiB avail
recovery: 16 MiB/s, 4 objects/s
Small clusters recover more slowly, but that’s pretty slow for an 80 OSD cluster. Is this Reef or Squid with mclock?
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
Please set your MUA to not wrap
1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up
9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up
16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up
27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up
75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up
... TiB 32.01 0.99 105 up
TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413
TiB 32.31
MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72
You don’t have a balancer enabled, or it isn’t working. Your available space is a function not only of the *full ratios but of your replication strategies and is relative to the *most full* OSD.
Send `ceph osd crush rule dump` and `ceph balancer status` and `ceph -v`
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 2/26/25 um 15:07 schrieb Deep Dish:
I ran ceph osd crush reweight 75 1.0 and it started recovering right away 3-4 Gbit/s sustained throughput. I know this is a bandaid, waiting on your guidance on how to adjust the wrights above.
Use something like this: ceph osd set norebalance ceph osd set nobackfill ceph osd df | awk '/9.1 TiB/ { print $1; }' | while read osdid ; do ceph osd crush reweight osd.$osdid 9.1; done ceph osd df | awk '/7.3 TiB/ { print $1; }' | while read osdid ; do ceph osd crush reweight osd.$osdid 7.3; done ceph osd df | awk '/7.2 TiB/ { print $1; }' | while read osdid ; do ceph osd crush reweight osd.$osdid 7.2; done ceph osd unset norebalance ceph osd unset nobackfill Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Well, sure, if you want to be all elegant about it ;)
I ran ceph osd crush reweight 75 1.0 and it started recovering right away 3-4 Gbit/s sustained throughput. I know this is a bandaid, waiting on your guidance on how to adjust the wrights above.
Use something like this:
ceph osd set norebalance ceph osd set nobackfill
ceph osd df | awk '/9.1 TiB/ { print $1; }' | while read osdid ; do ceph osd crush reweight osd.$osdid 9.1; done ceph osd df | awk '/7.3 TiB/ { print $1; }' | while read osdid ; do ceph osd crush reweight osd.$osdid 7.3; done ceph osd df | awk '/7.2 TiB/ { print $1; }' | while read osdid ; do ceph osd crush reweight osd.$osdid 7.2; done
ceph osd unset norebalance ceph osd unset nobackfill
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Anthony D'Atri
-
Anthony D'Atri
-
darren＠soothill.com
-
Deep Dish
-
Frédéric Nass
-
Nmz
-
Robert Sander