[ceph-users] backfill_toofull not clearing on Reef

Hello, I have an 80 OSD cluster (across 8 nodes). The average utilization across my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was replaced (same capacity). For the past week or so the cluster has been recovering, slowly, and reporting backfill_toofull. I can't figure out what's causing the issue given there's ample available capacity. I tried setting the following without any result: ceph osd set-backfillfull-ratio 0.90 # ceph -s ... health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull 11 pgs not deep-scrubbed in time ... data: volumes: 1/1 healthy pools: 13 pools, 1665 pgs objects: 27.81M objects, 79 TiB usage: 197 TiB used, 413 TiB / 610 TiB avail pgs: 16341449/128092605 objects misplaced (12.758%) 1371 active+clean 150 active+remapped+backfill_wait 121 active+remapped+backfill_wait+backfill_toofull 16 active+remapped+backfill_toofull 6 active+clean+scrubbing+deep 1 active+remapped+backfilling io: client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr recovery: 16 MiB/s, 4 objects/s # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 TiB 24.28 0.75 108 up 9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 TiB 36.76 1.14 103 up 16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 TiB 29.82 0.92 109 up 27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 TiB 26.23 0.81 108 up 35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6 TiB 35.51 1.10 113 up 43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5 TiB 37.92 1.17 106 up 51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8 TiB 33.63 1.04 108 up 59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9 TiB 32.70 1.01 101 up 67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9 TiB 31.99 0.99 112 up 74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2 TiB 28.68 0.89 110 up 3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5 TiB 28.45 0.88 112 up 11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1 TiB 21.50 0.67 115 up 19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6 TiB 36.51 1.13 115 up 25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2 TiB 29.08 0.90 91 up 33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2 TiB 27.27 0.84 109 up 41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5 TiB 24.36 0.75 105 up 49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1 TiB 29.98 0.93 107 up 56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0 TiB 31.77 0.98 116 up 62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5 TiB 38.47 1.19 108 up 69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1 TiB 30.05 0.93 109 up 5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2 TiB 31.40 0.97 109 up 13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3 TiB 31.12 0.96 116 up 22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8 TiB 34.72 1.07 114 up 31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0 TiB 31.08 0.96 101 up 39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7 TiB 35.60 1.10 110 up 47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0 TiB 31.89 0.99 108 up 55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4 TiB 39.43 1.22 116 up 65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9 TiB 32.62 1.01 116 up 73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6 TiB 36.13 1.12 111 up 78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7 TiB 35.50 1.10 118 up 4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6 TiB 27.11 0.84 131 up 12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9 TiB 23.80 0.74 106 up 20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4 TiB 26.11 0.81 106 up 28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5 TiB 38.59 1.19 121 up 36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9 TiB 31.98 0.99 108 up 44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8 TiB 33.68 1.04 90 up 52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1 TiB 29.76 0.92 108 up 60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0 TiB 31.04 0.96 104 up 68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8 TiB 33.83 1.05 119 up 76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7 TiB 35.77 1.11 107 up 6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1 TiB 33.01 1.02 111 up 14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3 TiB 30.96 0.96 84 up 23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8 TiB 33.96 1.05 95 up 30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5 TiB 38.67 1.20 93 up 38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0 TiB 31.72 0.98 101 up 46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7 TiB 36.05 1.12 81 up 53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6 TiB 36.12 1.12 88 up 63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2 TiB 42.94 1.33 87 up 75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 TiB 63.47 1.96 356 up 79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6 TiB 36.90 1.14 93 up 7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1 TiB 22.22 0.69 98 up 15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5 TiB 28.50 0.88 114 up 21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6 TiB 22.60 0.70 88 up 29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8 TiB 34.20 1.06 114 up 37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9 TiB 33.02 1.02 128 up 45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6 TiB 36.26 1.12 109 up 54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7 TiB 35.04 1.08 111 up 61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2 TiB 27.87 0.86 103 up 70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0 TiB 31.86 0.99 109 up 77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8 TiB 32.35 1.00 112 up 2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2 TiB 32.27 1.00 115 up 10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9 TiB 24.28 0.75 97 up 18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3 TiB 26.63 0.82 107 up 26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6 TiB 35.60 1.10 111 up 34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6 TiB 36.17 1.12 106 up 42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1 TiB 30.17 0.93 111 up 50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6 TiB 36.05 1.12 101 up 58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4 TiB 39.11 1.21 122 up 66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3 TiB 26.73 0.83 98 up 72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0 TiB 29.57 0.92 110 up 0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3 TiB 30.63 0.95 119 up 8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7 TiB 26.77 0.83 112 up 17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5 TiB 38.79 1.20 107 up 24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9 TiB 32.22 1.00 116 up 32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7 TiB 35.56 1.10 94 up 40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3 TiB 39.44 1.22 106 up 48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9 TiB 32.22 1.00 114 up 57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2 TiB 28.40 0.88 95 up 64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6 TiB 37.42 1.16 130 up 71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9 TiB 32.01 0.99 105 up TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413 TiB 32.31 MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 86.15359 root default -19 10.00000 host 01stor28 1 hdd 1.00000 osd.1 up 1.00000 1.00000 9 hdd 1.00000 osd.9 up 1.00000 1.00000 16 hdd 1.00000 osd.16 up 1.00000 1.00000 27 hdd 1.00000 osd.27 up 1.00000 1.00000 35 hdd 1.00000 osd.35 up 1.00000 1.00000 43 hdd 1.00000 osd.43 up 1.00000 1.00000 51 hdd 1.00000 osd.51 up 1.00000 1.00000 59 hdd 1.00000 osd.59 up 1.00000 1.00000 67 hdd 1.00000 osd.67 up 1.00000 1.00000 74 hdd 1.00000 osd.74 up 1.00000 1.00000 -21 10.00000 host 01stor32 3 hdd 1.00000 osd.3 up 1.00000 1.00000 11 hdd 1.00000 osd.11 up 1.00000 1.00000 19 hdd 1.00000 osd.19 up 1.00000 1.00000 25 hdd 1.00000 osd.25 up 1.00000 1.00000 33 hdd 1.00000 osd.33 up 1.00000 1.00000 41 hdd 1.00000 osd.41 up 1.00000 1.00000 49 hdd 1.00000 osd.49 up 1.00000 1.00000 56 hdd 1.00000 osd.56 up 1.00000 1.00000 62 hdd 1.00000 osd.62 up 1.00000 1.00000 69 hdd 1.00000 osd.69 up 1.00000 1.00000 -15 10.00000 host 02stor23 5 hdd 1.00000 osd.5 up 1.00000 1.00000 13 hdd 1.00000 osd.13 up 1.00000 1.00000 22 hdd 1.00000 osd.22 up 1.00000 1.00000 31 hdd 1.00000 osd.31 up 1.00000 1.00000 39 hdd 1.00000 osd.39 up 1.00000 1.00000 47 hdd 1.00000 osd.47 up 1.00000 1.00000 55 hdd 1.00000 osd.55 up 1.00000 1.00000 65 hdd 1.00000 osd.65 up 1.00000 1.00000 73 hdd 1.00000 osd.73 up 1.00000 1.00000 78 hdd 1.00000 osd.78 up 1.00000 1.00000 -11 10.00000 host 02stor27 4 hdd 1.00000 osd.4 up 1.00000 1.00000 12 hdd 1.00000 osd.12 up 1.00000 1.00000 20 hdd 1.00000 osd.20 up 1.00000 1.00000 28 hdd 1.00000 osd.28 up 1.00000 1.00000 36 hdd 1.00000 osd.36 up 1.00000 1.00000 44 hdd 1.00000 osd.44 up 1.00000 1.00000 52 hdd 1.00000 osd.52 up 1.00000 1.00000 60 hdd 1.00000 osd.60 up 1.00000 1.00000 68 hdd 1.00000 osd.68 up 1.00000 1.00000 76 hdd 1.00000 osd.76 up 1.00000 1.00000 -17 16.15359 host 03stor23 6 hdd 1.00000 osd.6 up 1.00000 1.00000 14 hdd 1.00000 osd.14 up 1.00000 1.00000 23 hdd 1.00000 osd.23 up 1.00000 1.00000 30 hdd 1.00000 osd.30 up 1.00000 1.00000 38 hdd 1.00000 osd.38 up 1.00000 1.00000 46 hdd 1.00000 osd.46 up 1.00000 1.00000 53 hdd 1.00000 osd.53 up 1.00000 1.00000 63 hdd 1.00000 osd.63 up 1.00000 1.00000 75 hdd 7.15359 osd.75 up 1.00000 1.00000 79 hdd 1.00000 osd.79 up 1.00000 1.00000 -13 10.00000 host 03stor27 7 hdd 1.00000 osd.7 up 1.00000 1.00000 15 hdd 1.00000 osd.15 up 1.00000 1.00000 21 hdd 1.00000 osd.21 up 1.00000 1.00000 29 hdd 1.00000 osd.29 up 1.00000 1.00000 37 hdd 1.00000 osd.37 up 1.00000 1.00000 45 hdd 1.00000 osd.45 up 1.00000 1.00000 54 hdd 1.00000 osd.54 up 1.00000 1.00000 61 hdd 1.00000 osd.61 up 1.00000 1.00000 70 hdd 1.00000 osd.70 up 1.00000 1.00000 77 hdd 1.00000 osd.77 up 1.00000 1.00000 -3 10.00000 host 04stor28 2 hdd 1.00000 osd.2 up 1.00000 1.00000 10 hdd 1.00000 osd.10 up 1.00000 1.00000 18 hdd 1.00000 osd.18 up 1.00000 1.00000 26 hdd 1.00000 osd.26 up 1.00000 1.00000 34 hdd 1.00000 osd.34 up 1.00000 1.00000 42 hdd 1.00000 osd.42 up 1.00000 1.00000 50 hdd 1.00000 osd.50 up 1.00000 1.00000 58 hdd 1.00000 osd.58 up 1.00000 1.00000 66 hdd 1.00000 osd.66 up 1.00000 1.00000 72 hdd 1.00000 osd.72 up 1.00000 1.00000 -7 10.00000 host 04stor32 0 hdd 1.00000 osd.0 up 1.00000 1.00000 8 hdd 1.00000 osd.8 up 1.00000 1.00000 17 hdd 1.00000 osd.17 up 1.00000 1.00000 24 hdd 1.00000 osd.24 up 1.00000 1.00000 32 hdd 1.00000 osd.32 up 1.00000 1.00000 40 hdd 1.00000 osd.40 up 1.00000 1.00000 48 hdd 1.00000 osd.48 up 1.00000 1.00000 57 hdd 1.00000 osd.57 up 1.00000 1.00000 64 hdd 1.00000 osd.64 up 1.00000 1.00000 71 hdd 1.00000 osd.71 up 1.00000 1.00000 # ceph health detail HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 137 pgs backfill_toofull pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting [0,67,4] pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting [71,44,22] pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting [58,51,23] pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting [69,45,22] pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting [24,3,63] pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting [35,50,14] pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting [28,16,69] pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6] pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting [78,26,29] pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30] pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting [43,11,17] pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting [68,33,2] pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66] pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting [78,17,6] pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting [57,67,13] pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38] pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting [1,0,28] pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting [47,14,66] pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14] pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting [27,45,38] pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72] pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting [52,69,46] pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting [55,26,32] pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting [34,59,11] pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2] pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting [55,70,46] pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23] pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting [37,25,59] pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23] pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting [11,26,24] pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting [59,60,46] pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6] pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting [35,57,46] pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting [28,9,72] pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting [5,43,72] pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting [70,72,14] pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting [78,10,23] pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting [66,7,53] pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13] pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting [25,15,63] pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting [78,76,34] pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting [24,76,54,16,25,79,22,2] pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting [10,9,56,4,47,61,32,79] pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting [70,41,36,48,23,31,1,2] pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting [10,29,78,25,1,44,64,79] pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting [28,26,15,23,57,16,25,78] pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting [59,46,19,7,72,68,55,8] pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting [76,54,25,31,0,46,58,59] pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting [31,76,1,66,8,56,79,21] pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting [74,0,70,44,39,34,33,6] pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting [44,21,43,18,69,64,38,31] [WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000 pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000 pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000 pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000 pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000 pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000 pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000 pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000 pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000 pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000 pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000 pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000 pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000 pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000 pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000 pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000 pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000 pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000 pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000 pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000 pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000 pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000 pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000 pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000 pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000