Den mån 4 mars 2024 kl 11:30 skrev Ml Ml <mliebherr99@googlemail.com>:
Hello,
i wonder why my autobalancer is not working here:
I think the short answer is "because you have so wildly varying sizes both for drives and hosts". If your drive sizes span from 0.5 to 9.5, there will naturally be skewed data, and it is not a huge surprise that the automation has some troubles getting it "good". When the balancer places a PG on a 0.5-sized drive compared to a 9.5-sized one, it eats up 19x more of the "free space" on the smaller one, so there are very few good options when the sizes are so different. Even if you placed all PGs correctly due to size, the 9.5-sized disk would end up getting 19x more IO than the small drive and for hdd, it seldom is possible to gracefully handle a 19-fold increase in IO, most of the time will probably be spent on seeks.
root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_ERR 1 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) 4 pool(s) full
=> osd.17 was too full (92% or something like that)
root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE ... %USE ... PGS TYPE NAME -25 209.50084 - 213 TiB ... 69.56 ... - datacenter xxx-dc-root -19 84.59369 - 86 TiB ... 56.97 ... - rack RZ1.Reihe4.R10 -3 35.49313 - 37 TiB ... 57.88 ... - host ceph02 2 hdd 1.70000 1.00000 1.7 TiB ... 58.77 ... 44 osd.2 3 hdd 1.00000 1.00000 2.7 TiB ... 22.14 ... 25 osd.3 7 hdd 2.50000 1.00000 2.7 TiB ... 58.84 ... 70 osd.7 9 hdd 9.50000 1.00000 9.5 TiB ... 63.07 ... 268 osd.9 13 hdd 2.67029 1.00000 2.7 TiB ... 53.59 ... 65 osd.13 16 hdd 2.89999 1.00000 2.7 TiB ... 59.35 ... 71 osd.16 19 hdd 1.70000 1.00000 1.7 TiB ... 48.98 ... 37 osd.19 23 hdd 2.38419 1.00000 2.4 TiB ... 59.33 ... 64 osd.23 24 hdd 1.39999 1.00000 1.7 TiB ... 51.23 ... 39 osd.24 28 hdd 3.63869 1.00000 3.6 TiB ... 64.17 ... 104 osd.28 31 hdd 2.70000 1.00000 2.7 TiB ... 64.73 ... 76 osd.31 32 hdd 3.39999 1.00000 3.3 TiB ... 67.28 ... 101 osd.32 -9 22.88817 - 23 TiB ... 56.96 ... - host ceph06 35 hdd 7.15259 1.00000 7.2 TiB ... 55.71 ... 182 osd.35 36 hdd 5.24519 1.00000 5.2 TiB ... 53.75 ... 128 osd.36 45 hdd 5.24519 1.00000 5.2 TiB ... 60.91 ... 144 osd.45 48 hdd 5.24519 1.00000 5.2 TiB ... 57.94 ... 139 osd.48 -17 26.21239 - 26 TiB ... 55.67 ... - host ceph08 37 hdd 6.67569 1.00000 6.7 TiB ... 58.17 ... 174 osd.37 40 hdd 9.53670 1.00000 9.5 TiB ... 58.54 ... 250 osd.40 46 hdd 5.00000 1.00000 5.0 TiB ... 52.39 ... 116 osd.46 47 hdd 5.00000 1.00000 5.0 TiB ... 50.05 ... 112 osd.47 -20 59.11053 - 60 TiB ... 82.47 ... - rack RZ1.Reihe4.R9 -4 23.09996 - 24 TiB ... 79.92 ... - host ceph03 5 hdd 1.70000 0.75006 1.7 TiB ... 87.24 ... 66 osd.5 6 hdd 1.70000 0.44998 1.7 TiB ... 47.30 ... 36 osd.6 10 hdd 2.70000 0.85004 2.7 TiB ... 83.23 ... 100 osd.10 15 hdd 2.70000 0.75006 2.7 TiB ... 74.26 ... 88 osd.15 17 hdd 0.50000 0.85004 1.6 TiB ... 91.44 ... 67 osd.17 20 hdd 2.00000 0.85004 1.7 TiB ... 88.41 ... 68 osd.20 21 hdd 2.79999 0.75006 2.7 TiB ... 77.25 ... 91 osd.21 25 hdd 1.70000 0.90002 1.7 TiB ... 78.31 ... 60 osd.25 26 hdd 2.70000 1.00000 2.7 TiB ... 82.75 ... 99 osd.26 27 hdd 2.70000 0.90002 2.7 TiB ... 84.26 ... 101 osd.27 63 hdd 1.89999 0.90002 1.7 TiB ... 84.15 ... 65 osd.63 -13 36.01057 - 36 TiB ... 84.12 ... - host ceph05 11 hdd 7.15259 0.90002 7.2 TiB ... 85.45 ... 273 osd.11 39 hdd 7.20000 0.85004 7.2 TiB ... 80.90 ... 257 osd.39 41 hdd 7.20000 0.75006 7.2 TiB ... 74.95 ... 239 osd.41 42 hdd 9.00000 1.00000 9.5 TiB ... 92.00 ... 392 osd.42 43 hdd 5.45799 1.00000 5.5 TiB ... 84.84 ... 207 osd.43 -21 65.79662 - 66 TiB ... 74.29 ... - rack RZ3.Reihe3.R10 -2 28.49664 - 29 TiB ... 74.79 ... - host ceph01 0 hdd 2.70000 1.00000 2.7 TiB ... 73.82 ... 88 osd.0 1 hdd 3.63869 1.00000 3.6 TiB ... 73.47 ... 121 osd.1 4 hdd 2.70000 1.00000 2.7 TiB ... 74.63 ... 89 osd.4 8 hdd 2.70000 1.00000 2.7 TiB ... 77.10 ... 92 osd.8 12 hdd 2.70000 1.00000 2.7 TiB ... 78.76 ... 94 osd.12 14 hdd 5.45799 1.00000 5.5 TiB ... 78.86 ... 193 osd.14 18 hdd 1.89999 1.00000 2.7 TiB ... 63.79 ... 76 osd.18 22 hdd 1.70000 1.00000 1.7 TiB ... 74.85 ... 57 osd.22 30 hdd 1.70000 1.00000 1.7 TiB ... 76.34 ... 59 osd.30 64 hdd 3.29999 1.00000 3.3 TiB ... 73.48 ... 110 osd.64 -11 12.39999 - 12 TiB ... 73.40 ... - host ceph04 34 hdd 5.20000 1.00000 5.2 TiB ... 72.81 ... 171 osd.34 44 hdd 7.20000 1.00000 7.2 TiB ... 73.83 ... 236 osd.44 -15 24.89998 - 25 TiB ... 74.15 ... - host ceph07 66 hdd 7.20000 1.00000 7.2 TiB ... 74.07 ... 236 osd.66 67 hdd 7.20000 1.00000 7.2 TiB ... 73.74 ... 236 osd.67 68 hdd 3.29999 1.00000 3.3 TiB ... 72.99 ... 110 osd.68 69 hdd 7.20000 1.00000 7.2 TiB ... 75.18 ... 241 osd.69 -1 0 - 0 B ... 0 ... - root default TOTAL 213 TiB ... 69.56
-- May the most significant bit of your life be positive.