Hello, i wonder why my autobalancer is not working here: root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_ERR 1 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) 4 pool(s) full => osd.17 was too full (92% or something like that) root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE ... %USE ... PGS TYPE NAME -25 209.50084 - 213 TiB ... 69.56 ... - datacenter xxx-dc-root -19 84.59369 - 86 TiB ... 56.97 ... - rack RZ1.Reihe4.R10 -3 35.49313 - 37 TiB ... 57.88 ... - host ceph02 2 hdd 1.70000 1.00000 1.7 TiB ... 58.77 ... 44 osd.2 3 hdd 1.00000 1.00000 2.7 TiB ... 22.14 ... 25 osd.3 7 hdd 2.50000 1.00000 2.7 TiB ... 58.84 ... 70 osd.7 9 hdd 9.50000 1.00000 9.5 TiB ... 63.07 ... 268 osd.9 13 hdd 2.67029 1.00000 2.7 TiB ... 53.59 ... 65 osd.13 16 hdd 2.89999 1.00000 2.7 TiB ... 59.35 ... 71 osd.16 19 hdd 1.70000 1.00000 1.7 TiB ... 48.98 ... 37 osd.19 23 hdd 2.38419 1.00000 2.4 TiB ... 59.33 ... 64 osd.23 24 hdd 1.39999 1.00000 1.7 TiB ... 51.23 ... 39 osd.24 28 hdd 3.63869 1.00000 3.6 TiB ... 64.17 ... 104 osd.28 31 hdd 2.70000 1.00000 2.7 TiB ... 64.73 ... 76 osd.31 32 hdd 3.39999 1.00000 3.3 TiB ... 67.28 ... 101 osd.32 -9 22.88817 - 23 TiB ... 56.96 ... - host ceph06 35 hdd 7.15259 1.00000 7.2 TiB ... 55.71 ... 182 osd.35 36 hdd 5.24519 1.00000 5.2 TiB ... 53.75 ... 128 osd.36 45 hdd 5.24519 1.00000 5.2 TiB ... 60.91 ... 144 osd.45 48 hdd 5.24519 1.00000 5.2 TiB ... 57.94 ... 139 osd.48 -17 26.21239 - 26 TiB ... 55.67 ... - host ceph08 37 hdd 6.67569 1.00000 6.7 TiB ... 58.17 ... 174 osd.37 40 hdd 9.53670 1.00000 9.5 TiB ... 58.54 ... 250 osd.40 46 hdd 5.00000 1.00000 5.0 TiB ... 52.39 ... 116 osd.46 47 hdd 5.00000 1.00000 5.0 TiB ... 50.05 ... 112 osd.47 -20 59.11053 - 60 TiB ... 82.47 ... - rack RZ1.Reihe4.R9 -4 23.09996 - 24 TiB ... 79.92 ... - host ceph03 5 hdd 1.70000 0.75006 1.7 TiB ... 87.24 ... 66 osd.5 6 hdd 1.70000 0.44998 1.7 TiB ... 47.30 ... 36 osd.6 10 hdd 2.70000 0.85004 2.7 TiB ... 83.23 ... 100 osd.10 15 hdd 2.70000 0.75006 2.7 TiB ... 74.26 ... 88 osd.15 17 hdd 0.50000 0.85004 1.6 TiB ... 91.44 ... 67 osd.17 20 hdd 2.00000 0.85004 1.7 TiB ... 88.41 ... 68 osd.20 21 hdd 2.79999 0.75006 2.7 TiB ... 77.25 ... 91 osd.21 25 hdd 1.70000 0.90002 1.7 TiB ... 78.31 ... 60 osd.25 26 hdd 2.70000 1.00000 2.7 TiB ... 82.75 ... 99 osd.26 27 hdd 2.70000 0.90002 2.7 TiB ... 84.26 ... 101 osd.27 63 hdd 1.89999 0.90002 1.7 TiB ... 84.15 ... 65 osd.63 -13 36.01057 - 36 TiB ... 84.12 ... - host ceph05 11 hdd 7.15259 0.90002 7.2 TiB ... 85.45 ... 273 osd.11 39 hdd 7.20000 0.85004 7.2 TiB ... 80.90 ... 257 osd.39 41 hdd 7.20000 0.75006 7.2 TiB ... 74.95 ... 239 osd.41 42 hdd 9.00000 1.00000 9.5 TiB ... 92.00 ... 392 osd.42 43 hdd 5.45799 1.00000 5.5 TiB ... 84.84 ... 207 osd.43 -21 65.79662 - 66 TiB ... 74.29 ... - rack RZ3.Reihe3.R10 -2 28.49664 - 29 TiB ... 74.79 ... - host ceph01 0 hdd 2.70000 1.00000 2.7 TiB ... 73.82 ... 88 osd.0 1 hdd 3.63869 1.00000 3.6 TiB ... 73.47 ... 121 osd.1 4 hdd 2.70000 1.00000 2.7 TiB ... 74.63 ... 89 osd.4 8 hdd 2.70000 1.00000 2.7 TiB ... 77.10 ... 92 osd.8 12 hdd 2.70000 1.00000 2.7 TiB ... 78.76 ... 94 osd.12 14 hdd 5.45799 1.00000 5.5 TiB ... 78.86 ... 193 osd.14 18 hdd 1.89999 1.00000 2.7 TiB ... 63.79 ... 76 osd.18 22 hdd 1.70000 1.00000 1.7 TiB ... 74.85 ... 57 osd.22 30 hdd 1.70000 1.00000 1.7 TiB ... 76.34 ... 59 osd.30 64 hdd 3.29999 1.00000 3.3 TiB ... 73.48 ... 110 osd.64 -11 12.39999 - 12 TiB ... 73.40 ... - host ceph04 34 hdd 5.20000 1.00000 5.2 TiB ... 72.81 ... 171 osd.34 44 hdd 7.20000 1.00000 7.2 TiB ... 73.83 ... 236 osd.44 -15 24.89998 - 25 TiB ... 74.15 ... - host ceph07 66 hdd 7.20000 1.00000 7.2 TiB ... 74.07 ... 236 osd.66 67 hdd 7.20000 1.00000 7.2 TiB ... 73.74 ... 236 osd.67 68 hdd 3.29999 1.00000 3.3 TiB ... 72.99 ... 110 osd.68 69 hdd 7.20000 1.00000 7.2 TiB ... 75.18 ... 241 osd.69 -1 0 - 0 B ... 0 ... - root default TOTAL 213 TiB ... 69.56 root@ceph01:~# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.256761", "last_optimize_started": "Mon Mar 4 10:25:10 2024", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } Where as reweight-by-utilization would change quite a bit!: root@ceph01:~# ceph osd test-reweight-by-utilization 110 .5 10 moved 260 / 6627 (3.92334%) avg 127.442 stddev 81.345 -> 78.169 (expected baseline 11.18) min osd.17 with 24 -> 16 pgs (0.188321 -> 0.125547 * mean) max osd.42 with 401 -> 320 pgs (3.14652 -> 2.51094 * mean) oload 110 max_change 0.5 max_change_osds 10 average_utilization 0.6956 overload_utilization 0.7652 osd.42 weight 1.0000 -> 0.7561 osd.6 weight 0.4500 -> 0.6616 osd.17 weight 0.8500 -> 0.6466 osd.20 weight 0.8500 -> 0.6688 osd.5 weight 0.7501 -> 0.6004 osd.11 weight 0.9000 -> 0.7326 osd.43 weight 1.0000 -> 0.8199 osd.27 weight 0.9000 -> 0.7430 osd.63 weight 0.9000 -> 0.7440 osd.10 weight 0.8500 -> 0.7104 no change root@ceph01:~# ceph versions { "mon": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 2 }, "osd": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 52 }, "mds": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 61 } } Cheers, Michael
Den mån 4 mars 2024 kl 11:30 skrev Ml Ml <mliebherr99@googlemail.com>:
Hello,
i wonder why my autobalancer is not working here:
I think the short answer is "because you have so wildly varying sizes both for drives and hosts". If your drive sizes span from 0.5 to 9.5, there will naturally be skewed data, and it is not a huge surprise that the automation has some troubles getting it "good". When the balancer places a PG on a 0.5-sized drive compared to a 9.5-sized one, it eats up 19x more of the "free space" on the smaller one, so there are very few good options when the sizes are so different. Even if you placed all PGs correctly due to size, the 9.5-sized disk would end up getting 19x more IO than the small drive and for hdd, it seldom is possible to gracefully handle a 19-fold increase in IO, most of the time will probably be spent on seeks.
root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_ERR 1 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) 4 pool(s) full
=> osd.17 was too full (92% or something like that)
root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE ... %USE ... PGS TYPE NAME -25 209.50084 - 213 TiB ... 69.56 ... - datacenter xxx-dc-root -19 84.59369 - 86 TiB ... 56.97 ... - rack RZ1.Reihe4.R10 -3 35.49313 - 37 TiB ... 57.88 ... - host ceph02 2 hdd 1.70000 1.00000 1.7 TiB ... 58.77 ... 44 osd.2 3 hdd 1.00000 1.00000 2.7 TiB ... 22.14 ... 25 osd.3 7 hdd 2.50000 1.00000 2.7 TiB ... 58.84 ... 70 osd.7 9 hdd 9.50000 1.00000 9.5 TiB ... 63.07 ... 268 osd.9 13 hdd 2.67029 1.00000 2.7 TiB ... 53.59 ... 65 osd.13 16 hdd 2.89999 1.00000 2.7 TiB ... 59.35 ... 71 osd.16 19 hdd 1.70000 1.00000 1.7 TiB ... 48.98 ... 37 osd.19 23 hdd 2.38419 1.00000 2.4 TiB ... 59.33 ... 64 osd.23 24 hdd 1.39999 1.00000 1.7 TiB ... 51.23 ... 39 osd.24 28 hdd 3.63869 1.00000 3.6 TiB ... 64.17 ... 104 osd.28 31 hdd 2.70000 1.00000 2.7 TiB ... 64.73 ... 76 osd.31 32 hdd 3.39999 1.00000 3.3 TiB ... 67.28 ... 101 osd.32 -9 22.88817 - 23 TiB ... 56.96 ... - host ceph06 35 hdd 7.15259 1.00000 7.2 TiB ... 55.71 ... 182 osd.35 36 hdd 5.24519 1.00000 5.2 TiB ... 53.75 ... 128 osd.36 45 hdd 5.24519 1.00000 5.2 TiB ... 60.91 ... 144 osd.45 48 hdd 5.24519 1.00000 5.2 TiB ... 57.94 ... 139 osd.48 -17 26.21239 - 26 TiB ... 55.67 ... - host ceph08 37 hdd 6.67569 1.00000 6.7 TiB ... 58.17 ... 174 osd.37 40 hdd 9.53670 1.00000 9.5 TiB ... 58.54 ... 250 osd.40 46 hdd 5.00000 1.00000 5.0 TiB ... 52.39 ... 116 osd.46 47 hdd 5.00000 1.00000 5.0 TiB ... 50.05 ... 112 osd.47 -20 59.11053 - 60 TiB ... 82.47 ... - rack RZ1.Reihe4.R9 -4 23.09996 - 24 TiB ... 79.92 ... - host ceph03 5 hdd 1.70000 0.75006 1.7 TiB ... 87.24 ... 66 osd.5 6 hdd 1.70000 0.44998 1.7 TiB ... 47.30 ... 36 osd.6 10 hdd 2.70000 0.85004 2.7 TiB ... 83.23 ... 100 osd.10 15 hdd 2.70000 0.75006 2.7 TiB ... 74.26 ... 88 osd.15 17 hdd 0.50000 0.85004 1.6 TiB ... 91.44 ... 67 osd.17 20 hdd 2.00000 0.85004 1.7 TiB ... 88.41 ... 68 osd.20 21 hdd 2.79999 0.75006 2.7 TiB ... 77.25 ... 91 osd.21 25 hdd 1.70000 0.90002 1.7 TiB ... 78.31 ... 60 osd.25 26 hdd 2.70000 1.00000 2.7 TiB ... 82.75 ... 99 osd.26 27 hdd 2.70000 0.90002 2.7 TiB ... 84.26 ... 101 osd.27 63 hdd 1.89999 0.90002 1.7 TiB ... 84.15 ... 65 osd.63 -13 36.01057 - 36 TiB ... 84.12 ... - host ceph05 11 hdd 7.15259 0.90002 7.2 TiB ... 85.45 ... 273 osd.11 39 hdd 7.20000 0.85004 7.2 TiB ... 80.90 ... 257 osd.39 41 hdd 7.20000 0.75006 7.2 TiB ... 74.95 ... 239 osd.41 42 hdd 9.00000 1.00000 9.5 TiB ... 92.00 ... 392 osd.42 43 hdd 5.45799 1.00000 5.5 TiB ... 84.84 ... 207 osd.43 -21 65.79662 - 66 TiB ... 74.29 ... - rack RZ3.Reihe3.R10 -2 28.49664 - 29 TiB ... 74.79 ... - host ceph01 0 hdd 2.70000 1.00000 2.7 TiB ... 73.82 ... 88 osd.0 1 hdd 3.63869 1.00000 3.6 TiB ... 73.47 ... 121 osd.1 4 hdd 2.70000 1.00000 2.7 TiB ... 74.63 ... 89 osd.4 8 hdd 2.70000 1.00000 2.7 TiB ... 77.10 ... 92 osd.8 12 hdd 2.70000 1.00000 2.7 TiB ... 78.76 ... 94 osd.12 14 hdd 5.45799 1.00000 5.5 TiB ... 78.86 ... 193 osd.14 18 hdd 1.89999 1.00000 2.7 TiB ... 63.79 ... 76 osd.18 22 hdd 1.70000 1.00000 1.7 TiB ... 74.85 ... 57 osd.22 30 hdd 1.70000 1.00000 1.7 TiB ... 76.34 ... 59 osd.30 64 hdd 3.29999 1.00000 3.3 TiB ... 73.48 ... 110 osd.64 -11 12.39999 - 12 TiB ... 73.40 ... - host ceph04 34 hdd 5.20000 1.00000 5.2 TiB ... 72.81 ... 171 osd.34 44 hdd 7.20000 1.00000 7.2 TiB ... 73.83 ... 236 osd.44 -15 24.89998 - 25 TiB ... 74.15 ... - host ceph07 66 hdd 7.20000 1.00000 7.2 TiB ... 74.07 ... 236 osd.66 67 hdd 7.20000 1.00000 7.2 TiB ... 73.74 ... 236 osd.67 68 hdd 3.29999 1.00000 3.3 TiB ... 72.99 ... 110 osd.68 69 hdd 7.20000 1.00000 7.2 TiB ... 75.18 ... 241 osd.69 -1 0 - 0 B ... 0 ... - root default TOTAL 213 TiB ... 69.56
-- May the most significant bit of your life be positive.
I think the short answer is "because you have so wildly varying sizes both for drives and hosts".
Arguably OP's OSDs *are* balanced in that their PGs are roughly in line with their sizes, but indeed the size disparity is problematic in some ways. Notably, the 500GB OSD should just be removed. I think balancing doesn't account for WAL/DB/other overhead, so it won't be accurately accounted for and can't hold much data nyway. This cluster shows evidence of reweight-by-utilization having been run, but only on two of the hosts. If the balancer module is active, those override weights will confound it.
If your drive sizes span from 0.5 to 9.5, there will naturally be skewed data, and it is not a huge surprise that the automation has some troubles getting it "good". When the balancer places a PG on a 0.5-sized drive compared to a 9.5-sized one, it eats up 19x more of the "free space" on the smaller one, so there are very few good options when the sizes are so different. Even if you placed all PGs correctly due to size, the 9.5-sized disk would end up getting 19x more IO than the small drive and for hdd, it seldom is possible to gracefully handle a 19-fold increase in IO, most of the time will probably be spent on seeks.
Did the balancer has enabled pools ? "ceph balancer pool ls" Actually I am wondering if the balancer do something when no pools are added. On Mon, Mar 4, 2024, 11:30 Ml Ml <mliebherr99@googlemail.com> wrote:
Hello,
i wonder why my autobalancer is not working here:
root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_ERR 1 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) 4 pool(s) full
=> osd.17 was too full (92% or something like that)
root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE ... %USE ... PGS TYPE NAME -25 209.50084 - 213 TiB ... 69.56 ... - datacenter xxx-dc-root -19 84.59369 - 86 TiB ... 56.97 ... - rack RZ1.Reihe4.R10 -3 35.49313 - 37 TiB ... 57.88 ... - host ceph02 2 hdd 1.70000 1.00000 1.7 TiB ... 58.77 ... 44 osd.2 3 hdd 1.00000 1.00000 2.7 TiB ... 22.14 ... 25 osd.3 7 hdd 2.50000 1.00000 2.7 TiB ... 58.84 ... 70 osd.7 9 hdd 9.50000 1.00000 9.5 TiB ... 63.07 ... 268 osd.9 13 hdd 2.67029 1.00000 2.7 TiB ... 53.59 ... 65 osd.13 16 hdd 2.89999 1.00000 2.7 TiB ... 59.35 ... 71 osd.16 19 hdd 1.70000 1.00000 1.7 TiB ... 48.98 ... 37 osd.19 23 hdd 2.38419 1.00000 2.4 TiB ... 59.33 ... 64 osd.23 24 hdd 1.39999 1.00000 1.7 TiB ... 51.23 ... 39 osd.24 28 hdd 3.63869 1.00000 3.6 TiB ... 64.17 ... 104 osd.28 31 hdd 2.70000 1.00000 2.7 TiB ... 64.73 ... 76 osd.31 32 hdd 3.39999 1.00000 3.3 TiB ... 67.28 ... 101 osd.32 -9 22.88817 - 23 TiB ... 56.96 ... - host ceph06 35 hdd 7.15259 1.00000 7.2 TiB ... 55.71 ... 182 osd.35 36 hdd 5.24519 1.00000 5.2 TiB ... 53.75 ... 128 osd.36 45 hdd 5.24519 1.00000 5.2 TiB ... 60.91 ... 144 osd.45 48 hdd 5.24519 1.00000 5.2 TiB ... 57.94 ... 139 osd.48 -17 26.21239 - 26 TiB ... 55.67 ... - host ceph08 37 hdd 6.67569 1.00000 6.7 TiB ... 58.17 ... 174 osd.37 40 hdd 9.53670 1.00000 9.5 TiB ... 58.54 ... 250 osd.40 46 hdd 5.00000 1.00000 5.0 TiB ... 52.39 ... 116 osd.46 47 hdd 5.00000 1.00000 5.0 TiB ... 50.05 ... 112 osd.47 -20 59.11053 - 60 TiB ... 82.47 ... - rack RZ1.Reihe4.R9 -4 23.09996 - 24 TiB ... 79.92 ... - host ceph03 5 hdd 1.70000 0.75006 1.7 TiB ... 87.24 ... 66 osd.5 6 hdd 1.70000 0.44998 1.7 TiB ... 47.30 ... 36 osd.6 10 hdd 2.70000 0.85004 2.7 TiB ... 83.23 ... 100 osd.10 15 hdd 2.70000 0.75006 2.7 TiB ... 74.26 ... 88 osd.15 17 hdd 0.50000 0.85004 1.6 TiB ... 91.44 ... 67 osd.17 20 hdd 2.00000 0.85004 1.7 TiB ... 88.41 ... 68 osd.20 21 hdd 2.79999 0.75006 2.7 TiB ... 77.25 ... 91 osd.21 25 hdd 1.70000 0.90002 1.7 TiB ... 78.31 ... 60 osd.25 26 hdd 2.70000 1.00000 2.7 TiB ... 82.75 ... 99 osd.26 27 hdd 2.70000 0.90002 2.7 TiB ... 84.26 ... 101 osd.27 63 hdd 1.89999 0.90002 1.7 TiB ... 84.15 ... 65 osd.63 -13 36.01057 - 36 TiB ... 84.12 ... - host ceph05 11 hdd 7.15259 0.90002 7.2 TiB ... 85.45 ... 273 osd.11 39 hdd 7.20000 0.85004 7.2 TiB ... 80.90 ... 257 osd.39 41 hdd 7.20000 0.75006 7.2 TiB ... 74.95 ... 239 osd.41 42 hdd 9.00000 1.00000 9.5 TiB ... 92.00 ... 392 osd.42 43 hdd 5.45799 1.00000 5.5 TiB ... 84.84 ... 207 osd.43 -21 65.79662 - 66 TiB ... 74.29 ... - rack RZ3.Reihe3.R10 -2 28.49664 - 29 TiB ... 74.79 ... - host ceph01 0 hdd 2.70000 1.00000 2.7 TiB ... 73.82 ... 88 osd.0 1 hdd 3.63869 1.00000 3.6 TiB ... 73.47 ... 121 osd.1 4 hdd 2.70000 1.00000 2.7 TiB ... 74.63 ... 89 osd.4 8 hdd 2.70000 1.00000 2.7 TiB ... 77.10 ... 92 osd.8 12 hdd 2.70000 1.00000 2.7 TiB ... 78.76 ... 94 osd.12 14 hdd 5.45799 1.00000 5.5 TiB ... 78.86 ... 193 osd.14 18 hdd 1.89999 1.00000 2.7 TiB ... 63.79 ... 76 osd.18 22 hdd 1.70000 1.00000 1.7 TiB ... 74.85 ... 57 osd.22 30 hdd 1.70000 1.00000 1.7 TiB ... 76.34 ... 59 osd.30 64 hdd 3.29999 1.00000 3.3 TiB ... 73.48 ... 110 osd.64 -11 12.39999 - 12 TiB ... 73.40 ... - host ceph04 34 hdd 5.20000 1.00000 5.2 TiB ... 72.81 ... 171 osd.34 44 hdd 7.20000 1.00000 7.2 TiB ... 73.83 ... 236 osd.44 -15 24.89998 - 25 TiB ... 74.15 ... - host ceph07 66 hdd 7.20000 1.00000 7.2 TiB ... 74.07 ... 236 osd.66 67 hdd 7.20000 1.00000 7.2 TiB ... 73.74 ... 236 osd.67 68 hdd 3.29999 1.00000 3.3 TiB ... 72.99 ... 110 osd.68 69 hdd 7.20000 1.00000 7.2 TiB ... 75.18 ... 241 osd.69 -1 0 - 0 B ... 0 ... - root default TOTAL 213 TiB ... 69.56
root@ceph01:~# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.256761", "last_optimize_started": "Mon Mar 4 10:25:10 2024", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] }
Where as reweight-by-utilization would change quite a bit!:
root@ceph01:~# ceph osd test-reweight-by-utilization 110 .5 10 moved 260 / 6627 (3.92334%) avg 127.442 stddev 81.345 -> 78.169 (expected baseline 11.18) min osd.17 with 24 -> 16 pgs (0.188321 -> 0.125547 * mean) max osd.42 with 401 -> 320 pgs (3.14652 -> 2.51094 * mean)
oload 110 max_change 0.5 max_change_osds 10 average_utilization 0.6956 overload_utilization 0.7652 osd.42 weight 1.0000 -> 0.7561 osd.6 weight 0.4500 -> 0.6616 osd.17 weight 0.8500 -> 0.6466 osd.20 weight 0.8500 -> 0.6688 osd.5 weight 0.7501 -> 0.6004 osd.11 weight 0.9000 -> 0.7326 osd.43 weight 1.0000 -> 0.8199 osd.27 weight 0.9000 -> 0.7430 osd.63 weight 0.9000 -> 0.7440 osd.10 weight 0.8500 -> 0.7104 no change
root@ceph01:~# ceph versions { "mon": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 2 }, "osd": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 52 }, "mds": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 61 } }
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The balancer will operate on all pools unless otherwise specified. Josh On Mon, Mar 4, 2024 at 1:12 PM Cedric <yipikai7@gmail.com> wrote:
Did the balancer has enabled pools ? "ceph balancer pool ls"
Actually I am wondering if the balancer do something when no pools are added.
On Mon, Mar 4, 2024, 11:30 Ml Ml <mliebherr99@googlemail.com> wrote:
Hello,
i wonder why my autobalancer is not working here:
root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_ERR 1 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) 4 pool(s) full
=> osd.17 was too full (92% or something like that)
root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE ... %USE ... PGS TYPE NAME -25 209.50084 - 213 TiB ... 69.56 ... - datacenter xxx-dc-root -19 84.59369 - 86 TiB ... 56.97 ... - rack RZ1.Reihe4.R10 -3 35.49313 - 37 TiB ... 57.88 ... - host ceph02 2 hdd 1.70000 1.00000 1.7 TiB ... 58.77 ... 44 osd.2 3 hdd 1.00000 1.00000 2.7 TiB ... 22.14 ... 25 osd.3 7 hdd 2.50000 1.00000 2.7 TiB ... 58.84 ... 70 osd.7 9 hdd 9.50000 1.00000 9.5 TiB ... 63.07 ... 268 osd.9 13 hdd 2.67029 1.00000 2.7 TiB ... 53.59 ... 65 osd.13 16 hdd 2.89999 1.00000 2.7 TiB ... 59.35 ... 71 osd.16 19 hdd 1.70000 1.00000 1.7 TiB ... 48.98 ... 37 osd.19 23 hdd 2.38419 1.00000 2.4 TiB ... 59.33 ... 64 osd.23 24 hdd 1.39999 1.00000 1.7 TiB ... 51.23 ... 39 osd.24 28 hdd 3.63869 1.00000 3.6 TiB ... 64.17 ... 104 osd.28 31 hdd 2.70000 1.00000 2.7 TiB ... 64.73 ... 76 osd.31 32 hdd 3.39999 1.00000 3.3 TiB ... 67.28 ... 101 osd.32 -9 22.88817 - 23 TiB ... 56.96 ... - host ceph06 35 hdd 7.15259 1.00000 7.2 TiB ... 55.71 ... 182 osd.35 36 hdd 5.24519 1.00000 5.2 TiB ... 53.75 ... 128 osd.36 45 hdd 5.24519 1.00000 5.2 TiB ... 60.91 ... 144 osd.45 48 hdd 5.24519 1.00000 5.2 TiB ... 57.94 ... 139 osd.48 -17 26.21239 - 26 TiB ... 55.67 ... - host ceph08 37 hdd 6.67569 1.00000 6.7 TiB ... 58.17 ... 174 osd.37 40 hdd 9.53670 1.00000 9.5 TiB ... 58.54 ... 250 osd.40 46 hdd 5.00000 1.00000 5.0 TiB ... 52.39 ... 116 osd.46 47 hdd 5.00000 1.00000 5.0 TiB ... 50.05 ... 112 osd.47 -20 59.11053 - 60 TiB ... 82.47 ... - rack RZ1.Reihe4.R9 -4 23.09996 - 24 TiB ... 79.92 ... - host ceph03 5 hdd 1.70000 0.75006 1.7 TiB ... 87.24 ... 66 osd.5 6 hdd 1.70000 0.44998 1.7 TiB ... 47.30 ... 36 osd.6 10 hdd 2.70000 0.85004 2.7 TiB ... 83.23 ... 100 osd.10 15 hdd 2.70000 0.75006 2.7 TiB ... 74.26 ... 88 osd.15 17 hdd 0.50000 0.85004 1.6 TiB ... 91.44 ... 67 osd.17 20 hdd 2.00000 0.85004 1.7 TiB ... 88.41 ... 68 osd.20 21 hdd 2.79999 0.75006 2.7 TiB ... 77.25 ... 91 osd.21 25 hdd 1.70000 0.90002 1.7 TiB ... 78.31 ... 60 osd.25 26 hdd 2.70000 1.00000 2.7 TiB ... 82.75 ... 99 osd.26 27 hdd 2.70000 0.90002 2.7 TiB ... 84.26 ... 101 osd.27 63 hdd 1.89999 0.90002 1.7 TiB ... 84.15 ... 65 osd.63 -13 36.01057 - 36 TiB ... 84.12 ... - host ceph05 11 hdd 7.15259 0.90002 7.2 TiB ... 85.45 ... 273 osd.11 39 hdd 7.20000 0.85004 7.2 TiB ... 80.90 ... 257 osd.39 41 hdd 7.20000 0.75006 7.2 TiB ... 74.95 ... 239 osd.41 42 hdd 9.00000 1.00000 9.5 TiB ... 92.00 ... 392 osd.42 43 hdd 5.45799 1.00000 5.5 TiB ... 84.84 ... 207 osd.43 -21 65.79662 - 66 TiB ... 74.29 ... - rack RZ3.Reihe3.R10 -2 28.49664 - 29 TiB ... 74.79 ... - host ceph01 0 hdd 2.70000 1.00000 2.7 TiB ... 73.82 ... 88 osd.0 1 hdd 3.63869 1.00000 3.6 TiB ... 73.47 ... 121 osd.1 4 hdd 2.70000 1.00000 2.7 TiB ... 74.63 ... 89 osd.4 8 hdd 2.70000 1.00000 2.7 TiB ... 77.10 ... 92 osd.8 12 hdd 2.70000 1.00000 2.7 TiB ... 78.76 ... 94 osd.12 14 hdd 5.45799 1.00000 5.5 TiB ... 78.86 ... 193 osd.14 18 hdd 1.89999 1.00000 2.7 TiB ... 63.79 ... 76 osd.18 22 hdd 1.70000 1.00000 1.7 TiB ... 74.85 ... 57 osd.22 30 hdd 1.70000 1.00000 1.7 TiB ... 76.34 ... 59 osd.30 64 hdd 3.29999 1.00000 3.3 TiB ... 73.48 ... 110 osd.64 -11 12.39999 - 12 TiB ... 73.40 ... - host ceph04 34 hdd 5.20000 1.00000 5.2 TiB ... 72.81 ... 171 osd.34 44 hdd 7.20000 1.00000 7.2 TiB ... 73.83 ... 236 osd.44 -15 24.89998 - 25 TiB ... 74.15 ... - host ceph07 66 hdd 7.20000 1.00000 7.2 TiB ... 74.07 ... 236 osd.66 67 hdd 7.20000 1.00000 7.2 TiB ... 73.74 ... 236 osd.67 68 hdd 3.29999 1.00000 3.3 TiB ... 72.99 ... 110 osd.68 69 hdd 7.20000 1.00000 7.2 TiB ... 75.18 ... 241 osd.69 -1 0 - 0 B ... 0 ... - root default TOTAL 213 TiB ... 69.56
root@ceph01:~# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.256761", "last_optimize_started": "Mon Mar 4 10:25:10 2024", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] }
Where as reweight-by-utilization would change quite a bit!:
root@ceph01:~# ceph osd test-reweight-by-utilization 110 .5 10 moved 260 / 6627 (3.92334%) avg 127.442 stddev 81.345 -> 78.169 (expected baseline 11.18) min osd.17 with 24 -> 16 pgs (0.188321 -> 0.125547 * mean) max osd.42 with 401 -> 320 pgs (3.14652 -> 2.51094 * mean)
oload 110 max_change 0.5 max_change_osds 10 average_utilization 0.6956 overload_utilization 0.7652 osd.42 weight 1.0000 -> 0.7561 osd.6 weight 0.4500 -> 0.6616 osd.17 weight 0.8500 -> 0.6466 osd.20 weight 0.8500 -> 0.6688 osd.5 weight 0.7501 -> 0.6004 osd.11 weight 0.9000 -> 0.7326 osd.43 weight 1.0000 -> 0.8199 osd.27 weight 0.9000 -> 0.7430 osd.63 weight 0.9000 -> 0.7440 osd.10 weight 0.8500 -> 0.7104 no change
root@ceph01:~# ceph versions { "mon": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 2 }, "osd": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 52 }, "mds": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 61 } }
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Cedric
-
Janne Johansson
-
Joshua Baergen
-
Ml Ml