Thomas, I have a master branch version of the code to test. The nautilus backport https://github.com/ceph/ceph/pull/31956 should be the same. Using your OSDMap, the code in master branch and some additional changes to osdmaptool I was able to balance your cluster. The osdmaptool changes simulate the mgr active balancer behavior. It took 302 rounds with 10 maximum upmaps per crush rule set of pools per round. With the default 1 minute sleeps inside the mgr it would take about 5 hours. Obviously, recovery/backfill has to finish before the cluster settles into the new configuration. It needed 3402 additional upmaps. Because all pools for a given crush rule are balanced together you can see that this is more balanced than Rich's configuration using Luminous. The pool hdb_backup which has its own rule was the most difficult to balance. This balancer code is subject to change before final release of the next Nautilus point release. Final layout: osd.0 pgs 33 osd.1 pgs 31 osd.2 pgs 34 osd.3 pgs 33 osd.4 pgs 31 osd.5 pgs 31 osd.6 pgs 32 osd.7 pgs 31 osd.8 pgs 264 osd.9 pgs 28 osd.10 pgs 29 osd.11 pgs 28 osd.12 pgs 263 osd.13 pgs 27 osd.14 pgs 27 osd.15 pgs 27 osd.16 pgs 262 osd.17 pgs 27 osd.18 pgs 27 osd.19 pgs 262 osd.20 pgs 28 osd.21 pgs 27 osd.22 pgs 28 osd.23 pgs 29 osd.24 pgs 29 osd.25 pgs 28 osd.26 pgs 263 osd.27 pgs 28 osd.28 pgs 28 osd.29 pgs 27 osd.30 pgs 262 osd.31 pgs 86 osd.32 pgs 86 osd.33 pgs 86 osd.34 pgs 86 osd.35 pgs 87 osd.36 pgs 86 osd.37 pgs 86 osd.38 pgs 86 osd.39 pgs 86 osd.40 pgs 86 osd.41 pgs 86 osd.42 pgs 49 osd.43 pgs 62 osd.44 pgs 62 osd.45 pgs 56 osd.46 pgs 49 osd.47 pgs 56 osd.48 pgs 56 osd.49 pgs 56 osd.50 pgs 49 osd.51 pgs 62 osd.52 pgs 56 osd.53 pgs 43 osd.54 pgs 55 osd.55 pgs 56 osd.56 pgs 49 osd.57 pgs 49 osd.58 pgs 49 osd.59 pgs 86 osd.60 pgs 86 osd.61 pgs 86 osd.62 pgs 86 osd.63 pgs 86 osd.64 pgs 87 osd.65 pgs 86 osd.66 pgs 87 osd.67 pgs 87 osd.68 pgs 87 osd.69 pgs 86 osd.70 pgs 86 osd.71 pgs 86 osd.72 pgs 86 osd.73 pgs 87 osd.74 pgs 87 osd.75 pgs 86 osd.76 pgs 49 osd.77 pgs 49 osd.78 pgs 55 osd.79 pgs 49 osd.80 pgs 62 osd.81 pgs 62 osd.82 pgs 62 osd.83 pgs 49 osd.84 pgs 56 osd.85 pgs 61 osd.86 pgs 49 osd.87 pgs 61 osd.88 pgs 62 osd.89 pgs 43 osd.90 pgs 49 osd.91 pgs 55 osd.92 pgs 62 osd.93 pgs 61 osd.94 pgs 55 osd.95 pgs 56 osd.96 pgs 49 osd.97 pgs 43 osd.98 pgs 49 osd.99 pgs 56 osd.100 pgs 43 osd.101 pgs 49 osd.102 pgs 43 osd.103 pgs 55 osd.104 pgs 43 osd.105 pgs 43 osd.106 pgs 62 osd.107 pgs 49 osd.108 pgs 62 osd.109 pgs 62 osd.110 pgs 62 osd.111 pgs 62 osd.112 pgs 49 osd.113 pgs 49 osd.114 pgs 49 osd.115 pgs 49 osd.116 pgs 62 osd.117 pgs 56 osd.118 pgs 55 osd.119 pgs 49 osd.120 pgs 43 osd.121 pgs 55 osd.122 pgs 31 osd.123 pgs 56 osd.124 pgs 62 osd.125 pgs 62 osd.126 pgs 62 osd.127 pgs 49 osd.128 pgs 49 osd.129 pgs 56 osd.130 pgs 61 osd.131 pgs 55 osd.132 pgs 55 osd.133 pgs 55 osd.134 pgs 49 osd.135 pgs 55 osd.136 pgs 55 osd.137 pgs 43 osd.138 pgs 55 osd.139 pgs 55 osd.140 pgs 49 osd.141 pgs 43 osd.142 pgs 55 osd.143 pgs 55 osd.144 pgs 55 osd.145 pgs 55 osd.146 pgs 55 osd.147 pgs 62 osd.148 pgs 55 osd.149 pgs 62 osd.150 pgs 49 osd.151 pgs 55 osd.152 pgs 55 osd.153 pgs 49 osd.154 pgs 49 osd.155 pgs 55 osd.156 pgs 43 osd.157 pgs 49 osd.157 pgs 49 osd.158 pgs 62 osd.159 pgs 49 osd.160 pgs 55 osd.161 pgs 49 osd.162 pgs 62 osd.163 pgs 49 osd.164 pgs 55 osd.165 pgs 62 osd.166 pgs 49 osd.167 pgs 43 osd.168 pgs 49 osd.169 pgs 61 osd.170 pgs 62 osd.171 pgs 55 osd.172 pgs 87 osd.173 pgs 87 osd.174 pgs 87 osd.175 pgs 87 osd.176 pgs 87 osd.177 pgs 86 osd.178 pgs 86 osd.179 pgs 87 osd.180 pgs 86 osd.181 pgs 86 osd.182 pgs 87 osd.183 pgs 87 osd.184 pgs 86 osd.185 pgs 86 osd.186 pgs 87 osd.187 pgs 86 osd.188 pgs 87 osd.189 pgs 43 osd.190 pgs 49 osd.191 pgs 55 osd.192 pgs 62 osd.193 pgs 55 osd.194 pgs 43 osd.195 pgs 62 osd.196 pgs 43 osd.197 pgs 49 osd.198 pgs 62 osd.199 pgs 55 osd.200 pgs 37 osd.201 pgs 55 osd.202 pgs 49 osd.203 pgs 49 osd.204 pgs 62 osd.205 pgs 62 osd.206 pgs 55 osd.207 pgs 49 osd.208 pgs 43 osd.209 pgs 43 osd.210 pgs 43 osd.211 pgs 49 osd.212 pgs 49 osd.213 pgs 49 osd.214 pgs 55 osd.215 pgs 43 osd.216 pgs 49 osd.217 pgs 49 osd.218 pgs 55 osd.219 pgs 62 osd.220 pgs 49 osd.221 pgs 55 osd.222 pgs 49 osd.223 pgs 55 osd.224 pgs 49 osd.225 pgs 55 osd.226 pgs 49 osd.227 pgs 55 osd.228 pgs 62 osd.229 pgs 62 osd.230 pgs 37 osd.231 pgs 55 osd.232 pgs 62 osd.233 pgs 55 osd.234 pgs 55 osd.235 pgs 62 osd.236 pgs 55 osd.237 pgs 62 osd.238 pgs 61 osd.239 pgs 37 osd.240 pgs 62 osd.241 pgs 62 osd.242 pgs 61 osd.243 pgs 62 osd.244 pgs 62 osd.245 pgs 62 osd.246 pgs 62 osd.247 pgs 55 osd.248 pgs 62 osd.249 pgs 49 osd.250 pgs 61 osd.251 pgs 49 osd.252 pgs 43 osd.253 pgs 55 osd.254 pgs 49 osd.255 pgs 55 osd.256 pgs 43 osd.257 pgs 55 osd.258 pgs 62 osd.259 pgs 61 osd.260 pgs 55 osd.261 pgs 43 osd.262 pgs 43 osd.263 pgs 49 osd.264 pgs 49 osd.265 pgs 62 osd.266 pgs 49 osd.267 pgs 62 osd.268 pgs 87 osd.269 pgs 86 osd.270 pgs 87 osd.271 pgs 86 osd.272 pgs 270 osd.273 pgs 271 osd.274 pgs 270 osd.275 pgs 270 osd.276 pgs 270 osd.277 pgs 270 osd.278 pgs 270 osd.279 pgs 270 osd.280 pgs 270 osd.281 pgs 270 osd.282 pgs 270 osd.283 pgs 271 osd.284 pgs 270 osd.285 pgs 271 osd.286 pgs 270 osd.287 pgs 270 osd.288 pgs 270 osd.289 pgs 270 osd.290 pgs 270 osd.291 pgs 270 osd.292 pgs 270 osd.293 pgs 270 osd.294 pgs 270 osd.295 pgs 270 osd.296 pgs 270 osd.297 pgs 270 osd.298 pgs 270 osd.299 pgs 270 osd.300 pgs 270 osd.301 pgs 270 osd.302 pgs 270 osd.303 pgs 270 osd.304 pgs 270 osd.305 pgs 270 osd.306 pgs 270 osd.307 pgs 270 osd.308 pgs 270 osd.309 pgs 270 osd.310 pgs 270 osd.311 pgs 270 osd.312 pgs 270 osd.313 pgs 270 osd.314 pgs 270 osd.315 pgs 270 osd.316 pgs 270 osd.317 pgs 270 osd.318 pgs 270 osd.319 pgs 270 osd.320 pgs 271 osd.321 pgs 271 osd.322 pgs 271 osd.323 pgs 271 osd.324 pgs 271 osd.325 pgs 270 osd.326 pgs 270 osd.327 pgs 270 osd.328 pgs 270 osd.329 pgs 270 osd.330 pgs 270 osd.331 pgs 270 osd.332 pgs 271 osd.333 pgs 270 osd.334 pgs 271 osd.335 pgs 270 osd.336 pgs 270 osd.337 pgs 271 osd.338 pgs 271 osd.339 pgs 270 osd.340 pgs 270 osd.341 pgs 270 osd.342 pgs 270 osd.343 pgs 270 osd.344 pgs 270 osd.345 pgs 271 osd.346 pgs 270 osd.347 pgs 271 osd.348 pgs 270 osd.349 pgs 270 osd.350 pgs 270 osd.351 pgs 270 osd.352 pgs 270 osd.353 pgs 270 osd.354 pgs 270 osd.355 pgs 270 osd.356 pgs 270 osd.357 pgs 270 osd.358 pgs 270 osd.359 pgs 270 osd.360 pgs 270 osd.361 pgs 270 osd.362 pgs 270 osd.363 pgs 270 osd.364 pgs 270 osd.365 pgs 270 osd.366 pgs 270 osd.367 pgs 270 osd.368 pgs 86 osd.369 pgs 86 osd.370 pgs 87 osd.371 pgs 270 osd.372 pgs 270 osd.373 pgs 270 osd.374 pgs 270 osd.375 pgs 270 osd.376 pgs 270 osd.377 pgs 270 osd.378 pgs 270 osd.379 pgs 270 osd.380 pgs 270 osd.381 pgs 270 osd.382 pgs 270 osd.383 pgs 270 osd.384 pgs 271 osd.385 pgs 270 osd.386 pgs 270 osd.387 pgs 270 osd.388 pgs 270 osd.389 pgs 270 osd.390 pgs 270 osd.391 pgs 270 osd.392 pgs 270 osd.393 pgs 270 osd.394 pgs 270 osd.395 pgs 270 osd.396 pgs 270 osd.397 pgs 270 osd.398 pgs 270 osd.399 pgs 270 osd.400 pgs 270 osd.401 pgs 270 osd.402 pgs 270 osd.403 pgs 270 osd.404 pgs 270 osd.405 pgs 270 osd.406 pgs 271 osd.407 pgs 270 osd.408 pgs 270 osd.409 pgs 270 osd.410 pgs 270 osd.411 pgs 270 osd.412 pgs 270 osd.413 pgs 270 osd.414 pgs 270 osd.415 pgs 270 osd.416 pgs 270 osd.417 pgs 270 osd.418 pgs 270 osd.419 pgs 87 osd.420 pgs 87 osd.421 pgs 86 osd.422 pgs 86 osd.423 pgs 86 osd.424 pgs 86 osd.425 pgs 86 osd.426 pgs 86 osd.427 pgs 86 osd.428 pgs 86 osd.429 pgs 86 osd.430 pgs 86 osd.431 pgs 86 osd.432 pgs 86 osd.433 pgs 86 osd.434 pgs 86 osd.435 pgs 58 osd.436 pgs 58 osd.437 pgs 58 osd.438 pgs 58 osd.439 pgs 58 osd.440 pgs 58 osd.441 pgs 58 David On 12/11/19 6:12 AM, Thomas Schneider wrote:
Hello David,
I'm experiencing issues with OSD balancing, too. My ceph cluster is running on release ceph version 14.2.4.1 (596a387fb278758406deabf997735a1f706660c9) nautilus (stable)
Would you be able to test (the latest code) on my OSDmap and verify if balancing would work? I have attached it to this email.
Regards Thomas