Lowering the weight is what I ended up doing. But this isn't ideal since afterwards the balancer will remove too many PGs from the OSD since now it has a lower weight. So I'll have to put the weight back once the cluster recovers and the balancer goes back to its business. But in any case - this is conceptually challenging - that the upmap balancer won't help for the case one would perhaps need it the most - when recovering from some kind of a disaster. So one can be under the illusion that everything is fine - OSDs are all balanced, cluster is running smooth. Then some non-trivial failure happens, and we are back to a pre upmap situation - the balance is completely thrown off. Also for larger clusters the pre-upmap imbalance is worse - and is also harder to fix (due to the large number of OSDs). I've done some analysis of what the expected imbalance is given various factors of the cluster - but that's a longer story ... Thanks for the input - I was really wondering if I was missing something with upmap ... Andras On 4/2/21 8:12 AM, Janne Johansson wrote:
Den fre 2 apr. 2021 kl 11:23 skrev Dan van der Ster <dan@vanderster.com>:
Hi again,
Oops, I'd missed the part about some PGs being degraded, which prevents the balancer from continuing. any upmaps which are directing PGs *to* those toofull OSDs. Or maybe it will be enough to just reweight those OSDs to 0.9. I was also thinking this, in that case, just lower OSD weight on the toofull OSDs like us old pre-upmap admins do. ;) When all the dust has settled, move weight up again.