Hi Andras, In 14.2.8 (the latest) version of Nautilus, each pool is balanced individually. Also, the latest version balances within 5 PGs by default. That could be a spread of 10, 5 below the mean and 5 above. I see a spread of 13 in your results. So either you are running an earlier Nautilus version or the mgr hasn't finished balancing. If you are running an earlier version, it would balance pools that use the same crush rule together. I'm working on a hybrid approach that will balance pools by rule, but pools will be grouped with a configurable PG count limit. Currently, we don't balance based on size. I don't think we know how much space a given PG uses. Unless Bluestore can or does efficiently track the space utilization, we can't reasonably balance based on size. David On 3/16/20 1:29 PM, Andras Pataki wrote:
I've been trying the upmap balancer on a new Nautilus cluster. We three main pools, a triple replicated pool (id:1) and two 6+3 erasure coded pools (id: 4 and 5). The balancer does a very nice job on the triple replicated pool, but does something strange on the EC pools. Here is a sample of the PG counts on OSDs (each line) per pool (columns):
OSD 1 2 3 4 5 ALL
0 25 - - 80 34 139 1 26 - - 74 42 142 2 25 - - 74 42 141 3 26 - - 75 41 142 4 25 - - 83 31 139 5 26 - - 80 35 141 6 26 - - 79 36 141 7 26 - - 72 44 142 8 25 - - 74 42 141 9 26 - - 78 38 142 10 26 - - 70 46 142 11 25 - - 78 36 139 12 26 - - 74 41 141 13 26 - - 78 37 141 14 26 - - 76 40 142 15 25 - - 82 33 140 16 26 - - 77 37 140 17 26 - - 73 43 142 18 26 - - 77 37 140 19 26 - - 79 35 140 20 26 - - 74 42 142 21 26 - - 78 36 140 22 26 - - 79 35 140 23 26 - - 81 34 141 24 26 - - 77 38 141 25 26 - - 79 35 140
Pool #1 has 25 or 26 PGs on each OSD extremely consistently. But it looks like pool #4 and #5 have a lot of variance, but their *sum* seems to be well balanced (and hence the total PGs per OSD is also very consistent). As if the balancer was balancing the two pools together as one. Both pools are EC with 6+3 profile, and use the same crush rule. Could that be the reason? If not - any ideas?
Also - a separate balancer related question: is there a way to have the balancer balace the size on each OSD per pool as opposed to the number of PGs? If not currently - is it something hard to implement?
Andras
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io