Philippe, I have a master branch version of the code to test.  The nautilus backport https://github.com/ceph/ceph/pull/31956 should be the same. Using your OSDMap, the code in master branch and some additional changes to osdmaptool I was able to balance your cluster.  The osdmaptool changes simulate the mgr active balancer behavior.  It never took no more than 0.13991 seconds to calculate more upmaps per round. And that's on a virtual machine used for development. It took 35 rounds with 10 maximum upmaps per crush rule set of pools per round.  With the default 1 minute sleeps inside the mgr it would take 35 minutes.  Obviously, recovery/backfill has to finish before the cluster settles into the new configuration.  It needed 397 additional upmaps and removed 8. Because all pools for a given crush rule are balanced together you can see that this is more balanced than Rich's configuration uising Luminous. This balancer code is subject to change before final release of the next Nautilus point release. Final layout: osd.0 pgs 146 osd.1 pgs 146 osd.2 pgs 146 osd.3 pgs 146 osd.4 pgs 146 osd.5 pgs 146 osd.6 pgs 146 osd.7 pgs 146 osd.8 pgs 146 osd.9 pgs 146 osd.10 pgs 146 osd.11 pgs 146 osd.12 pgs 74 osd.13 pgs 74 osd.14 pgs 73 osd.15 pgs 74 osd.16 pgs 74 osd.17 pgs 74 osd.18 pgs 73 osd.19 pgs 74 osd.20 pgs 73 osd.21 pgs 73 osd.22 pgs 74 osd.23 pgs 73 osd.24 pgs 73 osd.25 pgs 75 osd.26 pgs 74 osd.27 pgs 74 osd.28 pgs 73 osd.29 pgs 73 osd.30 pgs 73 osd.31 pgs 73 osd.32 pgs 74 osd.33 pgs 73 osd.34 pgs 73 osd.35 pgs 74 osd.36 pgs 74 osd.37 pgs 74 osd.38 pgs 74 osd.39 pgs 74 osd.40 pgs 73 osd.41 pgs 73 osd.42 pgs 73 osd.43 pgs 73 osd.44 pgs 74 osd.45 pgs 73 osd.46 pgs 73 osd.47 pgs 73 osd.48 pgs 73 osd.49 pgs 73 osd.50 pgs 73 osd.51 pgs 73 osd.52 pgs 75 osd.53 pgs 59 osd.54 pgs 74 osd.55 pgs 74 osd.56 pgs 74 osd.57 pgs 73 osd.58 pgs 74 osd.59 pgs 74 osd.60 pgs 74 osd.61 pgs 74 osd.62 pgs 73 osd.63 pgs 74 osd.64 pgs 73 osd.65 pgs 74 osd.66 pgs 74 osd.67 pgs 74 osd.68 pgs 73 osd.69 pgs 74 osd.70 pgs 73 osd.71 pgs 73 osd.72 pgs 73 osd.73 pgs 73 osd.74 pgs 73 osd.75 pgs 73 osd.76 pgs 73 osd.77 pgs 73 osd.78 pgs 73 osd.79 pgs 73 osd.80 pgs 73 osd.81 pgs 73 osd.82 pgs 73 osd.83 pgs 73 osd.84 pgs 73 osd.85 pgs 73 osd.86 pgs 73 osd.87 pgs 73 osd.88 pgs 73 osd.89 pgs 73 osd.90 pgs 73 osd.91 pgs 73 osd.92 pgs 73 osd.93 pgs 73 osd.94 pgs 73 osd.95 pgs 73 osd.96 pgs 73 osd.97 pgs 73 osd.98 pgs 73 osd.99 pgs 73 osd.100 pgs 146 osd.101 pgs 146 osd.102 pgs 146 osd.103 pgs 146 osd.104 pgs 146 osd.105 pgs 146 osd.106 pgs 146 osd.107 pgs 146 osd.108 pgs 146 osd.109 pgs 146 osd.110 pgs 146 osd.111 pgs 146 osd.112 pgs 73 osd.113 pgs 73 osd.114 pgs 73 osd.115 pgs 73 osd.116 pgs 73 osd.117 pgs 73 osd.118 pgs 73 osd.119 pgs 73 osd.120 pgs 73 osd.121 pgs 73 osd.122 pgs 73 osd.123 pgs 73 osd.124 pgs 73 osd.125 pgs 73 osd.126 pgs 73 osd.127 pgs 74 osd.128 pgs 73 osd.129 pgs 73 osd.130 pgs 73 osd.131 pgs 73 osd.132 pgs 73 osd.133 pgs 73 osd.134 pgs 73 osd.135 pgs 73 David On 12/10/19 9:59 PM, Philippe D'Anjou wrote:

Given I was told its an issue of too low PGs I am raising and testing this, although my SSDs which have about 150 each also are not well distributed. I attached my OSDmap, I'd appreciate if you could run your test on it like you did with the other guy, so I know if this will ever distribute equally or not..

If you're busy I understand that too, then ignore this.

Thanks in either case. Just have been dealing with this since months now and it gets frustrating.

best regards

Am Dienstag, 10. Dezember 2019, 03:53:17 OEZ hat David Zafman <dzafman@redhat.com> Folgendes geschrieben:

Please file a tracker with the symptom and examples. Please attach your OSDMap (ceph osd getmap > osdmap.bin).

Note that https://github.com/ceph/ceph/pull/31956 <https://github.com/ceph/ceph/pull/31956 >has the Nautilus version of improved upmap code.  It also changes osdmaptool to match the mgr behavior, so that one can observe the behavior of the upmap balancer offline.

Thanks

David

On 12/8/19 11:04 AM, Philippe D'Anjou wrote:

...

It's only getting worse after raising PGs now.

Anything between:  96   hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 97 KiB  13 GiB 4.2 TiB 53.62 0.76  54     up

and

 89   hdd 9.09470 1.00000 9.1 TiB 8.1 TiB 8.1 TiB 88 KiB  21 GiB 1001 GiB 89.25 1.27  87     up

How is that possible? I dont know how much more proof I need to present that there's a bug.

...

_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com