On Tue, Sep 29, 2020 at 12:34 PM Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
I think you found the answer!
When adding 100 new OSDs to the cluster, I increased both pg and pgp from 4096 to 16,384
Too much for your cluster, 4096 seems sufficient for a pool of size 10. You can still reduce it relatively cheaply while it hasn't been fully actuated yet Paul
********************************** [root@ceph1 ~]# ceph osd pool set ec82pool pg_num 16384 set pool 5 pg_num to 16384
[root@ceph1 ~]# ceph osd pool set ec82pool pgp_num 16384 set pool 5 pgp_num to 16384
**********************************
The pg number increased immediately as seen with "ceph -s"
But unknown to me, the pgp number did not increase immediately.
"ceph osd pool ls detail" shows that pgp is currently 11412
Each time we hit 5.000% misplaced, the pgp number increases by 1 or 2, this causes the % misplaced to increase again to ~5.1% ... which is why we thought the cluster was not re-balancing.
If I'd looked at the ceph.audit.log there are entries like this:
2020-09-23 01:13:11.564384 mon.ceph3b (mon.1) 50747 : audit [INF] from='mgr.90414409 10.1.0.80:0/7898' entity='mgr.ceph2' cmd=[{"prefix": "osd pool set", "pool": "ec82pool", "var": "pgp_num_actual", "val": "5076"}]: dispatch 2020-09-23 01:13:11.565598 mon.ceph1b (mon.0) 85947 : audit [INF] from='mgr.90414409 ' entity='mgr.ceph2' cmd=[{"prefix": "osd pool set", "pool": "ec82pool", "var": "pgp_num_actual", "val": "5076"}]: dispatch 2020-09-23 01:13:12.530584 mon.ceph1b (mon.0) 85949 : audit [INF] from='mgr.90414409 ' entity='mgr.ceph2' cmd='[{"prefix": "osd pool set", "pool": "ec82pool", "var": "pgp_num_actual", "val": "5076"}]': finished
Our assumption is that the pgp number will continue to increase till it reaches its set level, at which point the cluster will complete it's re-balance...
again, many thanks to you both for your help,
Jake
On 28/09/2020 17:35, Paul Emmerich wrote:
Hi,
5% misplaced is the default target ratio for misplaced PGs when any automated rebalancing happens, the sources for this are either the balancer or pg scaling. So I'd suspect that there's a PG change ongoing (either pg autoscaler or a manual change, both obey the target misplaced ratio). You can check this by running "ceph osd pool ls detail" and check for the value of pg target.
Also: Looks like you've set osd_scrub_during_recovery = false, this setting can be annoying on large erasure-coded setups on HDDs that see long recovery times. It's better to get IO priorities right; search mailing list for osd op queue cut off high.
Paul
-- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.