I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into?
Yes. Setting that lower than default can help avoid certain intermediate states that can result in a fairly-full cluster's OSDs going nearfull/backfillfull.
The documentation say about 'target_max_misplaced_ratio' "...the balancer will incrementally move a small fraction of unbalanced PGs in order to improve distribution. This fraction will not exceed a certain threshold that defaults to 5%...". does it mean that off all the PGs that has to be moved out/in of the node(s), maximum 5% will be moved at the same time?
Max 5% (I usually set it lower) of the misplaced-but-upmapped PGs will be un-upmapped and go into "backfill" or "backfill_wait". Since 50% is kind of a large number on large clusters, this means a lot of the PGs will be in backfill_wait. This isn't a huge problem in terms of movement, since other factors (like max_backfill when combined with wpq scheduler at least) will limit how many transfers are being done, but having 12 backfilling and 120 backfill_wait will just mean those in backfill_wait will not see scrubbing done to them while waiting, so I usually set the max_misplaced to a smaller number like 0.1% for larger clusters or pools with large number of PGs, which still leaves enough PGs that actually does move, but not too many lingering in _wait state so they can get their scrubs done as expected. The balancer runs every minute or so, so if the _wait queue dips, it will only take some 30-60 seconds for the balancer to make sure 0.1% is again heading for movement. -- May the most significant bit of your life be positive.