You set mds_beacon_grace ?
Yes, as I said. It seemed to have no effect or at least none that I
could see. The kick timeout seemed random after all. I even set it to
something ridiculous like 1800 and the MDS were still timed out.
Sometimes they got to 20M inodes, sometimes only to a few 100k. The ones
that got further often reported slow metadata operations, the less lucky
ones unhealthy MDS beacons. But none lasted for a full 1800s.
Yes, this optimization is having some struggles with
large cache sizes
:-(