On Wed, Mar 31, 2021 at 6:46 PM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
Hello Yongseok,
On Wed, Mar 31, 2021 at 1:13 AM Yongseok Oh <yongseok.oh(a)linecorp.com> wrote:
>
...
A few things I
have analyzed
- Rejoining process consumes a considerable amount of time. That's a known issue.
(Sometimes respawning MDS happened. Increasing mds_heartbeat_grace doesn't help.)
Please turn up logging to:
debug_mds = 5
to get an idea what the MDS is doing when respawn occurs.
If it helps, here is a log with 2/5 from a recent failover which took
3.5 minutes:
https://termbin.com/b022
This is 14.2.11 with the optimized recall/cache tuning.
Indeed rejoin is always the longest step -- even with cephfs_metadata
on SSDs. These MDSs had the cache limit set to 8GB, and you can see
that the rejoining MDS needed 56GB while booting.
I haven't had a chance to test the rejoin/openfiletables optimizations
yet. (
https://github.com/ceph/ceph/pull/37383)
But I had understood that this is intended to decrease that rejoin
memory usage -- will it also speed things up?
-- dan