3 May
2021
3 May
'21
4:36 p.m.
Hi Team,
I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW
- Deployment Type: Active Standby
- Testing Mode: Failover of MDS Node
- Setup : Octopus (15.2.7)
- OS: centos 8.3
- hardware: HP
- Ram: 128 GB on each Node
- OSD: 2 ( 1 tb each)
- Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:*
We have observed that whenever an active MDS Node is down it takes around*
40 seconds* to activate the standby MDS Node.
on further checking the logs for the new-handover MDS Node we have seen
delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election
1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling
monitor election
2. 5 second delay in which newly elected Monitor is elected
1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new
leader, mons cephnode1,cephnode3 in quorum (ranks 0,2)
3. the addition beacon grace time for which the system waits before
which it enables standby MDS node activation. (approx delay of 19 seconds)
1. defaults : sudo ceph config get mon mds_beacon_grace
15.000000
2. sudo ceph config get mon mds_beacon_interval
5.000000
3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1
mon.cephnode2(a)1(leader).mds e776 no beacon from mds.0.771 (gid:
639443 addr: [v2:
10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state:
up:active)* since 18.7951*
4. *in Total it takes around 40 seconds to handover and activate passive
standby node. *
*Query:*
1. Can these variables be configured ? which we have tried,but are not
aware of the overall impact on the ceph cluster because of these changes
1. By tuning these values we could reach the minimum time of 12
seconds in which the active node comes up.
2. Values taken to get the said time :
1. *mon_election_timeout* (default 5) - configured as 1
2. *mon_lease*(default 5) - configured as 2
3. *mds_beacon_grace* (default 15) - configured as 5
4. *mds_beacon_interval* (default 5) - configured as 1
We need to tune this setup to get the failover duration as low as 5-7
seconds.
Please suggest/support and share your inputs, my setup is ready and already
we are testing with multiple scenarios so that we are able to achive min
failover duration.
--
~ Lokendra
www.inertiaspeaks.com
www.inertiagroups.com
skype: lokendrarathour