[ Ceph MDS MON Config Variables ] Failover Delay issue
Hi Team, I was setting up the ceph cluster with - Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second. T*est Case: Power-off any active MDS Node for failover to happen* *Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs: 1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election 2. 5 second delay in which newly elected Monitor is elected 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new leader, mons cephnode1,cephnode3 in quorum (ranks 0,2) 3. the addition beacon grace time for which the system waits before which it enables standby MDS node activation. (approx delay of 19 seconds) 1. defaults : sudo ceph config get mon mds_beacon_grace 15.000000 2. sudo ceph config get mon mds_beacon_interval 5.000000 3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1 mon.cephnode2@1(leader).mds e776 no beacon from mds.0.771 (gid: 639443 addr: [v2: 10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state: up:active)* since 18.7951* 4. *in Total it takes around 40 seconds to handover and activate passive standby node. * *Query:* 1. Can these variables be configured ? which we have tried,but are not aware of the overall impact on the ceph cluster because of these changes 1. By tuning these values we could reach the minimum time of 12 seconds in which the active node comes up. 2. Values taken to get the said time : 1. *mon_election_timeout* (default 5) - configured as 1 2. *mon_lease*(default 5) - configured as 2 3. *mds_beacon_grace* (default 15) - configured as 5 4. *mds_beacon_interval* (default 5) - configured as 1 We need to tune this setup to get the failover duration as low as 5-7 seconds. Please suggest/support and share your inputs, my setup is ready and already we are testing with multiple scenarios so that we are able to achive min failover duration. -- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour
hello perhaps you should have more than one MDS active. mds: cephfs:3 {0=cephfs-d=up:active,1=cephfs-e=up:active,2=cephfs- a=up:active} 1 up:standby-replay I got 3 active mds and one standby. I'm using rook in kubernetes for this setup. oau Le lundi 03 mai 2021 à 19:06 +0530, Lokendra Rathour a écrit :
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election 2. 5 second delay in which newly elected Monitor is elected 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new leader, mons cephnode1,cephnode3 in quorum (ranks 0,2) 3. the addition beacon grace time for which the system waits before which it enables standby MDS node activation. (approx delay of 19 seconds) 1. defaults : sudo ceph config get mon mds_beacon_grace 15.000000 2. sudo ceph config get mon mds_beacon_interval 5.000000 3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1 mon.cephnode2@1(leader).mds e776 no beacon from mds.0.771 (gid: 639443 addr: [v2: 10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state: up:active)* since 18.7951* 4. *in Total it takes around 40 seconds to handover and activate passive standby node. *
*Query:*
1. Can these variables be configured ? which we have tried,but are not aware of the overall impact on the ceph cluster because of these changes 1. By tuning these values we could reach the minimum time of 12 seconds in which the active node comes up. 2. Values taken to get the said time : 1. *mon_election_timeout* (default 5) - configured as 1 2. *mon_lease*(default 5) - configured as 2 3. *mds_beacon_grace* (default 15) - configured as 5 4. *mds_beacon_interval* (default 5) - configured as 1
We need to tune this setup to get the failover duration as low as 5-7 seconds.
Please suggest/support and share your inputs, my setup is ready and already we are testing with multiple scenarios so that we are able to achive min failover duration.
Also there's a difference between 'standby-replay' (hot standby) and just 'standby'. We use CephFS for a couple of years now with standby-replay and the failover takes a couple of seconds max, depending on the current load. Have you tried to enable the standby-replay config and tested the failover? ceph fs set cephfs allow_standby_replay true Zitat von Olivier AUDRY <oaudry@predical.fr>:
hello
perhaps you should have more than one MDS active.
mds: cephfs:3 {0=cephfs-d=up:active,1=cephfs-e=up:active,2=cephfs- a=up:active} 1 up:standby-replay
I got 3 active mds and one standby.
I'm using rook in kubernetes for this setup.
oau
Le lundi 03 mai 2021 à 19:06 +0530, Lokendra Rathour a écrit :
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election 2. 5 second delay in which newly elected Monitor is elected 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new leader, mons cephnode1,cephnode3 in quorum (ranks 0,2) 3. the addition beacon grace time for which the system waits before which it enables standby MDS node activation. (approx delay of 19 seconds) 1. defaults : sudo ceph config get mon mds_beacon_grace 15.000000 2. sudo ceph config get mon mds_beacon_interval 5.000000 3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1 mon.cephnode2@1(leader).mds e776 no beacon from mds.0.771 (gid: 639443 addr: [v2: 10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state: up:active)* since 18.7951* 4. *in Total it takes around 40 seconds to handover and activate passive standby node. *
*Query:*
1. Can these variables be configured ? which we have tried,but are not aware of the overall impact on the ceph cluster because of these changes 1. By tuning these values we could reach the minimum time of 12 seconds in which the active node comes up. 2. Values taken to get the said time : 1. *mon_election_timeout* (default 5) - configured as 1 2. *mon_lease*(default 5) - configured as 2 3. *mds_beacon_grace* (default 15) - configured as 5 4. *mds_beacon_interval* (default 5) - configured as 1
We need to tune this setup to get the failover duration as low as 5-7 seconds.
Please suggest/support and share your inputs, my setup is ready and already we are testing with multiple scenarios so that we are able to achive min failover duration.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Eugen, Thankyou for the response. Yes we tried ceph-standby-replay but could not see much difference in the handover time. It was comming as 35 to 40 seconds in either case. Did you also changed these variables (as mentioned above) along with the hot-standby ? Couple of seconds is something we wish to achieve using cephfs. Best Regards, Lokendra On Mon, 3 May 2021, 19:27 Eugen Block, <eblock@nde.ag> wrote:
Also there's a difference between 'standby-replay' (hot standby) and just 'standby'. We use CephFS for a couple of years now with standby-replay and the failover takes a couple of seconds max, depending on the current load. Have you tried to enable the standby-replay config and tested the failover?
ceph fs set cephfs allow_standby_replay true
Zitat von Olivier AUDRY <oaudry@predical.fr>:
hello
perhaps you should have more than one MDS active.
mds: cephfs:3 {0=cephfs-d=up:active,1=cephfs-e=up:active,2=cephfs- a=up:active} 1 up:standby-replay
I got 3 active mds and one standby.
I'm using rook in kubernetes for this setup.
oau
Le lundi 03 mai 2021 à 19:06 +0530, Lokendra Rathour a écrit :
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election 2. 5 second delay in which newly elected Monitor is elected 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new leader, mons cephnode1,cephnode3 in quorum (ranks 0,2) 3. the addition beacon grace time for which the system waits before which it enables standby MDS node activation. (approx delay of 19 seconds) 1. defaults : sudo ceph config get mon mds_beacon_grace 15.000000 2. sudo ceph config get mon mds_beacon_interval 5.000000 3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1 mon.cephnode2@1(leader).mds e776 no beacon from mds.0.771 (gid: 639443 addr: [v2: 10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state: up:active)* since 18.7951* 4. *in Total it takes around 40 seconds to handover and activate passive standby node. *
*Query:*
1. Can these variables be configured ? which we have tried,but are not aware of the overall impact on the ceph cluster because of these changes 1. By tuning these values we could reach the minimum time of 12 seconds in which the active node comes up. 2. Values taken to get the said time : 1. *mon_election_timeout* (default 5) - configured as 1 2. *mon_lease*(default 5) - configured as 2 3. *mds_beacon_grace* (default 15) - configured as 5 4. *mds_beacon_interval* (default 5) - configured as 1
We need to tune this setup to get the failover duration as low as 5-7 seconds.
Please suggest/support and share your inputs, my setup is ready and already we are testing with multiple scenarios so that we are able to achive min failover duration.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
Yes we tried ceph-standby-replay but could not see much difference in the handover time. It was comming as 35 to 40 seconds in either case. Did you also changed these variables (as mentioned above) along with the hot-standby ?
no, we barely differ from the default configs and haven't changed much. But we're still running Nautilus so I can't really tell if Octopus makes a difference. Zitat von Lokendra Rathour <lokendrarathour@gmail.com>:
Hello Eugen,
Thankyou for the response. Yes we tried ceph-standby-replay but could not see much difference in the handover time. It was comming as 35 to 40 seconds in either case. Did you also changed these variables (as mentioned above) along with the hot-standby ? Couple of seconds is something we wish to achieve using cephfs.
Best Regards, Lokendra
On Mon, 3 May 2021, 19:27 Eugen Block, <eblock@nde.ag> wrote:
Also there's a difference between 'standby-replay' (hot standby) and just 'standby'. We use CephFS for a couple of years now with standby-replay and the failover takes a couple of seconds max, depending on the current load. Have you tried to enable the standby-replay config and tested the failover?
ceph fs set cephfs allow_standby_replay true
Zitat von Olivier AUDRY <oaudry@predical.fr>:
hello
perhaps you should have more than one MDS active.
mds: cephfs:3 {0=cephfs-d=up:active,1=cephfs-e=up:active,2=cephfs- a=up:active} 1 up:standby-replay
I got 3 active mds and one standby.
I'm using rook in kubernetes for this setup.
oau
Le lundi 03 mai 2021 à 19:06 +0530, Lokendra Rathour a écrit :
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election 2. 5 second delay in which newly elected Monitor is elected 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new leader, mons cephnode1,cephnode3 in quorum (ranks 0,2) 3. the addition beacon grace time for which the system waits before which it enables standby MDS node activation. (approx delay of 19 seconds) 1. defaults : sudo ceph config get mon mds_beacon_grace 15.000000 2. sudo ceph config get mon mds_beacon_interval 5.000000 3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1 mon.cephnode2@1(leader).mds e776 no beacon from mds.0.771 (gid: 639443 addr: [v2: 10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state: up:active)* since 18.7951* 4. *in Total it takes around 40 seconds to handover and activate passive standby node. *
*Query:*
1. Can these variables be configured ? which we have tried,but are not aware of the overall impact on the ceph cluster because of these changes 1. By tuning these values we could reach the minimum time of 12 seconds in which the active node comes up. 2. Values taken to get the said time : 1. *mon_election_timeout* (default 5) - configured as 1 2. *mon_lease*(default 5) - configured as 2 3. *mds_beacon_grace* (default 15) - configured as 5 4. *mds_beacon_interval* (default 5) - configured as 1
We need to tune this setup to get the failover duration as low as 5-7 seconds.
Please suggest/support and share your inputs, my setup is ready and already we are testing with multiple scenarios so that we are able to achive min failover duration.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ok, Will try with nautilus as well. But we are really configuring too many variables to achieve 10 seconds of failover time. Is it possible for you to share the setup details. Like we are using 2 node ceph cluster in health ok (configured replication factor and related variables) Hardware is HP, and to mount the fs on the client machine we are using native file system driver. During mount we are passing IP of both the nodes of ceph. Please share further details or related details of your setup if possible which might help us with the failover delay issue. Thanks again for your inputs and your response. We also need to achieve couple of seconds failover delay. Best regards, Lokendra On Mon, 3 May 2021, 20:47 Eugen Block, <eblock@nde.ag> wrote:
Hi,
Yes we tried ceph-standby-replay but could not see much difference in the handover time. It was comming as 35 to 40 seconds in either case. Did you also changed these variables (as mentioned above) along with the hot-standby ?
no, we barely differ from the default configs and haven't changed much. But we're still running Nautilus so I can't really tell if Octopus makes a difference.
Zitat von Lokendra Rathour <lokendrarathour@gmail.com>:
Hello Eugen,
Thankyou for the response. Yes we tried ceph-standby-replay but could not see much difference in the handover time. It was comming as 35 to 40 seconds in either case. Did you also changed these variables (as mentioned above) along with the hot-standby ? Couple of seconds is something we wish to achieve using cephfs.
Best Regards, Lokendra
On Mon, 3 May 2021, 19:27 Eugen Block, <eblock@nde.ag> wrote:
Also there's a difference between 'standby-replay' (hot standby) and just 'standby'. We use CephFS for a couple of years now with standby-replay and the failover takes a couple of seconds max, depending on the current load. Have you tried to enable the standby-replay config and tested the failover?
ceph fs set cephfs allow_standby_replay true
Zitat von Olivier AUDRY <oaudry@predical.fr>:
hello
perhaps you should have more than one MDS active.
mds: cephfs:3 {0=cephfs-d=up:active,1=cephfs-e=up:active,2=cephfs- a=up:active} 1 up:standby-replay
I got 3 active mds and one standby.
I'm using rook in kubernetes for this setup.
oau
Le lundi 03 mai 2021 à 19:06 +0530, Lokendra Rathour a écrit :
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election 2. 5 second delay in which newly elected Monitor is elected 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new leader, mons cephnode1,cephnode3 in quorum (ranks 0,2) 3. the addition beacon grace time for which the system waits before which it enables standby MDS node activation. (approx delay of 19 seconds) 1. defaults : sudo ceph config get mon mds_beacon_grace 15.000000 2. sudo ceph config get mon mds_beacon_interval 5.000000 3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1 mon.cephnode2@1(leader).mds e776 no beacon from mds.0.771 (gid: 639443 addr: [v2: 10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state: up:active)* since 18.7951* 4. *in Total it takes around 40 seconds to handover and activate passive standby node. *
*Query:*
1. Can these variables be configured ? which we have tried,but are not aware of the overall impact on the ceph cluster because of these changes 1. By tuning these values we could reach the minimum time of 12 seconds in which the active node comes up. 2. Values taken to get the said time : 1. *mon_election_timeout* (default 5) - configured as 1 2. *mon_lease*(default 5) - configured as 2 3. *mds_beacon_grace* (default 15) - configured as 5 4. *mds_beacon_interval* (default 5) - configured as 1
We need to tune this setup to get the failover duration as low as 5-7 seconds.
Please suggest/support and share your inputs, my setup is ready and already we are testing with multiple scenarios so that we are able to achive min failover duration.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, May 3, 2021 at 6:36 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election
In the process of killing the active MDS, are you also killing a monitor? -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Following up on this and other comments, there are 2 different time delays. One (1) is the time it takes from killing an MDS until a stand-by is made an active rank, and (2) the time it takes for the new active rank to restore all client sessions. My experience is that (1) takes close to 0 seconds while (2) can take between 20-30 seconds depending on how busy the clients are; the MDS will go through various states before reaching active. We usually have ca. 1600 client connections to our FS. With fewer clients, MDS fail-over is practically instantaneous. We are using latest mimic. From what you write, you seem to have a 40 seconds window for (1), which points to a problem different to MON config values. This is supported by your description including a MON election (??? this should never happen). Do you have have services co-located? Which of the times (1) or (2) are you referring to? How many FS clients do you have? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Patrick Donnelly <pdonnell@redhat.com> Sent: 03 May 2021 17:19:37 To: Lokendra Rathour Cc: Ceph Development; dev; ceph-users Subject: [ceph-users] Re: [ Ceph MDS MON Config Variables ] Failover Delay issue On Mon, May 3, 2021 at 6:36 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election
In the process of killing the active MDS, are you also killing a monitor? -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Frank, Thanks for your inputs. *Responding to your Queries , Kindly refer below:* - *Do you have services co-located? * - [loke] : Yes they are colocated: - Cephnode1 : MDS,MGR,MON,RGW,OSD,MDS - Cephnode2: MDS,MGR,MON,RGW,OSD,MDS - Cephnode3: MON - Which of the times (1) or (2) are you referring to? - For you part One (1) : we can say it like by counting the time since the I/O is stopped till the I/O is resumed, which includes - call for new mon election - election of mon leader - calling of MDS standby acting by new Mon Leader - resuming I/O stuck threads - and other internal process.(i am only point what i could read from logs) - *How many FS Clients do you have?* - we are testing with only one client mounting using native fs driver at the moment, where we pass both IP Address of both the MDS Daemon(in our case both the Ceph Nodes) using following method: - sudo mount -t ceph 10.0.4.10,10.0.4.11:6789:/volumes/path/ /mnt/cephconf -o name=foo,secret=AQAus49gdCHvIxAAB89BcDYqYSqJ8yOJBg5grw== *one input*:if we only shut-down MDS Active Daemon, we only get 4-7 Seconds, i.e if we are not rebooting the physical node but only the service MDS. When we reboot Physical node , Cephnode1 or Cephnode2 ( Mon,Mgr,RGW,OSD also gets rebooted along with MDS) we realizing around 40 seconds. Best Regards, Lokendra On Mon, May 3, 2021 at 10:30 PM Frank Schilder <frans@dtu.dk> wrote:
Following up on this and other comments, there are 2 different time delays. One (1) is the time it takes from killing an MDS until a stand-by is made an active rank, and (2) the time it takes for the new active rank to restore all client sessions. My experience is that (1) takes close to 0 seconds while (2) can take between 20-30 seconds depending on how busy the clients are; the MDS will go through various states before reaching active. We usually have ca. 1600 client connections to our FS. With fewer clients, MDS fail-over is practically instantaneous. We are using latest mimic.
From what you write, you seem to have a 40 seconds window for (1), which points to a problem different to MON config values. This is supported by your description including a MON election (??? this should never happen). Do you have have services co-located? Which of the times (1) or (2) are you referring to? How many FS clients do you have?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Patrick Donnelly <pdonnell@redhat.com> Sent: 03 May 2021 17:19:37 To: Lokendra Rathour Cc: Ceph Development; dev; ceph-users Subject: [ceph-users] Re: [ Ceph MDS MON Config Variables ] Failover Delay issue
On Mon, May 3, 2021 at 6:36 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes
around*
40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election
In the process of killing the active MDS, are you also killing a monitor?
-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour
I wouldn't recommend a colocated MDS in a production environment. Zitat von Lokendra Rathour <lokendrarathour@gmail.com>:
Hello Frank, Thanks for your inputs.
*Responding to your Queries , Kindly refer below:*
- *Do you have services co-located? * - [loke] : Yes they are colocated: - Cephnode1 : MDS,MGR,MON,RGW,OSD,MDS - Cephnode2: MDS,MGR,MON,RGW,OSD,MDS - Cephnode3: MON - Which of the times (1) or (2) are you referring to? - For you part One (1) : we can say it like by counting the time since the I/O is stopped till the I/O is resumed, which includes
- call for new mon election - election of mon leader - calling of MDS standby acting by new Mon Leader - resuming I/O stuck threads - and other internal process.(i am only point what i could read from logs)
- *How many FS Clients do you have?* - we are testing with only one client mounting using native fs driver at the moment, where we pass both IP Address of both the MDS Daemon(in our case both the Ceph Nodes) using following method: - sudo mount -t ceph 10.0.4.10,10.0.4.11:6789:/volumes/path/ /mnt/cephconf -o name=foo,secret=AQAus49gdCHvIxAAB89BcDYqYSqJ8yOJBg5grw==
*one input*:if we only shut-down MDS Active Daemon, we only get 4-7 Seconds, i.e if we are not rebooting the physical node but only the service MDS. When we reboot Physical node , Cephnode1 or Cephnode2 ( Mon,Mgr,RGW,OSD also gets rebooted along with MDS) we realizing around 40 seconds.
Best Regards, Lokendra
On Mon, May 3, 2021 at 10:30 PM Frank Schilder <frans@dtu.dk> wrote:
Following up on this and other comments, there are 2 different time delays. One (1) is the time it takes from killing an MDS until a stand-by is made an active rank, and (2) the time it takes for the new active rank to restore all client sessions. My experience is that (1) takes close to 0 seconds while (2) can take between 20-30 seconds depending on how busy the clients are; the MDS will go through various states before reaching active. We usually have ca. 1600 client connections to our FS. With fewer clients, MDS fail-over is practically instantaneous. We are using latest mimic.
From what you write, you seem to have a 40 seconds window for (1), which points to a problem different to MON config values. This is supported by your description including a MON election (??? this should never happen). Do you have have services co-located? Which of the times (1) or (2) are you referring to? How many FS clients do you have?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Patrick Donnelly <pdonnell@redhat.com> Sent: 03 May 2021 17:19:37 To: Lokendra Rathour Cc: Ceph Development; dev; ceph-users Subject: [ceph-users] Re: [ Ceph MDS MON Config Variables ] Failover Delay issue
On Mon, May 3, 2021 at 6:36 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes
around*
40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election
In the process of killing the active MDS, are you also killing a monitor?
-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I concur, having this heavily collocated set-up will not perform any better than you observe. Do you really have 2 MDS daemons per host? I just saw hat you have only 2 disks, probably 1 per node. In this set-up, you cannot really expect good fail-over times due to the amount of simultaneous fails that need to be handled: - MON fail - OSD fail - MDS fail - MGR fail - (all!) PGs become degraded - FS data and meta data are on the same disks, so replaying journals, data IO and meta data IO all go to the same drive(s) There are plenty of bottle-necks in this set-up that render this test highly unrealistic. You should try to get more production ready hardware, a 2-disk ceph cluster isn't. I wouldn't waste time trying to adjust configs to a small test case, these config changes will not do any good for proper production systems. The set-up you have is good for learning to administrate ceph, it is not providing a point for comparison with a production system and will have heavily degraded performance. Ceph requires a not exactly small minimum size before it starts working well. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: 03 May 2021 20:53:51 To: ceph-users@ceph.io Subject: [ceph-users] Re: [ Ceph MDS MON Config Variables ] Failover Delay issue I wouldn't recommend a colocated MDS in a production environment. Zitat von Lokendra Rathour <lokendrarathour@gmail.com>:
Hello Frank, Thanks for your inputs.
*Responding to your Queries , Kindly refer below:*
- *Do you have services co-located? * - [loke] : Yes they are colocated: - Cephnode1 : MDS,MGR,MON,RGW,OSD,MDS - Cephnode2: MDS,MGR,MON,RGW,OSD,MDS - Cephnode3: MON - Which of the times (1) or (2) are you referring to? - For you part One (1) : we can say it like by counting the time since the I/O is stopped till the I/O is resumed, which includes
- call for new mon election - election of mon leader - calling of MDS standby acting by new Mon Leader - resuming I/O stuck threads - and other internal process.(i am only point what i could read from logs)
- *How many FS Clients do you have?* - we are testing with only one client mounting using native fs driver at the moment, where we pass both IP Address of both the MDS Daemon(in our case both the Ceph Nodes) using following method: - sudo mount -t ceph 10.0.4.10,10.0.4.11:6789:/volumes/path/ /mnt/cephconf -o name=foo,secret=AQAus49gdCHvIxAAB89BcDYqYSqJ8yOJBg5grw==
*one input*:if we only shut-down MDS Active Daemon, we only get 4-7 Seconds, i.e if we are not rebooting the physical node but only the service MDS. When we reboot Physical node , Cephnode1 or Cephnode2 ( Mon,Mgr,RGW,OSD also gets rebooted along with MDS) we realizing around 40 seconds.
Best Regards, Lokendra
On Mon, May 3, 2021 at 10:30 PM Frank Schilder <frans@dtu.dk> wrote:
Following up on this and other comments, there are 2 different time delays. One (1) is the time it takes from killing an MDS until a stand-by is made an active rank, and (2) the time it takes for the new active rank to restore all client sessions. My experience is that (1) takes close to 0 seconds while (2) can take between 20-30 seconds depending on how busy the clients are; the MDS will go through various states before reaching active. We usually have ca. 1600 client connections to our FS. With fewer clients, MDS fail-over is practically instantaneous. We are using latest mimic.
From what you write, you seem to have a 40 seconds window for (1), which points to a problem different to MON config values. This is supported by your description including a MON election (??? this should never happen). Do you have have services co-located? Which of the times (1) or (2) are you referring to? How many FS clients do you have?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Patrick Donnelly <pdonnell@redhat.com> Sent: 03 May 2021 17:19:37 To: Lokendra Rathour Cc: Ceph Development; dev; ceph-users Subject: [ceph-users] Re: [ Ceph MDS MON Config Variables ] Failover Delay issue
On Mon, May 3, 2021 at 6:36 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes
around*
40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election
In the process of killing the active MDS, are you also killing a monitor?
-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes Patric, In the process of killing MDS we are also *killing Monitor along with OSD,Mgr and RGW*. we are performing Poweroff/Reboot the complete node (with MDS,Mon,RGW,OSD,Mgr daemon). Cluster: 2 Nodes with MDS|Mon|RGW|OSD each and third node with 1 Mon. Note : when I am only stopping the MDS service it takes 4-7 Seconds to activate and resume the standy MDS Node. Thanks for your inputs. Best Regards, Lokendra On Mon, May 3, 2021 at 8:50 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Mon, May 3, 2021 at 6:36 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Team, I was setting up the ceph cluster with
- Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second.
T*est Case: Power-off any active MDS Node for failover to happen*
*Observation:* We have observed that whenever an active MDS Node is down it takes
around*
40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs:
1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election
In the process of killing the active MDS, are you also killing a monitor?
-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour
participants (5)
-
Eugen Block
-
Frank Schilder
-
Lokendra Rathour
-
Olivier AUDRY
-
Patrick Donnelly