Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG
Andrew Walker-Brown <andrew_jbrown@hotmail.com> 6/15/2021 10:26 AM
With an unstable link/port you could see the issues you describe. Ping doesn’t have the packet rate for you to necessarily have a packet in
We also run with Dell VLT switches (40 GB) everything is active/active, so multiple paths as Andrew describes in his config Our config allows us: bring down one of the switches for upgrades bring down an iscsi gatway for patching all the while at least one path is up and servicing Thanks Joe transit at exactly the same time as the port fails temporarily. Iperf on the other hand could certainly show the issue, higher packet rate and more likely to have packets in flight at the time of a link fail...combined with packet loss/retries gives poor throughput. Depending on what you want to happen, there are a number of tuning options both on the switches and Linux. If you want the LAG to be down if any link fails, the you should be able to config this on the switches and/or Linux (minimum number of links = 2 if you have 2 links in the lag). You can also tune the link monitoring, how frequently the links are checked (e.g. miimon) etc. Bringing this value down from the default of 100ms may allow you to detect a link failure more quickly. But you then run into the chance if detecting a transient failure that wouldn’t have caused any issues....and the LAG becoming more unstable. Flapping/unstable links are the worst kind of situation. Ideally you’d pick that up quickly from monitoring/alerts and either fix immediately or take the link down until you can fix it. I run 2x10G from my hosts into separate switches (Dell S series – VLT between switches). Pulling a single interface has no impact on Ceph, any packet loss is tiny and we’re not exceeding 10G bandwidth per host. If you’re running 1G links and the LAG is already busy, a link failure could be causing slow writes to the host, just down to congestion...which then starts to impact the wider cluster based on how Ceph works. Just caveating the above with - I’m relatively new to Ceph myself.... Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: huxiaoyu@horebdata.cn<mailto:huxiaoyu@horebdata.cn> Sent: 15 June 2021 17:52 To: Serkan Çoban<mailto:cobanserkan@gmail.com> Cc: ceph-users<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2 MC-LAG When i pull out the cable, then the bond is working properly. Does it mean that the port is somehow flapping? Ping can still work, but the iperf test yields very low results. huxiaoyu@horebdata.cn From: Serkan Çoban Date: 2021-06-15 18:47 To: huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG Do you observe the same behaviour when you pull a cable? Maybe a flapping port might cause this kind of behaviour, other than that you should't see any network disconnects. Are you sure about LACP configuration, what is the output of 'cat /proc/net/bonding/bond0' On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Cephers,
I encountered the following networking issue several times, and i
wonder whether there is a solution for networking HA solution.
We build ceph using L2 multi chassis link aggregation group (MC-LAG )
to provide switch redundancy. On each host, we use 802.3ad, LACP
mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate.
Did i miss something important here? and how to really achieve networking HA in Ceph cluster?
best regards,
Samuel
huxiaoyu@horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Andrew Walker-Brown <andrew_jbrown@hotmail.com> 6/15/2021 10:26 AM
With an unstable link/port you could see the issues you describe. Ping doesn’t have the packet rate for you to necessarily have a packet in
Is it true that MC-LAG and 803.2ad, by its default, are working on active-active. What else should i take care to ensure fault tolerance when one path is bad? best regards, samuel huxiaoyu@horebdata.cn From: Joe Comeau Date: 2021-06-15 23:44 To: ceph-users@ceph.io Subject: [ceph-users] Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG We also run with Dell VLT switches (40 GB) everything is active/active, so multiple paths as Andrew describes in his config Our config allows us: bring down one of the switches for upgrades bring down an iscsi gatway for patching all the while at least one path is up and servicing Thanks Joe transit at exactly the same time as the port fails temporarily. Iperf on the other hand could certainly show the issue, higher packet rate and more likely to have packets in flight at the time of a link fail...combined with packet loss/retries gives poor throughput. Depending on what you want to happen, there are a number of tuning options both on the switches and Linux. If you want the LAG to be down if any link fails, the you should be able to config this on the switches and/or Linux (minimum number of links = 2 if you have 2 links in the lag). You can also tune the link monitoring, how frequently the links are checked (e.g. miimon) etc. Bringing this value down from the default of 100ms may allow you to detect a link failure more quickly. But you then run into the chance if detecting a transient failure that wouldn’t have caused any issues....and the LAG becoming more unstable. Flapping/unstable links are the worst kind of situation. Ideally you’d pick that up quickly from monitoring/alerts and either fix immediately or take the link down until you can fix it. I run 2x10G from my hosts into separate switches (Dell S series – VLT between switches). Pulling a single interface has no impact on Ceph, any packet loss is tiny and we’re not exceeding 10G bandwidth per host. If you’re running 1G links and the LAG is already busy, a link failure could be causing slow writes to the host, just down to congestion...which then starts to impact the wider cluster based on how Ceph works. Just caveating the above with - I’m relatively new to Ceph myself.... Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: huxiaoyu@horebdata.cn<mailto:huxiaoyu@horebdata.cn> Sent: 15 June 2021 17:52 To: Serkan Çoban<mailto:cobanserkan@gmail.com> Cc: ceph-users<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2 MC-LAG When i pull out the cable, then the bond is working properly. Does it mean that the port is somehow flapping? Ping can still work, but the iperf test yields very low results. huxiaoyu@horebdata.cn From: Serkan Çoban Date: 2021-06-15 18:47 To: huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG Do you observe the same behaviour when you pull a cable? Maybe a flapping port might cause this kind of behaviour, other than that you should't see any network disconnects. Are you sure about LACP configuration, what is the output of 'cat /proc/net/bonding/bond0' On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Cephers,
I encountered the following networking issue several times, and i
wonder whether there is a solution for networking HA solution.
We build ceph using L2 multi chassis link aggregation group (MC-LAG )
to provide switch redundancy. On each host, we use 802.3ad, LACP
mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate.
Did i miss something important here? and how to really achieve networking HA in Ceph cluster?
best regards,
Samuel
huxiaoyu@horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Andrew Walker-Brown <andrew_jbrown@hotmail.com> 6/15/2021 10:26 AM
With an unstable link/port you could see the issues you describe. Ping doesn’t have the packet rate for you to necessarily have a packet in
Depends on when you configure the switch port. For dell : Interface Ethernet 1/1/20 No switchport Channel-group 10 mode active ! ‘Mode active’ set it as a dynamic lacp lag. Otherwise it would be ‘mode static’ With active mode, you then have a transmit hashing policy, usually set globally. On Linux the bond would be set as ‘bond-mode 802.3ad’ and then ‘bond-xmit-hash-policy layer3+4’ - or whatever hashing policy you want. Sent from my iPhone On 16 Jun 2021, at 08:57, huxiaoyu@horebdata.cn wrote: Is it true that MC-LAG and 803.2ad, by its default, are working on active-active. What else should i take care to ensure fault tolerance when one path is bad? best regards, samuel huxiaoyu@horebdata.cn From: Joe Comeau Date: 2021-06-15 23:44 To: ceph-users@ceph.io Subject: [ceph-users] Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG We also run with Dell VLT switches (40 GB) everything is active/active, so multiple paths as Andrew describes in his config Our config allows us: bring down one of the switches for upgrades bring down an iscsi gatway for patching all the while at least one path is up and servicing Thanks Joe transit at exactly the same time as the port fails temporarily. Iperf on the other hand could certainly show the issue, higher packet rate and more likely to have packets in flight at the time of a link fail...combined with packet loss/retries gives poor throughput. Depending on what you want to happen, there are a number of tuning options both on the switches and Linux. If you want the LAG to be down if any link fails, the you should be able to config this on the switches and/or Linux (minimum number of links = 2 if you have 2 links in the lag). You can also tune the link monitoring, how frequently the links are checked (e.g. miimon) etc. Bringing this value down from the default of 100ms may allow you to detect a link failure more quickly. But you then run into the chance if detecting a transient failure that wouldn’t have caused any issues....and the LAG becoming more unstable. Flapping/unstable links are the worst kind of situation. Ideally you’d pick that up quickly from monitoring/alerts and either fix immediately or take the link down until you can fix it. I run 2x10G from my hosts into separate switches (Dell S series – VLT between switches). Pulling a single interface has no impact on Ceph, any packet loss is tiny and we’re not exceeding 10G bandwidth per host. If you’re running 1G links and the LAG is already busy, a link failure could be causing slow writes to the host, just down to congestion...which then starts to impact the wider cluster based on how Ceph works. Just caveating the above with - I’m relatively new to Ceph myself.... Sent from Mail<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986&data=04%7C01%7C%7Cf6c3c985114b423fb75e08d9309c5248%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637594270222669937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZrCWPvW06jX69X8PWR0S1tBxhQlzKPQnix4wzBMoN4s%3D&reserved=0> for Windows 10 From: huxiaoyu@horebdata.cn<mailto:huxiaoyu@horebdata.cn> Sent: 15 June 2021 17:52 To: Serkan Çoban<mailto:cobanserkan@gmail.com> Cc: ceph-users<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2 MC-LAG When i pull out the cable, then the bond is working properly. Does it mean that the port is somehow flapping? Ping can still work, but the iperf test yields very low results. huxiaoyu@horebdata.cn From: Serkan Çoban Date: 2021-06-15 18:47 To: huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG Do you observe the same behaviour when you pull a cable? Maybe a flapping port might cause this kind of behaviour, other than that you should't see any network disconnects. Are you sure about LACP configuration, what is the output of 'cat /proc/net/bonding/bond0' On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Cephers,
I encountered the following networking issue several times, and i
wonder whether there is a solution for networking HA solution.
We build ceph using L2 multi chassis link aggregation group (MC-LAG )
to provide switch redundancy. On each host, we use 802.3ad, LACP
mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate.
Did i miss something important here? and how to really achieve networking HA in Ceph cluster?
best regards,
Samuel
huxiaoyu@horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, June 16, 2021 10:18 AM, Andrew Walker-Brown <andrew_jbrown@hotmail.com> wrote:
With active mode, you then have a transmit hashing policy, usually set globally.
On Linux the bond would be set as ‘bond-mode 802.3ad’ and then ‘bond-xmit-hash-policy layer3+4’ - or whatever hashing policy you want.
Does the transmit hash policy need to be the same on the Linux server and on the switch side? or can they be different?
Ideally you’d want to have it the transmit hash the same...but so long load is being pretty evenly spread over all the links in the lag, then you’re fine. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: mabi<mailto:mabi@protonmail.ch> Sent: 16 June 2021 09:29 To: Andrew Walker-Brown<mailto:andrew_jbrown@hotmail.com> Cc: huxiaoyu@horebdata.cn<mailto:huxiaoyu@horebdata.cn>; Joe Comeau<mailto:Joe.Comeau@hli.ubc.ca>; ceph-users<mailto:ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, June 16, 2021 10:18 AM, Andrew Walker-Brown <andrew_jbrown@hotmail.com> wrote:
With active mode, you then have a transmit hashing policy, usually set globally.
On Linux the bond would be set as ‘bond-mode 802.3ad’ and then ‘bond-xmit-hash-policy layer3+4’ - or whatever hashing policy you want.
Does the transmit hash policy need to be the same on the Linux server and on the switch side? or can they be different?
participants (4)
-
Andrew Walker-Brown
-
huxiaoyu＠horebdata.cn
-
Joe Comeau
-
mabi