Hi Stefan, I think you gave me the right pointers. Last summer I was looking up exactly this, how do Dell switches hash connections onto members of a LAG. What I found was, that the only option was by MAC. I did a test with iperf using several connections between the same two servers, or from one to many. This test confirmed what I found in the documentation, all connections between 2 servers shared a single 10G member, while one-to-many connections were distributed over multiple members. Back then, I thought this was it and didn't look into this further. Now, after your hints, I went back to the manual and find that the switches actually do support more advanced hash functions - at least after enabling ECMP. By default it is disabled. I'm not sure if I was reading a manual for the wrong switch family, no idea where I found "MAC only" statement. I got in touch with Dell support to help me here, the manual on load balancing is not exactly great. I can use MACs, IP, port, VLAN ID and a few other packet fields for hashing. I hope not only in layer 3 routing. In particular, including the VLAN ID should help spreading client and replication traffic out a bit better. And Dell also supports defining salts to avoid polarisation, which I believe is hurting us as well at the moment. I have one last question. The Dell manual states that one can enable monitoring of load balancing and it will check every 15secs for imbalance across the members of a LAG. You wrote "... and with OVS you can balance the load between the LACP links (by default it evaluates every 10 seconds if it should move flows around)." How is this done? The hash function doesn't change, so how can port mappings be re-arranged in a predictable way? The Dell switches will only create log events, nothing more. The Dell manual uses the term "dynamic load balancing", but generating log messages is not really the same. Am missing something? For us, I think a bit more clever hashing and, maybe, higher priority for the replication VLAN will do. As far as I can see, our cluster is essentially running on 10G internally and anything better than that should do and be easy to achieve. Thanks for putting me on the right track. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 12 February 2021 18:59:23 To: Stefan Kooman Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Network design issues By the way, thanks for reminding me of bmon! Of course. I have a decent collection of live monitoring tools installed and bmon was one of the first. How could I forget? Another tool I became good friends with is atop. It gives a really good overview of the entire system, including network, disks, swap paging, you name it. I forgot about that too. Have a good weekend. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder Sent: 12 February 2021 18:52:05 To: Stefan Kooman Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] Network design issues Hi Stefan, OK, I added the ceph-users again :) Thanks for your reply, this is a lot of useful pointers. Yes, its Dell EMC switches running OS9 and I believe they support per-VLAN bandwidth reservations. It would be the easiest to configure and test. At the moment, I always see the slow ping times on both, the front- and back interface at the same time on exactly the same OSD pairs. If I reserve bandwidth to the replication VLAN and the slow ping times on the back interface disappear, this would be a really strong clue. I will go through everything after the weekend. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan(a)bit.nl> Sent: 12 February 2021 18:18 To: Frank Schilder Subject: Re: [ceph-users] Network design issues On 2/12/21 5:27 PM, Frank Schilder wrote: ^^ This, I recently switched to Thunderbird because of mail migration (from Mutt) ... and I'm not used to it yet. I *tried* to reply to all (incl. list) but might have screwed up. I would consider this as of general interest. Ah yes, I've read that thread. Interesting. I haven't tested it out yet, but will do so. quite a few people have run into issues with that settig. We haven't had any issues with it yet. But perhaps I should downscale it as well. Yeah sure, I would suspect that as well. Or "discards" from a switch because of errors, but those are less likely. Are these Dell / EMC switches? You might be able to give priority on a VLAN level, or "shape" bandwith based on VLANs. I know that Aristas (that we use) have support for that in newish firmware. You might also want to support "pause" frames (ethernet flow control) as that might help during congestion (a back off protocol), see: https://en.wikipedia.org/wiki/Ethernet_flow_control Just to note: we don't have a separate replication network / interfaces. We only have one network. Wido (den Hollander) and myself as well, don't see any added benefit of a separate network. You only waste bandwith if you split them up. And make debugging more complex in certain failure scenarios. Do you know what hashing is in use for the LACP port-channel? You want to use mac, ip and port (5-tuple). We use OpenvSwitch (OVS) a lot, and with OVS you can balance the load between the LACP links (by default it evaluates every 10 seconds if it should move flows around). I doubt there is a silver bullet, but hey, you never know. Do change one thing at a time, otherwise it will be hard to know what the effect is of each of the changes (they might even cancel each other out). Yes, your issue makes more sense to me now. Do you have any metrics from the load on the individual links? Even bmon might be a useful tool. You might want to capture metrics (like every second or so) to detect "bursts" of traffic that might cause issues. Just to make sure you are on the right track. We use telegraf as metric collecting agent sending them to influxdb, but there are many more options. And then there are also other things to tune: tcp checksum offload et al. You might also hit IRQ balance issue, and there are also ways to overcome those. Are those single CPU systems? And / or AMD? NUMA might be a thing as well, and ideally you have the Ceph OSD daemons pinned to the CPU with the network / storage adapters connected. Finally, this might be of use: http://www.brendangregg.com/usemethod.html ;-). Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi Stefan, thanks for the additional info. Dell will put me in touch with their deployment team soonish and then I can ask about matching abilities. It turns out that the problem I observed might have a much more profane reason. I saw really long periods with slow ping time yesterday and finally managed to pin it down to a flapping link. My best bet is that an SFP transceiver has gone bad. What I'm really surprised about is, that the switch seems not to have any flapping detection. It happily takes the port up and down several times per second. Unfortunately, I can't find anything about server-sided flapping detection on mode=4 bonds nor for members of a LAG on the switch. Do you know of anything that does that? I might be looking for the wrong term. We have quite high redundancy. I can loose up to 3 ports on a server before the aggregated bandwidth might get too small. Therefore, I would be happy to take the occasional false positive as long as we don't miss the real flaps. Something like "permanently shut down interface if it does a down-up 3 times per second" would be perfect. Ideally without having to watch the logs. For the future, I plan to go 25G active-passive without preferred port. This config will handle flapping gracefully. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan(a)bit.nl> Sent: 15 February 2021 21:24:09 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] Re: Network design issues On 2/15/21 5:38 PM, Frank Schilder wrote: When the workload is perfectly static, nothing changes. But that will hardly every be the case. Here the info for OVS on this: "Every 10 seconds, vswitchd rebalances the bond members (see bond_rebalance()). To rebalance, vswitchd examines the statistics for the number of bytes transmitted by each member over approximately the past minute, with data sent more recently weighted more heavily than data sent less recently. It considers each of the members in order from most-loaded to least-loaded. If highly loaded member H is significantly more heavily loaded than the least-loaded member L, and member H carries at least two hashes, then vswitchd shifts one of H’s hashes to L. However, vswitchd will only shift a hash from H to L if it will decrease the ratio of the load between H and L by at least 0.1. Currently, “significantly more loaded” means that H must carry at least 1 Mbps more traffic, and that traffic must be at least 3% greater than L’s." So if it makes sense to move one or more flows on other links, it will do so. I guess the Dell switches will do something similar. Good to hear, I hope you manage to solve it. Gr. Stefan