3y ago I bought it on ebay to my home lab for 750$ with transport and duty and additional tax, so it's possible https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&a… BR, Sebastian On 15.02.2021 12:57, huxiaoyu(a)horebdata.cn wrote: > Just wondering from where one can buy Arista 7050QX-32S for 700 USD? > > > > > > huxiaoyu(a)horebdata.cn > > From: Martin Verges > Date: 2021-02-15 12:46 > To: mj > CC: ceph-users(a)ceph.io > Subject: [ceph-users] Re: 10G stackabe lacp switches > Hello, > > you can buy Arista 7050QX-32S are 40G Switches. They come for around > 500-700€ each and can be stacked using MLAG. They work great. > If you like to spend more money, 7060QX-32S are 100G Switches > available for around 1500€. > > -- > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges(a)croit.io > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > Am Mo., 15. Feb. 2021 um 12:16 Uhr schrieb mj <lists(a)merit.unu.edu>du>: >> >> Hi, >> >> Hapy to report that we recently upgraded our three-host 24 OSD cluster >> from HDD filestore to SSD BlueStore. After a few months of use, their >> WEAR is still at 1%, and the cluster performance ("rados bench" etc) has >> dramatically improved. So all in all: yes, we're happy Samsung PM883 >> ceph users. :-) >> >> We currently have a "meshed" ceph setup, with the three hosts connected >> directly to each other over 10G ethernet, as described here: >> >> https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28… >> >> As we would like to be able to add more storage hosts, we need to loose >> the meshed network setup. >> >> My idea is to add two stacked 10G ethernet switches to the setup, so we >> can start using lacp bonded networking over two physical switches. >> >> Looking around, we can get refurb Cisco Small Business 550X for around >> 1300 euro. We also noticed that mikrotik and TP-Link have some even >> nicer-priced 10G switches, but those all lack bonding. :-( >> >> Therfore I'm asking here: anyone here with suggestions on what to look >> at, for nice-priced 10G stackable switches..? >> >> We would like to continue using ethernet, as we use that everywhere, and >> also performance-wise we're happy with what we currently have. >> >> Last december I wrote to mikrotik support, asking if they will support >> stacking / LACP any time soon, and their answer was: probably 2nd half >> of 2021. >> >> So, anyone here with interesting insights to share for ceph 10G ethernet >> storage networking? >> >> Thanks, >> MJ >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

Hello MJ, Arista has a good documentation available for example at https://www.arista.com/en/um-eos/eos-multi-chassis-link-aggregation or https://eos.arista.com/mlag-basic-configuration/. Don't worry, when you know what exactly you want to configure, it's just a few lines of config in the end. It's more or less: On Both Switches: ------------------------- no spanning-tree vlan-id 4094 ! vlan 4094 trunk group mlag ! interface Port-Channel2000 switchport mode trunk switchport trunk group mlag ! interface Ethernet35 description MLAG link channel-group 2000 mode active ! interface Ethernet36 description MLAG link channel-group 2000 mode active ! On switch 1 in addition: ------------------------------- interface Vlan4094 ip address 10.2.0.1/30 ! mlag configuration domain-id mlag local-interface Vlan4094 peer-address 10.2.0.2 peer-link Port-Channel2000 ! On switch 2 in addition: ------------------------------- interface Vlan4094 ip address 10.2.0.2/30 ! mlag configuration domain-id mlag local-interface Vlan4094 peer-address 10.2.0.1 peer-link Port-Channel2000 ! After that you create port-channels for the ports you like to bundle with a specific mlag id On both switches as an example: --------------------------------------------- interface Port-Channel1 switchport access vlan 101 mlag 1 ! interface Ethernet7/1 description "MLAG bonding to Server XXX" channel-group 1 mode active ! -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 20. Feb. 2021 um 16:38 Uhr schrieb mj <lists(a)merit.unu.edu>du>: > > Hi, > > Just to follow-up this dialogue: > > After Martin's tip for the arista's, we started looked around, and while > buying from ebay did not appeal to us, we did manage to find a couple of > refurb equipment resellers that offer them for a (little) bit more, but > then including a year warranty. > > And actually, for an extra 175 euro, I was able to upgrade warranty from > one to three years. > > Now I just need to get an ok from management here. And then the fun > begins: start reading up on arista EOS MLAG config, en doing a lot of > testing with new stuff. > > Now I wonder if it's accepted to ask arista-specific (MLAG) config > questions here on this list... > > Have a nice weekend all! > > MJ > > On 2/15/21 1:41 PM, Sebastian Trojanowski wrote: > > 3y ago I bought it on ebay to my home lab for 750$ with transport and > > duty and additional tax, so it's possible > > > > https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&a… > > > > > > BR, > > Sebastian > > > > On 15.02.2021 12:57, huxiaoyu(a)horebdata.cn wrote: > >> Just wondering from where one can buy Arista 7050QX-32S for 700 USD? > >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

I wanted to say thank you to all of those who have offered us (both on- and offlist as well) help and support for this arista move. It feels good to know that with it, we are on a path that many here are walking on. Lastly, if I may, one more question: My arista seller tells me that the arista 7050QX-32S supports 10G fiber uplinks, but I need 10G UTP RJ45 uplink to our HP core switch. (our HP has 1G fibers, and a 8 * 10G UTP RJ45 linecard. They tell me that the 7050QX-32S can do UTP RJ45, but only 1G speed, and not 10G. :-( Is that something that the (many) arista experts here can confirm or deny..? (other than: "You need to buy it, and then try it") Thanks! MJ On 21/02/2021 11:37, Martin Verges wrote: > Hello MJ, > > Arista has a good documentation available for example at > https://www.arista.com/en/um-eos/eos-multi-chassis-link-aggregation or > https://eos.arista.com/mlag-basic-configuration/. Don't worry, when > you know what exactly you want to configure, it's just a few lines of > config in the end. > > It's more or less: > > On Both Switches: > ------------------------- > no spanning-tree vlan-id 4094 > ! > vlan 4094 > trunk group mlag > ! > interface Port-Channel2000 > switchport mode trunk > switchport trunk group mlag > ! > interface Ethernet35 > description MLAG link > channel-group 2000 mode active > ! > interface Ethernet36 > description MLAG link > channel-group 2000 mode active > ! > > On switch 1 in addition: > ------------------------------- > interface Vlan4094 > ip address 10.2.0.1/30 > ! > mlag configuration > domain-id mlag > local-interface Vlan4094 > peer-address 10.2.0.2 > peer-link Port-Channel2000 > ! > > On switch 2 in addition: > ------------------------------- > interface Vlan4094 > ip address 10.2.0.2/30 > ! > mlag configuration > domain-id mlag > local-interface Vlan4094 > peer-address 10.2.0.1 > peer-link Port-Channel2000 > ! > > After that you create port-channels for the ports you like to bundle > with a specific mlag id > > On both switches as an example: > --------------------------------------------- > interface Port-Channel1 > switchport access vlan 101 > mlag 1 > ! > interface Ethernet7/1 > description "MLAG bonding to Server XXX" > channel-group 1 mode active > ! > > -- > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges(a)croit.io > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > Am Sa., 20. Feb. 2021 um 16:38 Uhr schrieb mj <lists(a)merit.unu.edu>du>: >> >> Hi, >> >> Just to follow-up this dialogue: >> >> After Martin's tip for the arista's, we started looked around, and while >> buying from ebay did not appeal to us, we did manage to find a couple of >> refurb equipment resellers that offer them for a (little) bit more, but >> then including a year warranty. >> >> And actually, for an extra 175 euro, I was able to upgrade warranty from >> one to three years. >> >> Now I just need to get an ok from management here. And then the fun >> begins: start reading up on arista EOS MLAG config, en doing a lot of >> testing with new stuff. >> >> Now I wonder if it's accepted to ask arista-specific (MLAG) config >> questions here on this list... >> >> Have a nice weekend all! >> >> MJ >> >> On 2/15/21 1:41 PM, Sebastian Trojanowski wrote: >>> 3y ago I bought it on ebay to my home lab for 750$ with transport and >>> duty and additional tax, so it's possible >>> >>> https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&a… >>> >>> >>> BR, >>> Sebastian >>> >>> On 15.02.2021 12:57, huxiaoyu(a)horebdata.cn wrote: >>>> Just wondering from where one can buy Arista 7050QX-32S for 700 USD? >>>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi Stefan and Martin, Thanks very much for the quick responses! Lot's of interesting and relevant info! The arista option looks interesting, but those will not use the UTP ethernet cabling that we currently use. But given prices, perhap we should perhaps look into changing our cables from UTP to SFP+, I guess? Will check it out. Thanks again! MJ On 2/15/21 12:50 PM, Stefan Kooman wrote: > On 2/15/21 12:16 PM, mj wrote: > >> As we would like to be able to add more storage hosts, we need to >> loose the meshed network setup. >> >> My idea is to add two stacked 10G ethernet switches to the setup, so >> we can start using lacp bonded networking over two physical switches. >> >> Looking around, we can get refurb Cisco Small Business 550X for around >> 1300 euro. We also noticed that mikrotik and TP-Link have some even >> nicer-priced 10G switches, but those all lack bonding. :-( >> >> Therfore I'm asking here: anyone here with suggestions on what to look >> at, for nice-priced 10G stackable switches..? > > Just to note: stacking support does not necessarily also means MCLAG > (LACP over multiple switches). I believe the Cisco's do support it. > Sometimes stacking is nothing more than a management feature, i.e. able > to manage the switches as it was one switch. So be sure to check if LACP > over multiple chassis is supported. > > Gr. Stefan > >

Fabien Sirjean (fsirjean) writes: Same config here. Have set it up with LACP on the CEPH node side (bonding). There are a few gotchas with vPC which make me think Arista MLAG might be a bit simpler in the future, while keeping the 3064PQ as "core" devices. Indeed :) I paid 700 € on eBay.de for mine.

On 2/15/21 1:38 PM, Eneko Lacunza wrote: Since we are now on SSD OSDs only, and our aim is to be able to add more OSD nodes, yes: I think we should aim for more than 10G bandwidth. Thanks!

Hello, this is not an answer to the question directly, but you could consider the following to double bandwidth: * Run each ceph node with two NICs, each has an own IP, e.g. one node has 192.0.2.10/24 and 192.0.2.11/24 * In ceph.conf you bind 50% of the OSDs to each of those IPs: [osd.XY] ... public_addr = ... cluster_addr = 192.0.2.x * With an equally distributed traffic (enough OSD) should nearly double your bandwidth * To get redundancy, you could extend that config using keepalived/vrrp to switchover the IP of a failing NIC/switch to the other one. I am pretty aware that we also have linux bonding with mode slb, but to my experience that didn't work very well with COTS switches, maybe due to ARP learing issues. (We ended up buying Juniper QFX-5100 with MLAG support). Best Regards, Andreas P.S. I didn't try out the setup from above yet. If anyone did already or will do, I would be happy about feedback. On 16.02.21 16:56, Mario Giammarco wrote: -- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet

Sorry; Netgear M4300 switches, not M4100. Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com -----Original Message----- From: DHilsbos(a)performair.com [mailto:DHilsbos@performair.com] Sent: Monday, February 15, 2021 8:39 AM To: lists(a)merit.unu.edu; ceph-users(a)ceph.io Subject: [ceph-users] Re: 10G stackabe lacp switches MJ; I was looking at something similar, and reached out one of my VARs, and they recommended Netgear M4100 series switches. We don't user any of them yet, so I can't provide first-hand experience. On the subject of UTP vs SFP+; I'm told that SFP+ with DAC cables experience lower latencies than 10 Gig over copper. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com -----Original Message----- From: mj [mailto:lists@merit.unu.edu] Sent: Monday, February 15, 2021 4:16 AM To: ceph-users(a)ceph.io Subject: [ceph-users] 10G stackabe lacp switches Hi, Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-) We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28… As we would like to be able to add more storage hosts, we need to loose the meshed network setup. My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches. Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-( Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..? We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have. Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021. So, anyone here with interesting insights to share for ceph 10G ethernet storage networking? Thanks, MJ _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io