Hi, Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-) We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28r... As we would like to be able to add more storage hosts, we need to loose the meshed network setup. My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches. Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-( Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..? We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have. Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021. So, anyone here with interesting insights to share for ceph 10G ethernet storage networking? Thanks, MJ
Hello, you can buy Arista 7050QX-32S are 40G Switches. They come for around 500-700€ each and can be stacked using MLAG. They work great. If you like to spend more money, 7060QX-32S are 100G Switches available for around 1500€. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Mo., 15. Feb. 2021 um 12:16 Uhr schrieb mj <lists@merit.unu.edu>:
Hi,
Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-)
We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28r...
As we would like to be able to add more storage hosts, we need to loose the meshed network setup.
My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches.
Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-(
Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..?
We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have.
Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021.
So, anyone here with interesting insights to share for ceph 10G ethernet storage networking?
Thanks, MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just wondering from where one can buy Arista 7050QX-32S for 700 USD? huxiaoyu@horebdata.cn From: Martin Verges Date: 2021-02-15 12:46 To: mj CC: ceph-users@ceph.io Subject: [ceph-users] Re: 10G stackabe lacp switches Hello, you can buy Arista 7050QX-32S are 40G Switches. They come for around 500-700€ each and can be stacked using MLAG. They work great. If you like to spend more money, 7060QX-32S are 100G Switches available for around 1500€. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Mo., 15. Feb. 2021 um 12:16 Uhr schrieb mj <lists@merit.unu.edu>:
Hi,
Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-)
We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28r...
As we would like to be able to add more storage hosts, we need to loose the meshed network setup.
My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches.
Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-(
Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..?
We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have.
Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021.
So, anyone here with interesting insights to share for ceph 10G ethernet storage networking?
Thanks, MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
3y ago I bought it on ebay to my home lab for 750$ with transport and duty and additional tax, so it's possible https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&_nkw=7050QX-32&_sacat=0&LH_TitleDesc=0&_osacat=0&_odkw=7050QX-32S BR, Sebastian On 15.02.2021 12:57, huxiaoyu@horebdata.cn wrote:
Just wondering from where one can buy Arista 7050QX-32S for 700 USD?
huxiaoyu@horebdata.cn
From: Martin Verges Date: 2021-02-15 12:46 To: mj CC: ceph-users@ceph.io Subject: [ceph-users] Re: 10G stackabe lacp switches Hello,
you can buy Arista 7050QX-32S are 40G Switches. They come for around 500-700€ each and can be stacked using MLAG. They work great. If you like to spend more money, 7060QX-32S are 100G Switches available for around 1500€.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
Am Mo., 15. Feb. 2021 um 12:16 Uhr schrieb mj <lists@merit.unu.edu>:
Hi,
Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-)
We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28r...
As we would like to be able to add more storage hosts, we need to loose the meshed network setup.
My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches.
Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-(
Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..?
We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have.
Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021.
So, anyone here with interesting insights to share for ceph 10G ethernet storage networking?
Thanks, MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Just to follow-up this dialogue: After Martin's tip for the arista's, we started looked around, and while buying from ebay did not appeal to us, we did manage to find a couple of refurb equipment resellers that offer them for a (little) bit more, but then including a year warranty. And actually, for an extra 175 euro, I was able to upgrade warranty from one to three years. Now I just need to get an ok from management here. And then the fun begins: start reading up on arista EOS MLAG config, en doing a lot of testing with new stuff. Now I wonder if it's accepted to ask arista-specific (MLAG) config questions here on this list... Have a nice weekend all! MJ On 2/15/21 1:41 PM, Sebastian Trojanowski wrote:
3y ago I bought it on ebay to my home lab for 750$ with transport and duty and additional tax, so it's possible
BR, Sebastian
On 15.02.2021 12:57, huxiaoyu@horebdata.cn wrote:
Just wondering from where one can buy Arista 7050QX-32S for 700 USD?
Hello MJ, Arista has a good documentation available for example at https://www.arista.com/en/um-eos/eos-multi-chassis-link-aggregation or https://eos.arista.com/mlag-basic-configuration/. Don't worry, when you know what exactly you want to configure, it's just a few lines of config in the end. It's more or less: On Both Switches: ------------------------- no spanning-tree vlan-id 4094 ! vlan 4094 trunk group mlag ! interface Port-Channel2000 switchport mode trunk switchport trunk group mlag ! interface Ethernet35 description MLAG link channel-group 2000 mode active ! interface Ethernet36 description MLAG link channel-group 2000 mode active ! On switch 1 in addition: ------------------------------- interface Vlan4094 ip address 10.2.0.1/30 ! mlag configuration domain-id mlag local-interface Vlan4094 peer-address 10.2.0.2 peer-link Port-Channel2000 ! On switch 2 in addition: ------------------------------- interface Vlan4094 ip address 10.2.0.2/30 ! mlag configuration domain-id mlag local-interface Vlan4094 peer-address 10.2.0.1 peer-link Port-Channel2000 ! After that you create port-channels for the ports you like to bundle with a specific mlag id On both switches as an example: --------------------------------------------- interface Port-Channel1 switchport access vlan 101 mlag 1 ! interface Ethernet7/1 description "MLAG bonding to Server XXX" channel-group 1 mode active ! -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 20. Feb. 2021 um 16:38 Uhr schrieb mj <lists@merit.unu.edu>:
Hi,
Just to follow-up this dialogue:
After Martin's tip for the arista's, we started looked around, and while buying from ebay did not appeal to us, we did manage to find a couple of refurb equipment resellers that offer them for a (little) bit more, but then including a year warranty.
And actually, for an extra 175 euro, I was able to upgrade warranty from one to three years.
Now I just need to get an ok from management here. And then the fun begins: start reading up on arista EOS MLAG config, en doing a lot of testing with new stuff.
Now I wonder if it's accepted to ask arista-specific (MLAG) config questions here on this list...
Have a nice weekend all!
MJ
On 2/15/21 1:41 PM, Sebastian Trojanowski wrote:
3y ago I bought it on ebay to my home lab for 750$ with transport and duty and additional tax, so it's possible
BR, Sebastian
On 15.02.2021 12:57, huxiaoyu@horebdata.cn wrote:
Just wondering from where one can buy Arista 7050QX-32S for 700 USD?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, We are using arista with MLAG on our storage layer. I would be fine with sharing questions and answers on this list, i guess more people can benefit from them. But as its a bit offtopic, feel also free to reach out to me directly, Best, Mart From mobile
On Feb 21, 2021, at 18:39, Martin Verges <martin.verges@croit.io> wrote:
Hello MJ,
Arista has a good documentation available for example at https://www.arista.com/en/um-eos/eos-multi-chassis-link-aggregation or https://eos.arista.com/mlag-basic-configuration/. Don't worry, when you know what exactly you want to configure, it's just a few lines of config in the end.
It's more or less:
On Both Switches: ------------------------- no spanning-tree vlan-id 4094 ! vlan 4094 trunk group mlag ! interface Port-Channel2000 switchport mode trunk switchport trunk group mlag ! interface Ethernet35 description MLAG link channel-group 2000 mode active ! interface Ethernet36 description MLAG link channel-group 2000 mode active !
On switch 1 in addition: ------------------------------- interface Vlan4094 ip address 10.2.0.1/30 ! mlag configuration domain-id mlag local-interface Vlan4094 peer-address 10.2.0.2 peer-link Port-Channel2000 !
On switch 2 in addition: ------------------------------- interface Vlan4094 ip address 10.2.0.2/30 ! mlag configuration domain-id mlag local-interface Vlan4094 peer-address 10.2.0.1 peer-link Port-Channel2000 !
After that you create port-channels for the ports you like to bundle with a specific mlag id
On both switches as an example: --------------------------------------------- interface Port-Channel1 switchport access vlan 101 mlag 1 ! interface Ethernet7/1 description "MLAG bonding to Server XXX" channel-group 1 mode active !
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
Am Sa., 20. Feb. 2021 um 16:38 Uhr schrieb mj <lists@merit.unu.edu>:
Hi,
Just to follow-up this dialogue:
After Martin's tip for the arista's, we started looked around, and while buying from ebay did not appeal to us, we did manage to find a couple of refurb equipment resellers that offer them for a (little) bit more, but then including a year warranty.
And actually, for an extra 175 euro, I was able to upgrade warranty from one to three years.
Now I just need to get an ok from management here. And then the fun begins: start reading up on arista EOS MLAG config, en doing a lot of testing with new stuff.
Now I wonder if it's accepted to ask arista-specific (MLAG) config questions here on this list...
Have a nice weekend all!
MJ
On 2/15/21 1:41 PM, Sebastian Trojanowski wrote: 3y ago I bought it on ebay to my home lab for 750$ with transport and duty and additional tax, so it's possible
BR, Sebastian
On 15.02.2021 12:57, huxiaoyu@horebdata.cn wrote:
Just wondering from where one can buy Arista 7050QX-32S for 700 USD?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I wanted to say thank you to all of those who have offered us (both on- and offlist as well) help and support for this arista move. It feels good to know that with it, we are on a path that many here are walking on. Lastly, if I may, one more question: My arista seller tells me that the arista 7050QX-32S supports 10G fiber uplinks, but I need 10G UTP RJ45 uplink to our HP core switch. (our HP has 1G fibers, and a 8 * 10G UTP RJ45 linecard. They tell me that the 7050QX-32S can do UTP RJ45, but only 1G speed, and not 10G. :-( Is that something that the (many) arista experts here can confirm or deny..? (other than: "You need to buy it, and then try it") Thanks! MJ On 21/02/2021 11:37, Martin Verges wrote:
Hello MJ,
Arista has a good documentation available for example at https://www.arista.com/en/um-eos/eos-multi-chassis-link-aggregation or https://eos.arista.com/mlag-basic-configuration/. Don't worry, when you know what exactly you want to configure, it's just a few lines of config in the end.
It's more or less:
On Both Switches: ------------------------- no spanning-tree vlan-id 4094 ! vlan 4094 trunk group mlag ! interface Port-Channel2000 switchport mode trunk switchport trunk group mlag ! interface Ethernet35 description MLAG link channel-group 2000 mode active ! interface Ethernet36 description MLAG link channel-group 2000 mode active !
On switch 1 in addition: ------------------------------- interface Vlan4094 ip address 10.2.0.1/30 ! mlag configuration domain-id mlag local-interface Vlan4094 peer-address 10.2.0.2 peer-link Port-Channel2000 !
On switch 2 in addition: ------------------------------- interface Vlan4094 ip address 10.2.0.2/30 ! mlag configuration domain-id mlag local-interface Vlan4094 peer-address 10.2.0.1 peer-link Port-Channel2000 !
After that you create port-channels for the ports you like to bundle with a specific mlag id
On both switches as an example: --------------------------------------------- interface Port-Channel1 switchport access vlan 101 mlag 1 ! interface Ethernet7/1 description "MLAG bonding to Server XXX" channel-group 1 mode active !
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
Am Sa., 20. Feb. 2021 um 16:38 Uhr schrieb mj <lists@merit.unu.edu>:
Hi,
Just to follow-up this dialogue:
After Martin's tip for the arista's, we started looked around, and while buying from ebay did not appeal to us, we did manage to find a couple of refurb equipment resellers that offer them for a (little) bit more, but then including a year warranty.
And actually, for an extra 175 euro, I was able to upgrade warranty from one to three years.
Now I just need to get an ok from management here. And then the fun begins: start reading up on arista EOS MLAG config, en doing a lot of testing with new stuff.
Now I wonder if it's accepted to ask arista-specific (MLAG) config questions here on this list...
Have a nice weekend all!
MJ
On 2/15/21 1:41 PM, Sebastian Trojanowski wrote:
3y ago I bought it on ebay to my home lab for 750$ with transport and duty and additional tax, so it's possible
BR, Sebastian
On 15.02.2021 12:57, huxiaoyu@horebdata.cn wrote:
Just wondering from where one can buy Arista 7050QX-32S for 700 USD?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2/15/21 12:16 PM, mj wrote:
As we would like to be able to add more storage hosts, we need to loose the meshed network setup.
My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches.
Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-(
Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..?
Just to note: stacking support does not necessarily also means MCLAG (LACP over multiple switches). I believe the Cisco's do support it. Sometimes stacking is nothing more than a management feature, i.e. able to manage the switches as it was one switch. So be sure to check if LACP over multiple chassis is supported. Gr. Stefan
Hi Stefan and Martin, Thanks very much for the quick responses! Lot's of interesting and relevant info! The arista option looks interesting, but those will not use the UTP ethernet cabling that we currently use. But given prices, perhap we should perhaps look into changing our cables from UTP to SFP+, I guess? Will check it out. Thanks again! MJ On 2/15/21 12:50 PM, Stefan Kooman wrote:
On 2/15/21 12:16 PM, mj wrote:
As we would like to be able to add more storage hosts, we need to loose the meshed network setup.
My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches.
Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-(
Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..?
Just to note: stacking support does not necessarily also means MCLAG (LACP over multiple switches). I believe the Cisco's do support it. Sometimes stacking is nothing more than a management feature, i.e. able to manage the switches as it was one switch. So be sure to check if LACP over multiple chassis is supported.
Gr. Stefan
Hi, Yes, Cisco supports VPC (virtual port-channel) for LACP over multiple switches. We're using 2x10G VPC on our Ceph and Proxmox hosts with 2 Cisco Nexus 3064PQ (48 x 10G SFP+ & 4 x 40G). These refs can be found for ~1000€ in refurb with lifetime warranty. Happy users here :-) Cheers, Fabien Le 15/02/2021 à 12:50, Stefan Kooman a écrit :
On 2/15/21 12:16 PM, mj wrote:
As we would like to be able to add more storage hosts, we need to loose the meshed network setup.
My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches.
Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-(
Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..?
Just to note: stacking support does not necessarily also means MCLAG (LACP over multiple switches). I believe the Cisco's do support it. Sometimes stacking is nothing more than a management feature, i.e. able to manage the switches as it was one switch. So be sure to check if LACP over multiple chassis is supported.
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Fabien Sirjean (fsirjean) writes:
Hi,
Yes, Cisco supports VPC (virtual port-channel) for LACP over multiple switches.
We're using 2x10G VPC on our Ceph and Proxmox hosts with 2 Cisco Nexus 3064PQ (48 x 10G SFP+ & 4 x 40G).
Same config here. Have set it up with LACP on the CEPH node side (bonding). There are a few gotchas with vPC which make me think Arista MLAG might be a bit simpler in the future, while keeping the 3064PQ as "core" devices.
These refs can be found for ~1000€ in refurb with lifetime warranty.
Indeed :) I paid 700 € on eBay.de for mine.
Hi,a El 15/2/21 a las 12:16, mj escribió:
Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-)
We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28r...
As we would like to be able to add more storage hosts, we need to loose the meshed network setup.
My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches.
Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-(
Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..?
We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have.
Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021.
So, anyone here with interesting insights to share for ceph 10G ethernet storage networking?
Do you really need MLAG? (the 2x10G bandwith?). If not, just use 2 simple switches (Mikrotik for example) and in Proxmox use an active-pasive bond, with default interface in all nodes to the same switch. Cheers -- Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO/ https://www.linkedin.com/company/37269706/
On 2/15/21 1:38 PM, Eneko Lacunza wrote:
Do you really need MLAG? (the 2x10G bandwith?). If not, just use 2 simple switches (Mikrotik for example) and in Proxmox use an active-pasive bond, with default interface in all nodes to the same switch.
Since we are now on SSD OSDs only, and our aim is to be able to add more OSD nodes, yes: I think we should aim for more than 10G bandwidth. Thanks!
Il giorno lun 15 feb 2021 alle ore 15:16 mj <lists@merit.unu.edu> ha scritto:
On 2/15/21 1:38 PM, Eneko Lacunza wrote:
Do you really need MLAG? (the 2x10G bandwith?). If not, just use 2 simple switches (Mikrotik for example) and in Proxmox use an active-pasive bond, with default interface in all nodes to the same switch.
Since we are now on SSD OSDs only, and our aim is to be able to add more OSD nodes, yes: I think we should aim for more than 10G bandwidth.
So go for 40G. LACP will not give you 2x10 magical bandwidth doubling.
BTW: I am using mikrotik 10g switches and they have great value. BTW2: if you use Proxmox you do not need LACP you can use linux round robin support that has the same performance of LACP and it does not require switches support.
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, this is not an answer to the question directly, but you could consider the following to double bandwidth: * Run each ceph node with two NICs, each has an own IP, e.g. one node has 192.0.2.10/24 and 192.0.2.11/24 * In ceph.conf you bind 50% of the OSDs to each of those IPs: [osd.XY] ... public_addr = ... cluster_addr = 192.0.2.x * With an equally distributed traffic (enough OSD) should nearly double your bandwidth * To get redundancy, you could extend that config using keepalived/vrrp to switchover the IP of a failing NIC/switch to the other one. I am pretty aware that we also have linux bonding with mode slb, but to my experience that didn't work very well with COTS switches, maybe due to ARP learing issues. (We ended up buying Juniper QFX-5100 with MLAG support). Best Regards, Andreas P.S. I didn't try out the setup from above yet. If anyone did already or will do, I would be happy about feedback. On 16.02.21 16:56, Mario Giammarco wrote:
Il giorno lun 15 feb 2021 alle ore 15:16 mj <lists@merit.unu.edu> ha scritto:
On 2/15/21 1:38 PM, Eneko Lacunza wrote:
Do you really need MLAG? (the 2x10G bandwith?). If not, just use 2 simple switches (Mikrotik for example) and in Proxmox use an active-pasive bond, with default interface in all nodes to the same switch.
Since we are now on SSD OSDs only, and our aim is to be able to add more OSD nodes, yes: I think we should aim for more than 10G bandwidth.
So go for 40G. LACP will not give you 2x10 magical bandwidth doubling.
BTW: I am using mikrotik 10g switches and they have great value. BTW2: if you use Proxmox you do not need LACP you can use linux round robin support that has the same performance of LACP and it does not require switches support.
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
MJ; I was looking at something similar, and reached out one of my VARs, and they recommended Netgear M4100 series switches. We don't user any of them yet, so I can't provide first-hand experience. On the subject of UTP vs SFP+; I'm told that SFP+ with DAC cables experience lower latencies than 10 Gig over copper. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: mj [mailto:lists@merit.unu.edu] Sent: Monday, February 15, 2021 4:16 AM To: ceph-users@ceph.io Subject: [ceph-users] 10G stackabe lacp switches Hi, Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-) We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28r... As we would like to be able to add more storage hosts, we need to loose the meshed network setup. My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches. Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-( Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..? We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have. Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021. So, anyone here with interesting insights to share for ceph 10G ethernet storage networking? Thanks, MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry; Netgear M4300 switches, not M4100. Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: DHilsbos@performair.com [mailto:DHilsbos@performair.com] Sent: Monday, February 15, 2021 8:39 AM To: lists@merit.unu.edu; ceph-users@ceph.io Subject: [ceph-users] Re: 10G stackabe lacp switches MJ; I was looking at something similar, and reached out one of my VARs, and they recommended Netgear M4100 series switches. We don't user any of them yet, so I can't provide first-hand experience. On the subject of UTP vs SFP+; I'm told that SFP+ with DAC cables experience lower latencies than 10 Gig over copper. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: mj [mailto:lists@merit.unu.edu] Sent: Monday, February 15, 2021 4:16 AM To: ceph-users@ceph.io Subject: [ceph-users] 10G stackabe lacp switches Hi, Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-) We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28r... As we would like to be able to add more storage hosts, we need to loose the meshed network setup. My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches. Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-( Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..? We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have. Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021. So, anyone here with interesting insights to share for ceph 10G ethernet storage networking? Thanks, MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (12)
-
Andreas John
-
DHilsbos＠performair.com
-
Eneko Lacunza
-
Fabien Sirjean
-
huxiaoyu＠horebdata.cn
-
Mario Giammarco
-
Mart van Santen
-
Martin Verges
-
mj
-
Phil Regnauld
-
Sebastian Trojanowski
-
Stefan Kooman