Hello, are cluster running with BGP in the field just fine. Out of curiosity, is there someone here that has his Ceph cluster running with BGP in production? As far as I remember, here at croit with multiple hundred supported clusters, we never encountered a BGP deployment in the field. It's always just the theoretical or testing where we hear from BGP. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx On Tue, 6 Jul 2021 at 07:11, Stefan Kooman <stefan(a)bit.nl> wrote: > On 7/5/21 6:26 PM, German Anders wrote: > > Hi All, > > > > I have an already created and functional ceph cluster (latest > luminous > > release) with two networks one for the public (layer 2+3) and the other > for > > the cluster, the public one uses VLAN and its 10GbE and the other one > uses > > Infiniband with 56Gb/s, the cluster works ok. The public network uses > > Juniper QFX5100 switches with VLAN in layer2+3 configuration but the > > network team needs to move to a full layer3 and they want to use BGP, so > > the question is, how can we move to that schema? What are the > > considerations? Is it possible? Is there any step-by-step way to move to > > that schema? Also is anything better than BGP or other alternatives? > > Ceph doesn't care at all. Just as long as the nodes can communicate to > each other, it's fine. It depends on your failure domains how easy you > can move to this L3 model. Do you have separate datacenters that you can > do one by one, or separate racks? > > And you can do BGP on different levels: router, top of rack switches, or > even on the Ceph host itselfs (FRR). > > We use BGP / VXLAN / EVPN for our Ceph cluster. But it all depends on > why your networking teams wants to change to L3, and why. > > There are no step by step guides, as most deployments are unique. > > This might be a good time to reconsider a separate cluster network. > Normally there is no need for that, and might make things simpler. > > Do you have separate storage switches? Whre are your clients connected > to (separate switches or connected to storage switches as well). > > are cluster running with BGP in the field just fine. > > Gr. Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

I'm not aware of any directly, but I know rook-ceph is used on Kubernetes, and Kubernetes is sometimes deployed with BGP based SDN layers. So there may be a few deployments that do it that way. ________________________________________ From: Martin Verges <martin.verges(a)croit.io> Sent: Monday, July 5, 2021 11:23 PM To: ceph-users Subject: [ceph-users] Re: Ceph with BGP? Check twice before you click! This email originated from outside PNNL. Hello, are cluster running with BGP in the field just fine. Out of curiosity, is there someone here that has his Ceph cluster running with BGP in production? As far as I remember, here at croit with multiple hundred supported clusters, we never encountered a BGP deployment in the field. It's always just the theoretical or testing where we hear from BGP. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft.me%2FMa… croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io%… YouTube: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgoo.gl%2F… On Tue, 6 Jul 2021 at 07:11, Stefan Kooman <stefan(a)bit.nl> wrote: > On 7/5/21 6:26 PM, German Anders wrote: > > Hi All, > > > > I have an already created and functional ceph cluster (latest > luminous > > release) with two networks one for the public (layer 2+3) and the other > for > > the cluster, the public one uses VLAN and its 10GbE and the other one > uses > > Infiniband with 56Gb/s, the cluster works ok. The public network uses > > Juniper QFX5100 switches with VLAN in layer2+3 configuration but the > > network team needs to move to a full layer3 and they want to use BGP, so > > the question is, how can we move to that schema? What are the > > considerations? Is it possible? Is there any step-by-step way to move to > > that schema? Also is anything better than BGP or other alternatives? > > Ceph doesn't care at all. Just as long as the nodes can communicate to > each other, it's fine. It depends on your failure domains how easy you > can move to this L3 model. Do you have separate datacenters that you can > do one by one, or separate racks? > > And you can do BGP on different levels: router, top of rack switches, or > even on the Ceph host itselfs (FRR). > > We use BGP / VXLAN / EVPN for our Ceph cluster. But it all depends on > why your networking teams wants to change to L3, and why. > > There are no step by step guides, as most deployments are unique. > > This might be a good time to reconsider a separate cluster network. > Normally there is no need for that, and might make things simpler. > > Do you have separate storage switches? Whre are your clients connected > to (separate switches or connected to storage switches as well). > > are cluster running with BGP in the field just fine. > > Gr. Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Without using FRR you are still L2 below ToR. Do you have any experience with FRR? You should think about multiple uplinks and how you handle them with FRR. Other than that, it should work, we have been in production with BGP above ToR for years without any problem. On Tue, Jul 6, 2021 at 4:10 PM German Anders <yodasbunker(a)gmail.com> wrote: > > Hello Stefan, thanks a lot for the reply. > > We have everything in the same datacenter, and the fault domains are per > rack. Regarding BGP, the idea of the networking team is to stop using layer > 2+3 and move everything to full layer 3, for this they want to implement > BGP and the idea is that each node is connected to a different ToR edge > switch and that the clients have a single IP and can then reach the entire > cluster. > Currently, we have an environment configured with a specific VLAN > without GW, and they want to get the VLAN out of the way and that each node > has its own IP with its own GW (and that is the ToR Switch). We already > have a separate cluster network that is running on Infiniband and it's > completely separated. So the idea is to use BGP on the public network only. > > Thanks in advance, > > Cheers, > > On Tue, Jul 6, 2021 at 2:10 AM Stefan Kooman <stefan(a)bit.nl> wrote: > > > On 7/5/21 6:26 PM, German Anders wrote: > > > Hi All, > > > > > > I have an already created and functional ceph cluster (latest > > luminous > > > release) with two networks one for the public (layer 2+3) and the other > > for > > > the cluster, the public one uses VLAN and its 10GbE and the other one > > uses > > > Infiniband with 56Gb/s, the cluster works ok. The public network uses > > > Juniper QFX5100 switches with VLAN in layer2+3 configuration but the > > > network team needs to move to a full layer3 and they want to use BGP, so > > > the question is, how can we move to that schema? What are the > > > considerations? Is it possible? Is there any step-by-step way to move to > > > that schema? Also is anything better than BGP or other alternatives? > > > > Ceph doesn't care at all. Just as long as the nodes can communicate to > > each other, it's fine. It depends on your failure domains how easy you > > can move to this L3 model. Do you have separate datacenters that you can > > do one by one, or separate racks? > > > > And you can do BGP on different levels: router, top of rack switches, or > > even on the Ceph host itselfs (FRR). > > > > We use BGP / VXLAN / EVPN for our Ceph cluster. But it all depends on > > why your networking teams wants to change to L3, and why. > > > > There are no step by step guides, as most deployments are unique. > > > > This might be a good time to reconsider a separate cluster network. > > Normally there is no need for that, and might make things simpler. > > > > Do you have separate storage switches? Whre are your clients connected > > to (separate switches or connected to storage switches as well). > > > > This is not easy to answer without all the details. But for sure there > > are cluster running with BGP in the field just fine. > > > > Gr. Stefan > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Sorry, I did not use it before, we evaluated it and went for L2 below ToR. You need to design the BGP network with your network team because every node will have an ASN too. You need to set up a test network before going to production. On Tue, Jul 6, 2021 at 4:33 PM German Anders <yodasbunker(a)gmail.com> wrote: > > Great, now regarding FRR no, the truth is that I never had experience before, is there any specific guide to frr-ceph configuration or any advice? Actually, we have for the public network a bond created with two 10GbE link, so we can break the bond and get two 10GbE fibers per node for the public network > > On Tue, Jul 6, 2021 at 10:27 AM Serkan Çoban <cobanserkan(a)gmail.com> wrote: >> >> Without using FRR you are still L2 below ToR. Do you have any >> experience with FRR? >> You should think about multiple uplinks and how you handle them with FRR. >> Other than that, it should work, we have been in production with BGP >> above ToR for years without any problem. >> >> On Tue, Jul 6, 2021 at 4:10 PM German Anders <yodasbunker(a)gmail.com> wrote: >> > >> > Hello Stefan, thanks a lot for the reply. >> > >> > We have everything in the same datacenter, and the fault domains are per >> > rack. Regarding BGP, the idea of the networking team is to stop using layer >> > 2+3 and move everything to full layer 3, for this they want to implement >> > BGP and the idea is that each node is connected to a different ToR edge >> > switch and that the clients have a single IP and can then reach the entire >> > cluster. >> > Currently, we have an environment configured with a specific VLAN >> > without GW, and they want to get the VLAN out of the way and that each node >> > has its own IP with its own GW (and that is the ToR Switch). We already >> > have a separate cluster network that is running on Infiniband and it's >> > completely separated. So the idea is to use BGP on the public network only. >> > >> > Thanks in advance, >> > >> > Cheers, >> > >> > On Tue, Jul 6, 2021 at 2:10 AM Stefan Kooman <stefan(a)bit.nl> wrote: >> > >> > > On 7/5/21 6:26 PM, German Anders wrote: >> > > > Hi All, >> > > > >> > > > I have an already created and functional ceph cluster (latest >> > > luminous >> > > > release) with two networks one for the public (layer 2+3) and the other >> > > for >> > > > the cluster, the public one uses VLAN and its 10GbE and the other one >> > > uses >> > > > Infiniband with 56Gb/s, the cluster works ok. The public network uses >> > > > Juniper QFX5100 switches with VLAN in layer2+3 configuration but the >> > > > network team needs to move to a full layer3 and they want to use BGP, so >> > > > the question is, how can we move to that schema? What are the >> > > > considerations? Is it possible? Is there any step-by-step way to move to >> > > > that schema? Also is anything better than BGP or other alternatives? >> > > >> > > Ceph doesn't care at all. Just as long as the nodes can communicate to >> > > each other, it's fine. It depends on your failure domains how easy you >> > > can move to this L3 model. Do you have separate datacenters that you can >> > > do one by one, or separate racks? >> > > >> > > And you can do BGP on different levels: router, top of rack switches, or >> > > even on the Ceph host itselfs (FRR). >> > > >> > > We use BGP / VXLAN / EVPN for our Ceph cluster. But it all depends on >> > > why your networking teams wants to change to L3, and why. >> > > >> > > There are no step by step guides, as most deployments are unique. >> > > >> > > This might be a good time to reconsider a separate cluster network. >> > > Normally there is no need for that, and might make things simpler. >> > > >> > > Do you have separate storage switches? Whre are your clients connected >> > > to (separate switches or connected to storage switches as well). >> > > >> > > This is not easy to answer without all the details. But for sure there >> > > are cluster running with BGP in the field just fine. >> > > >> > > Gr. Stefan >> > > _______________________________________________ >> > > ceph-users mailing list -- ceph-users(a)ceph.io >> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io