Hello Nghia, I once asked a similar question about network architecture and got the same answer as Martin wrote from Wido den Hollander: There is no need to have a public and cluster network with Ceph. Working as a Ceph consultant I've deployed multi-PB Ceph clusters with a single public network without any problems. Each node has a single IP-address, nothing more, nothing less. In the current Ceph manual you can read It is possible to run a Ceph Storage Cluster with two networks: a public (front-side) network and a cluster (back-side) network. However, this approach complicates network configuration (both hardware and software) and does not usually have a significant impact on overall performance. For this reason, we generally recommend that dual-NIC systems either be configured with two IPs on the same network, or bonded. I followed the advice from Wido "One system, one IP address" and everything works fine. So, you should be fine with one interface for MONs, MGRs, and OSDs. Best Willi On 5/8/20 11:57 AM, Nghia Viet Tran wrote: > Hi Martin, > > Thanks for your response. You mean one network interface for only MON > hosts or for the whole cluster including OSD hosts? I’m confusing now > because there are some projects that only useone public network for the > whole cluster. That means the rebalancing, replicating objects and > heartbeats from OSD hostswould affects the performance of Ceph client. > > *From: *Martin Verges &lt;martin.verges(a)croit.io&gt; > *Date: *Friday, May 8, 2020 at 16:20 > *To: *Nghia Viet Tran &lt;Nghia.Viet.Tran(a)mgm-tp.com&gt; > *Cc: *&quot;ceph-users(a)ceph.io&quot; &lt;ceph-users(a)ceph.io&gt; > *Subject: *Re: [ceph-users] Cluster network and public network > > Hello Nghia, > > just use one network interface card and use frontend and backend traffic > on the same. No problem with that. > > If you have a dual port card, use both ports as an LACP channel and > maybe separate it using VLANs if you want to, but not required as well. > > > -- > > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io> > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > Am Fr., 8. Mai 2020 um 09:29 Uhr schrieb Nghia Viet Tran > &lt;Nghia.Viet.Tran(a)mgm-tp.com <mailto:Nghia.Viet.Tran@mgm-tp.com>>: > > Hi everyone, > > I have a question about the network setup. From the document, It’s > recommended to have 2 NICs per hosts as described in below picture > > Diagram > > In the picture, OSD hosts will connect to the Cluster network for > replicate and heartbeat between OSDs, therefore, we definitely need > 2 NICs for it. But seems there are no connections between Ceph MON > and Cluster network. Can we install 1 NIC on Ceph MON then? > > I appreciated any comments! > > Thank you! > > -- > > Nghia Viet Tran (Mr) > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

During steady-state, sure. Heartbeats go over that, as do replication ops when clients write data. During heavy recovery or backfill, including healing from failures, balancing, adding/removing drives, much more will be used. Convention wisdom has been to not let that traffic DoS clients, or clients to DoS heartbeats. But this I think dates to a time when 1Gb/s networks were common. If one’s using modern multiple/bonded 25Gb/s or 40Gb/s links …. If your public network is saturated, that actually is a problem, last thing you want is to add recovery traffic, or to slow down heartbeats. For most people, it isn’t saturated. How do you define “full” ? TOR uplinks? TORs to individual nodes? Switch backplanes? Are you using bonding with the wrong hash policy? For an increasing percentage of folks deploying production-quality clusters, agreed.

Anthony D'Atri (anthony.datri) writes: [...] See Frank Schilder's post about a meltdown which he believes could have been caused by beacon/hearbeat being drowned out by other recovery/IO trafic, not at the network level, but at the processing level on the OSDs. If indeed there are cases where the OSDs are too busy to send (or process) heartbeat/beacon messaging, it wouldn't help to have a separate network ? Cheers, Phil

On 5/8/20 12:13 PM, Willi Schiegel wrote: Great to hear! I'm still behind this idea and all the clusters I design have a single (or LACP) network going to the host. One IP address per node where all traffic goes over. That's Ceph, SSH, (SNMP) Monitoring, etc. Wido > Best > Willi > > On 5/8/20 11:57 AM, Nghia Viet Tran wrote: >> Hi Martin, >> >> Thanks for your response. You mean one network interface for only MON >> hosts or for the whole cluster including OSD hosts? I’m confusing now >> because there are some projects that only useone public network for >> the whole cluster. That means the rebalancing, replicating objects and >> heartbeats from OSD hostswould affects the performance of Ceph client. >> >> *From: *Martin Verges &lt;martin.verges(a)croit.io&gt; >> *Date: *Friday, May 8, 2020 at 16:20 >> *To: *Nghia Viet Tran &lt;Nghia.Viet.Tran(a)mgm-tp.com&gt; >> *Cc: *&quot;ceph-users(a)ceph.io&quot; &lt;ceph-users(a)ceph.io&gt; >> *Subject: *Re: [ceph-users] Cluster network and public network >> >> Hello Nghia, >> >> just use one network interface card and use frontend and backend >> traffic on the same. No problem with that. >> >> If you have a dual port card, use both ports as an LACP channel and >> maybe separate it using VLANs if you want to, but not required as well. >> >> >> -- >> >> Martin Verges >> Managing director >> >> Mobile: +49 174 9335695 >> E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io> >> Chat: https://t.me/MartinVerges >> >> croit GmbH, Freseniusstr. 31h, 81247 Munich >> CEO: Martin Verges - VAT-ID: DE310638492 >> Com. register: Amtsgericht Munich HRB 231263 >> >> Web: https://croit.io >> YouTube: https://goo.gl/PGE1Bx >> >> Am Fr., 8. Mai 2020 um 09:29 Uhr schrieb Nghia Viet Tran >> &lt;Nghia.Viet.Tran(a)mgm-tp.com <mailto:Nghia.Viet.Tran@mgm-tp.com>>: >> >>     Hi everyone, >> >>     I have a question about the network setup. From the document, It’s >>     recommended to have 2 NICs per hosts as described in below picture >> >>     Diagram >> >>     In the picture, OSD hosts will connect to the Cluster network for >>     replicate and heartbeat between OSDs, therefore, we definitely need >>     2 NICs for it. But seems there are no connections between Ceph MON >>     and Cluster network. Can we install 1 NIC on Ceph MON then? >> >>     I appreciated any comments! >> >>     Thank you! >> >>     -- >>     Nghia Viet Tran (Mr) >> >>     _______________________________________________ >>     ceph-users mailing list -- ceph-users(a)ceph.io >>     <mailto:ceph-users@ceph.io> >>     To unsubscribe send an email to ceph-users-leave(a)ceph.io >>     <mailto:ceph-users-leave@ceph.io> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi, On 11/05/2020 08:50, Wido den Hollander wrote: We have an 'old-style' cluster, seperated LAN/cluster network. We would like to move over to the 'new-style'. Is it as easy as: define the NICs in a 2x10G LACP bond0, and add both NICs to the bond0 config, and add configure like: and add our cluster IP as a second IP, like On all nodes, reboot, and everything will work? Or are there ceph specifics to consider? Thanks, MJ

FWIW this has not been my experience, at least through Luminous. What I’ve seen is that when the cluster/replication net is configured but unavailable, OSD heartbeats fail and peers report them to the mons as down. The mons send out a map accordingly, and the affected OSDs report “I’m not dead yet!”. Flap flap flap. YMMV

Dear all, looks like I need to be more precise: I did not mean to have a back network configured but it is taken down. Of course this won't work. What I mean is that you: 1. remove the cluster network definition from the cluster config (ceph.conf and/or ceph config ...) 2. restart OSDs to apply the change 3. remove the physical network Step 2 will most likely require down time as you write, because during the transition some OSDs will think all OSDs listen on 2 while other OSDs think everyone is listening on 1 network. If you can afford to take all clients down and do a full cluster restart, this is doable. If you set noout,nodown,pause and maybe some other flags (norebalance,nobackfill,norecover), wait for all client *and* recovery I/O to complete, it is probably possible to do this transition without disconnecting clients by just restarting all OSDs failure domain by failure domain. After the transition things should work fine with just 1 network. In any case, my recommendation would be to keep both networks if they are on different VLAN IDs. Then, nothing special is required to do the transition and this is what I did to simplify the physical networking (two logical networks, identical physical networking). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman &lt;stefan(a)bit.nl&gt; Sent: 13 May 2020 07:40 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Cluster network and public network On 2020-05-12 18:59, Anthony D'Atri wrote: +1. This has also been my experience. And it's quit hard to debug as well (confusing / seemingly contradictory messages). It uses the back network to replicate data ... and as long as it can't (client) IO wont go through. Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Besides the recoverry scenario , in a write only scenario the cluster network will use the almost the same bandwith as public network. linyunfan Anthony D'Atri &lt;anthony.datri(a)gmail.com&gt; 于2020年5月9日周六 下午4:32写道： > > > > Hi, > > > > I deployed few clusters with two networks as well as only one network. > > There has little impact between them for my experience. > > > > I did a performance test on nautilus cluster with two networks last week. > > What I found is that the cluster network has low bandwidth usage > > During steady-state, sure. Heartbeats go over that, as do replication ops when clients write data. > > During heavy recovery or backfill, including healing from failures, balancing, adding/removing drives, much more will be used. > > Convention wisdom has been to not let that traffic DoS clients, or clients to DoS heartbeats. > > But this I think dates to a time when 1Gb/s networks were common. If one’s using modern multiple/bonded 25Gb/s or 40Gb/s links …. > > > while public network bandwidth is nearly full. > > If your public network is saturated, that actually is a problem, last thing you want is to add recovery traffic, or to slow down heartbeats. For most people, it isn’t saturated. > > How do you define “full” ? TOR uplinks? TORs to individual nodes? Switch backplanes? Are you using bonding with the wrong hash policy? > > > As a result, I don't think the cluster network is necessary. > > For an increasing percentage of folks deploying production-quality clusters, agreed. > > > > > > > Willi Schiegel &lt;willi.schiegel(a)technologit.de&gt; 于2020年5月8日周五 下午6:14写道： > > > >> Hello Nghia, > >> > >> I once asked a similar question about network architecture and got the > >> same answer as Martin wrote from Wido den Hollander: > >> > >> There is no need to have a public and cluster network with Ceph. Working > >> as a Ceph consultant I've deployed multi-PB Ceph clusters with a single > >> public network without any problems. Each node has a single IP-address, > >> nothing more, nothing less. > >> > >> In the current Ceph manual you can read > >> > >> It is possible to run a Ceph Storage Cluster with two networks: a public > >> (front-side) network and a cluster (back-side) network. However, this > >> approach complicates network configuration (both hardware and software) > >> and does not usually have a significant impact on overall performance. > >> For this reason, we generally recommend that dual-NIC systems either be > >> configured with two IPs on the same network, or bonded. > >> > >> I followed the advice from Wido "One system, one IP address" and > >> everything works fine. So, you should be fine with one interface for > >> MONs, MGRs, and OSDs. > >> > >> Best > >> Willi > >> > >> On 5/8/20 11:57 AM, Nghia Viet Tran wrote: > >>> Hi Martin, > >>> > >>> Thanks for your response. You mean one network interface for only MON > >>> hosts or for the whole cluster including OSD hosts? I’m confusing now > >>> because there are some projects that only useone public network for the > >>> whole cluster. That means the rebalancing, replicating objects and > >>> heartbeats from OSD hostswould affects the performance of Ceph client. > >>> > >>> *From: *Martin Verges &lt;martin.verges(a)croit.io&gt; > >>> *Date: *Friday, May 8, 2020 at 16:20 > >>> *To: *Nghia Viet Tran &lt;Nghia.Viet.Tran(a)mgm-tp.com&gt; > >>> *Cc: *&quot;ceph-users(a)ceph.io&quot; &lt;ceph-users(a)ceph.io&gt; > >>> *Subject: *Re: [ceph-users] Cluster network and public network > >>> > >>> Hello Nghia, > >>> > >>> just use one network interface card and use frontend and backend traffic > >>> on the same. No problem with that. > >>> > >>> If you have a dual port card, use both ports as an LACP channel and > >>> maybe separate it using VLANs if you want to, but not required as well. > >>> > >>> > >>> -- > >>> > >>> Martin Verges > >>> Managing director > >>> > >>> Mobile: +49 174 9335695 > >>> E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io> > >>> Chat: https://t.me/MartinVerges > >>> > >>> croit GmbH, Freseniusstr. 31h, 81247 Munich > >>> CEO: Martin Verges - VAT-ID: DE310638492 > >>> Com. register: Amtsgericht Munich HRB 231263 > >>> > >>> Web: https://croit.io > >>> YouTube: https://goo.gl/PGE1Bx > >>> > >>> Am Fr., 8. Mai 2020 um 09:29 Uhr schrieb Nghia Viet Tran > >>> &lt;Nghia.Viet.Tran(a)mgm-tp.com <mailto:Nghia.Viet.Tran@mgm-tp.com>>: > >>> > >>> Hi everyone, > >>> > >>> I have a question about the network setup. From the document, It’s > >>> recommended to have 2 NICs per hosts as described in below picture > >>> > >>> Diagram > >>> > >>> In the picture, OSD hosts will connect to the Cluster network for > >>> replicate and heartbeat between OSDs, therefore, we definitely need > >>> 2 NICs for it. But seems there are no connections between Ceph MON > >>> and Cluster network. Can we install 1 NIC on Ceph MON then? > >>> > >>> I appreciated any comments! > >>> > >>> Thank you! > >>> > >>> -- > >>> > >>> Nghia Viet Tran (Mr) > >>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> <mailto:ceph-users@ceph.io> > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> <mailto:ceph-users-leave@ceph.io> > >>> > >>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

That is correct.I didn't explain it clearly. I said that is because in some write only scenario the public network and cluster network will all be saturated the same time. linyunfan Janne Johansson &lt;icepic.dz(a)gmail.com&gt; 于2020年5月14日周四 下午3:42写道： > > Den tors 14 maj 2020 kl 08:42 skrev lin yunfan &lt;lin.yunfan(a)gmail.com&gt;om>: >> >> Besides the recoverry scenario , in a write only scenario the cluster >> network will use the almost the same bandwith as public network. > > > That would depend on the replication factor. If it is high, I would assume every MB from the client network would make (repl-factor - 1) times the data on the private network to send replication requests to the other OSD hosts with the same amount of data. > > -- > May the most significant bit of your life be positive.

Den tors 14 maj 2020 kl 10:46 skrev Amudhan P &lt;amudhan83(a)gmail.com&gt;om>: I guess this depends on what parameters you use. All in all I think using one network is probably better, and the cases where I have seen missing heartbeats, it's not the network that prevents packets from coming over, it is the OSDs that make themselves busy doing something else instead of heartbeating that makes them flip out, so if you can set it up with LACP or other kinds of bonding, you allow the OSD hosts to use the network optimally regardless of what state they are in. -- May the most significant bit of your life be positive.