Hi,
I deployed few clusters with two networks as well as only one network.
There has little impact between them for my experience.
I did a performance test on nautilus cluster with two networks last week.
What I found is that the cluster network has low bandwidth usage
During steady-state, sure. Heartbeats go over that, as do replication ops when clients
write data.
During heavy recovery or backfill, including healing from failures, balancing,
adding/removing drives, much more will be used.
Convention wisdom has been to not let that traffic DoS clients, or clients to DoS
heartbeats.
But this I think dates to a time when 1Gb/s networks were common. If one’s using modern
multiple/bonded 25Gb/s or 40Gb/s links ….
while public network bandwidth is nearly full.
If your public network is saturated, that actually is a problem, last thing you want is to
add recovery traffic, or to slow down heartbeats. For most people, it isn’t saturated.
How do you define “full” ? TOR uplinks? TORs to individual nodes? Switch backplanes?
Are you using bonding with the wrong hash policy?
As a result, I don't think the cluster network is
necessary.
For an increasing percentage of folks deploying production-quality clusters, agreed.
Willi Schiegel <willi.schiegel(a)technologit.de> 于2020年5月8日周五 下午6:14写道:
Hello Nghia,
I once asked a similar question about network architecture and got the
same answer as Martin wrote from Wido den Hollander:
There is no need to have a public and cluster network with Ceph. Working
as a Ceph consultant I've deployed multi-PB Ceph clusters with a single
public network without any problems. Each node has a single IP-address,
nothing more, nothing less.
In the current Ceph manual you can read
It is possible to run a Ceph Storage Cluster with two networks: a public
(front-side) network and a cluster (back-side) network. However, this
approach complicates network configuration (both hardware and software)
and does not usually have a significant impact on overall performance.
For this reason, we generally recommend that dual-NIC systems either be
configured with two IPs on the same network, or bonded.
I followed the advice from Wido "One system, one IP address" and
everything works fine. So, you should be fine with one interface for
MONs, MGRs, and OSDs.
Best
Willi
On 5/8/20 11:57 AM, Nghia Viet Tran wrote:
Hi Martin,
Thanks for your response. You mean one network interface for only MON
hosts or for the whole cluster including OSD hosts? I’m confusing now
because there are some projects that only useone public network for the
whole cluster. That means the rebalancing, replicating objects and
heartbeats from OSD hostswould affects the performance of Ceph client.
*From: *Martin Verges <martin.verges(a)croit.io>
*Date: *Friday, May 8, 2020 at 16:20
*To: *Nghia Viet Tran <Nghia.Viet.Tran(a)mgm-tp.com>
*Cc: *"ceph-users(a)ceph.io" <ceph-users(a)ceph.io>
*Subject: *Re: [ceph-users] Cluster network and public network
Hello Nghia,
just use one network interface card and use frontend and backend traffic
on the same. No problem with that.
If you have a dual port card, use both ports as an LACP channel and
maybe separate it using VLANs if you want to, but not required as well.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io>
Chat:
https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web:
https://croit.io
YouTube:
https://goo.gl/PGE1Bx
Am Fr., 8. Mai 2020 um 09:29 Uhr schrieb Nghia Viet Tran
<Nghia.Viet.Tran(a)mgm-tp.com <mailto:Nghia.Viet.Tran@mgm-tp.com>>:
Hi everyone,
I have a question about the network setup. From the document, It’s
recommended to have 2 NICs per hosts as described in below picture
Diagram
In the picture, OSD hosts will connect to the Cluster network for
replicate and heartbeat between OSDs, therefore, we definitely need
2 NICs for it. But seems there are no connections between Ceph MON
and Cluster network. Can we install 1 NIC on Ceph MON then?
I appreciated any comments!
Thank you!
--
Nghia Viet Tran (Mr)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io