20 Sep
2020
20 Sep
'20
12:55 p.m.
People,
I am interested in experimenting with CEPH on say 4 or 8 small form
factor computers (SBCs?) - any suggestions about how to get started?
I haven't bought anything yet - I have some working Fedora Workstations
and Servers and a laptop but I don't want to experiment on them . .
Thanks,
Phil.
--
Philip Rhoades
PO Box 896
Cowra NSW 2794
Australia
E-mail: phil(a)pricom.com.au
20 Sep
20 Sep
2:49 p.m.
- pat yourself on the back for choosing ceph, there are a lot of
experts(not including me :)) here willing to help(during office hours)
- decide what you like to use ceph for, and how much storage you need.
- Running just an osd on a server has not that many implications so you
could rethink your test environment
- Read here about when you need high frequency cpu's, cores, how much
GB's of ram per osd/TB,
- Don't think having a few 1Gbit can replace a >10Gbit. Ceph doesn't use
such bonds optimal. I already asked about this years ago. Having a 10Gbe
might make a SBC solution more costly than estimated.
- I would use rhel7/centos7 as your base os.
-----Original Message-----
To: ceph-users(a)ceph.io
Subject: [ceph-users] Setting up a small experimental CEPH network
People,
I am interested in experimenting with CEPH on say 4 or 8 small form
factor computers (SBCs?) - any suggestions about how to get started?
I haven't bought anything yet - I have some working Fedora Workstations
and Servers and a laptop but I don't want to experiment on them . .
Thanks,
Phil.
--
Philip Rhoades
PO Box 896
Cowra NSW 2794
Australia
11:10 p.m.
On 2020-09-20 12:19, Marc Roos wrote:
- pat yourself on the back for choosing ceph, there are a lot of
experts(not including me :)) here willing to help(during office hours)
- decide what you like to use ceph for, and how much storage you need.
- Running just an osd on a server has not that many implications so you
could rethink your test environment
- Read here about when you need high frequency cpu's, cores, how much
GB's of ram per osd/TB,
- Don't think having a few 1Gbit can replace a >10Gbit. Ceph doesn't use
such bonds optimal. I already asked about this years ago. Having a 10Gbe
might make a SBC solution more costly than estimated.
My experience with bonding and Ceph is pretty good (OpenvSwitch). Ceph
uses lots of tcp connections, and those can get shifted (balanced)
between interfaces depending on load.
The latency improvement of 10 Gb/s (and faster) is the main advantage
IMHO. Sure Ceph wants plenty of bandwith when you want to rebalance /
backfill / recover your cluster. But it really depends on your
expectations and what you want to achieve. A three node SolidRun
Honeycomb LX2K cluster [1] should be able to run anything you want and
provide decent performance.
But Ceph works best when you scale horizontally ... you might be
surprised what aggregated throughput you can get from a lot of small nodes.
Gr. Stefan
[1]: https://www.solid-run.com/nxp-lx2160a-family/honeycomb-workstation/
21 Sep
21 Sep
4:12 a.m.
On 21/09/2020 5:40 am, Stefan Kooman wrote:
My experience with bonding and Ceph is pretty good
(OpenvSwitch). Ceph
uses lots of tcp connections, and those can get shifted (balanced)
between interfaces depending on load.
Same here - I'm running 4*1GB (LACP, Balance-TCP) on a 5 node cluster
with 19 OSD's. 20 Active VM's and it idles at under 1 MiB/s, spikes up
to 100MiB/s no problem. When doing a heavy rebalance/repair data rates
on any one node can hit 400MiBs+
It scales out really well.
--
Lindsay
12:38 p.m.
I tested something in the past[1] where I could notice that an osd
staturated a bond link and did not use the available 2nd one. I think I
maybe made a mistake in writing down it was a 1x replicated pool.
However it has been written here multiple times that these osd processes
are single thread, so afaik they cannot use more than on link, and at
the moment your osd has a saturated link, your clients will notice this.
[1]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html
-----Original Message-----
From: Lindsay Mathieson [mailto:lindsay.mathieson@gmail.com]
Sent: maandag 21 september 2020 2:42
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Setting up a small experimental CEPH network
On 21/09/2020 5:40 am, Stefan Kooman wrote:
My experience with bonding and Ceph is pretty good
(OpenvSwitch). Ceph
uses lots of tcp connections, and those can get
shifted (balanced)
between interfaces depending on load.
Same here - I'm running 4*1GB (LACP, Balance-TCP) on a 5 node cluster
with 19 OSD's. 20 Active VM's and it idles at under 1 MiB/s, spikes up
to 100MiB/s no problem. When doing a heavy rebalance/repair data rates
on any one node can hit 400MiBs+
It scales out really well.
--
Lindsay
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
4:03 p.m.
Hi all,
we use heavily bonded interfaces (6x10G) and also needed to look at this balancing
question. We use LACP bonding and, while the host OS probably tries to balance outgoing
traffic over all NICs, the real decision is made by the switches (incoming traffic). Our
switches hash packets to a port by (source?) MAC address, meaning that it is not the
number of TCP/IP connections that helps balancing, but only the number of MAC addresses.
In an LACP bond, all NICs have the same MAC address and balancing happens by (physical)
host. The more hosts, the better it will work.
In a way, for us this is a problem and not at the same time. We have about 550 physical
clients (an HPC cluster) and 12 OSD hosts, which means that we probably have a good load
on every single NIC for client traffic.
On the other hand, rebalancing between 12 servers is unlikely to use all NICs effectively.
So far, we don't have enough disks per host to notice that, but it could become
visible at some point. Basically, the host with the worst switch-sided hashing for
incoming traffic will become the bottleneck.
On some switches the hashing method for LACP bonds can be configured, however, not with
much detail. I have not seen a possibility to use IP:PORT for hashing to a switch port.
I have no experience with bonding mode 6 (ALB) that might provide a per-connection
hashing. Would be interested to hear how it performs.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Marc Roos <M.Roos(a)f1-outsourcing.eu>
Sent: 21 September 2020 11:08:55
To: ceph-users; lindsay.mathieson
Subject: [ceph-users] Re: Setting up a small experimental CEPH network
I tested something in the past[1] where I could notice that an osd
staturated a bond link and did not use the available 2nd one. I think I
maybe made a mistake in writing down it was a 1x replicated pool.
However it has been written here multiple times that these osd processes
are single thread, so afaik they cannot use more than on link, and at
the moment your osd has a saturated link, your clients will notice this.
[1]
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html
-----Original Message-----
From: Lindsay Mathieson [mailto:lindsay.mathieson@gmail.com]
Sent: maandag 21 september 2020 2:42
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Setting up a small experimental CEPH network
On 21/09/2020 5:40 am, Stefan Kooman wrote:
My experience with bonding and Ceph is pretty good
(OpenvSwitch). Ceph
uses lots of tcp connections, and those can get
shifted (balanced)
between interfaces depending on load.
Same here - I'm running 4*1GB (LACP, Balance-TCP) on a 5 node cluster
with 19 OSD's. 20 Active VM's and it idles at under 1 MiB/s, spikes up
to 100MiB/s no problem. When doing a heavy rebalance/repair data rates
on any one node can hit 400MiBs+
It scales out really well.
--
Lindsay
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:41 p.m.
we use heavily bonded interfaces (6x10G) and also
needed to look at this balancing question. We use LACP bonding and, while the host OS
probably tries to balance outgoing traffic over all NICs
I tested something in the past[1] where I could notice
that an osd
staturated a bond link and did not use the available 2nd one.
This is exactly what I wrote about, and it doesn’t have to be this way.
When using Linux bonding, be sure to set the xmit hash policy to layer3+4 and the mode on
both sides to active/active.
active/backup cuts your potential bandwidth, and a layer 1 / config problem on the backup
link will be latent until you need it most, eg. when you do switch maintenance and assume
that your bonds will all failover for continuity.
However it has been written here multiple times that
these osd processes
are single thread, so afaik they cannot use more than on link, and at
the moment your osd has a saturated link, your clients will notice this.
Threads have nothing to do with links. Even if they did, real clusters have multiple OSDs
per node, right?
Context, brother, context.
"1 osd per node cluster”
“This is typical for a 'single line of communication' using lacp. Afaik
the streams to the nodes are independent from each other anyway, so
maybe it is possible to 'fork' the transmitting process, so linux can
detect it as a separate stream and thus use the other link.”
This is NOT typical of production, WDL’s microserver experiment notwithstanding.
In real life, you’re going to have, what, at least 8 OSDs per node? Each with streams to
multiple clients (and other OSDs). With dozens/hundreds/thousands of streams and a proper
bonding (or equal-cost routing) setup, the *streams* are going to be hashed across
available links by the bonding driver.
-----Original Message-----
From: Lindsay Mathieson [mailto:lindsay.mathieson@gmail.com]
Sent: maandag 21 september 2020 2:42
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Setting up a small experimental CEPH network
On 21/09/2020 5:40 am, Stefan Kooman wrote:
My experience with bonding and Ceph is pretty
good (OpenvSwitch). Ceph
uses lots of tcp connections, and those can get
shifted (balanced)
between interfaces depending on load.
Same here - I'm running 4*1GB (LACP, Balance-TCP) on a 5 node cluster
with 19 OSD's. 20 Active VM's and it idles at under 1 MiB/s, spikes up
to 100MiB/s no problem. When doing a heavy rebalance/repair data rates
on any one node can hit 400MiBs+
It scales out really well.
--
Lindsay
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:59 a.m.
Depending what you’re looking to accomplish, setting up a cluster in VMs (VirtualBox,
Fusion, cloud provider, etc) may meet your needs without having to buy anything.
- Don't think having a few 1Gbit can replace a >10Gbit. Ceph doesn't use
such bonds optimal. I already asked about this years ago. Having a 10Gbe
might make a SBC solution more costly than estimated.
10GE is out of scope for sure, but a note about bonding: more than once I’ve seen NICs
bonded improperly, or tested improperly.
* Bonding doesn’t make a single faster interface - you need multiple clients to be able to
utilize it well
* If the hash policy isn’t set right, it’s easy to have *all* of your traffic routed over
a single link, and your throughput will be capped at half what you expect
20 Sep
20 Sep
11:53 p.m.
On 2020-09-20 10:25, Philip Rhoades wrote:
People,
I am interested in experimenting with CEPH on say 4 or 8 small form
factor computers (SBCs?) - any suggestions about how to get started?
What do you want to learn? What Ceph features are you interested in
(RGW, CephFS, RBD)? The experience will greatly depend on the speed of
the storage you plan to use. SBC with NVMe/SSD/SATA are preffered over
SD card/eMMC. A 64 bit CPU is required for modern Ceph. Aarch64 should
just work. A bunch of new Raspberry Pi's (8GB RAM) and a couple of USB3
connected disks should get you a fun cluster to play with.
Gr. Stefan
22 Sep
22 Sep
2:18 a.m.
Perhaps not SBCs, but I have 4x HP 6300s and have been running
Kubernetes together with Ceph/Rook for more than 3 years. The HPs can be
picked up around 80-120eu. I learned so much in 3 years, last time I had
that was when I started using Linux. This was money well spent and still
is, it runs nextcloud, home automation, Wifi controller (unifi
controller), websites -- and all that safely on Ceph.
Also a big plus of those HP typical business desktops is Intel AMT. It's
kind of a poor-man's Remote Management Console (similar-ish to iDRAC and
iLO). This integrates nicely with, in my case Ubuntu MAAS. So this
allows me to programmatically spin up one of those nodes. Or take a node
out for maintenance.
The 4 nodes and network switch run around 100W which I think is pretty
ok, but again, this is not SBC territory; though if you think the SBCs
will run with a lot less power, think again. If you use spinners expect
them to be the main power user in your setup. I never expected them to
really do 6W when idle and that really adds up when you have multiple
disks per node. If you don't need the sheer data capacity, choose SSDs.
I migrated to SSDs for at least the OS disks and that got me into the
100W baseline. Also take at least 8GB per node, including 1 OSD, and an
extra 4GB for every extra OSD.
Hans
1332
days inactive
1333
days old
9 comments
7 participants
participants (7)
-
Anthony D'Atri -
Frank Schilder -
Hans van den Bogert -
Lindsay Mathieson -
Marc Roos -
Philip Rhoades -
Stefan Kooman