Setting up a small experimental CEPH network
People, I am interested in experimenting with CEPH on say 4 or 8 small form factor computers (SBCs?) - any suggestions about how to get started? I haven't bought anything yet - I have some working Fedora Workstations and Servers and a laptop but I don't want to experiment on them . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: phil@pricom.com.au
- pat yourself on the back for choosing ceph, there are a lot of experts(not including me :)) here willing to help(during office hours) - decide what you like to use ceph for, and how much storage you need. - Running just an osd on a server has not that many implications so you could rethink your test environment - Read here about when you need high frequency cpu's, cores, how much GB's of ram per osd/TB, - Don't think having a few 1Gbit can replace a >10Gbit. Ceph doesn't use such bonds optimal. I already asked about this years ago. Having a 10Gbe might make a SBC solution more costly than estimated. - I would use rhel7/centos7 as your base os. -----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Setting up a small experimental CEPH network People, I am interested in experimenting with CEPH on say 4 or 8 small form factor computers (SBCs?) - any suggestions about how to get started? I haven't bought anything yet - I have some working Fedora Workstations and Servers and a laptop but I don't want to experiment on them . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia
On 2020-09-20 12:19, Marc Roos wrote:
- pat yourself on the back for choosing ceph, there are a lot of experts(not including me :)) here willing to help(during office hours) - decide what you like to use ceph for, and how much storage you need. - Running just an osd on a server has not that many implications so you could rethink your test environment - Read here about when you need high frequency cpu's, cores, how much GB's of ram per osd/TB, - Don't think having a few 1Gbit can replace a >10Gbit. Ceph doesn't use such bonds optimal. I already asked about this years ago. Having a 10Gbe might make a SBC solution more costly than estimated.
My experience with bonding and Ceph is pretty good (OpenvSwitch). Ceph uses lots of tcp connections, and those can get shifted (balanced) between interfaces depending on load. The latency improvement of 10 Gb/s (and faster) is the main advantage IMHO. Sure Ceph wants plenty of bandwith when you want to rebalance / backfill / recover your cluster. But it really depends on your expectations and what you want to achieve. A three node SolidRun Honeycomb LX2K cluster [1] should be able to run anything you want and provide decent performance. But Ceph works best when you scale horizontally ... you might be surprised what aggregated throughput you can get from a lot of small nodes. Gr. Stefan [1]: https://www.solid-run.com/nxp-lx2160a-family/honeycomb-workstation/
On 21/09/2020 5:40 am, Stefan Kooman wrote:
My experience with bonding and Ceph is pretty good (OpenvSwitch). Ceph uses lots of tcp connections, and those can get shifted (balanced) between interfaces depending on load.
Same here - I'm running 4*1GB (LACP, Balance-TCP) on a 5 node cluster with 19 OSD's. 20 Active VM's and it idles at under 1 MiB/s, spikes up to 100MiB/s no problem. When doing a heavy rebalance/repair data rates on any one node can hit 400MiBs+ It scales out really well. -- Lindsay
I tested something in the past[1] where I could notice that an osd staturated a bond link and did not use the available 2nd one. I think I maybe made a mistake in writing down it was a 1x replicated pool. However it has been written here multiple times that these osd processes are single thread, so afaik they cannot use more than on link, and at the moment your osd has a saturated link, your clients will notice this. [1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html -----Original Message----- From: Lindsay Mathieson [mailto:lindsay.mathieson@gmail.com] Sent: maandag 21 september 2020 2:42 To: ceph-users@ceph.io Subject: [ceph-users] Re: Setting up a small experimental CEPH network On 21/09/2020 5:40 am, Stefan Kooman wrote:
My experience with bonding and Ceph is pretty good (OpenvSwitch). Ceph
uses lots of tcp connections, and those can get shifted (balanced) between interfaces depending on load.
Same here - I'm running 4*1GB (LACP, Balance-TCP) on a 5 node cluster with 19 OSD's. 20 Active VM's and it idles at under 1 MiB/s, spikes up to 100MiB/s no problem. When doing a heavy rebalance/repair data rates on any one node can hit 400MiBs+ It scales out really well. -- Lindsay _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, we use heavily bonded interfaces (6x10G) and also needed to look at this balancing question. We use LACP bonding and, while the host OS probably tries to balance outgoing traffic over all NICs, the real decision is made by the switches (incoming traffic). Our switches hash packets to a port by (source?) MAC address, meaning that it is not the number of TCP/IP connections that helps balancing, but only the number of MAC addresses. In an LACP bond, all NICs have the same MAC address and balancing happens by (physical) host. The more hosts, the better it will work. In a way, for us this is a problem and not at the same time. We have about 550 physical clients (an HPC cluster) and 12 OSD hosts, which means that we probably have a good load on every single NIC for client traffic. On the other hand, rebalancing between 12 servers is unlikely to use all NICs effectively. So far, we don't have enough disks per host to notice that, but it could become visible at some point. Basically, the host with the worst switch-sided hashing for incoming traffic will become the bottleneck. On some switches the hashing method for LACP bonds can be configured, however, not with much detail. I have not seen a possibility to use IP:PORT for hashing to a switch port. I have no experience with bonding mode 6 (ALB) that might provide a per-connection hashing. Would be interested to hear how it performs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc Roos <M.Roos@f1-outsourcing.eu> Sent: 21 September 2020 11:08:55 To: ceph-users; lindsay.mathieson Subject: [ceph-users] Re: Setting up a small experimental CEPH network I tested something in the past[1] where I could notice that an osd staturated a bond link and did not use the available 2nd one. I think I maybe made a mistake in writing down it was a 1x replicated pool. However it has been written here multiple times that these osd processes are single thread, so afaik they cannot use more than on link, and at the moment your osd has a saturated link, your clients will notice this. [1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html -----Original Message----- From: Lindsay Mathieson [mailto:lindsay.mathieson@gmail.com] Sent: maandag 21 september 2020 2:42 To: ceph-users@ceph.io Subject: [ceph-users] Re: Setting up a small experimental CEPH network On 21/09/2020 5:40 am, Stefan Kooman wrote:
My experience with bonding and Ceph is pretty good (OpenvSwitch). Ceph
uses lots of tcp connections, and those can get shifted (balanced) between interfaces depending on load.
Same here - I'm running 4*1GB (LACP, Balance-TCP) on a 5 node cluster with 19 OSD's. 20 Active VM's and it idles at under 1 MiB/s, spikes up to 100MiB/s no problem. When doing a heavy rebalance/repair data rates on any one node can hit 400MiBs+ It scales out really well. -- Lindsay _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
we use heavily bonded interfaces (6x10G) and also needed to look at this balancing question. We use LACP bonding and, while the host OS probably tries to balance outgoing traffic over all NICs
I tested something in the past[1] where I could notice that an osd staturated a bond link and did not use the available 2nd one.
This is exactly what I wrote about, and it doesn’t have to be this way. When using Linux bonding, be sure to set the xmit hash policy to layer3+4 and the mode on both sides to active/active. active/backup cuts your potential bandwidth, and a layer 1 / config problem on the backup link will be latent until you need it most, eg. when you do switch maintenance and assume that your bonds will all failover for continuity.
However it has been written here multiple times that these osd processes are single thread, so afaik they cannot use more than on link, and at the moment your osd has a saturated link, your clients will notice this.
Threads have nothing to do with links. Even if they did, real clusters have multiple OSDs per node, right?
[1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html
Context, brother, context. "1 osd per node cluster” “This is typical for a 'single line of communication' using lacp. Afaik the streams to the nodes are independent from each other anyway, so maybe it is possible to 'fork' the transmitting process, so linux can detect it as a separate stream and thus use the other link.” This is NOT typical of production, WDL’s microserver experiment notwithstanding. In real life, you’re going to have, what, at least 8 OSDs per node? Each with streams to multiple clients (and other OSDs). With dozens/hundreds/thousands of streams and a proper bonding (or equal-cost routing) setup, the *streams* are going to be hashed across available links by the bonding driver.
-----Original Message----- From: Lindsay Mathieson [mailto:lindsay.mathieson@gmail.com] Sent: maandag 21 september 2020 2:42 To: ceph-users@ceph.io Subject: [ceph-users] Re: Setting up a small experimental CEPH network
On 21/09/2020 5:40 am, Stefan Kooman wrote:
My experience with bonding and Ceph is pretty good (OpenvSwitch). Ceph
uses lots of tcp connections, and those can get shifted (balanced) between interfaces depending on load.
Same here - I'm running 4*1GB (LACP, Balance-TCP) on a 5 node cluster with 19 OSD's. 20 Active VM's and it idles at under 1 MiB/s, spikes up to 100MiB/s no problem. When doing a heavy rebalance/repair data rates on any one node can hit 400MiBs+
It scales out really well.
-- Lindsay _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Depending what you’re looking to accomplish, setting up a cluster in VMs (VirtualBox, Fusion, cloud provider, etc) may meet your needs without having to buy anything.
- Don't think having a few 1Gbit can replace a >10Gbit. Ceph doesn't use such bonds optimal. I already asked about this years ago. Having a 10Gbe might make a SBC solution more costly than estimated.
10GE is out of scope for sure, but a note about bonding: more than once I’ve seen NICs bonded improperly, or tested improperly. * Bonding doesn’t make a single faster interface - you need multiple clients to be able to utilize it well * If the hash policy isn’t set right, it’s easy to have *all* of your traffic routed over a single link, and your throughput will be capped at half what you expect
On 2020-09-20 10:25, Philip Rhoades wrote:
People,
I am interested in experimenting with CEPH on say 4 or 8 small form factor computers (SBCs?) - any suggestions about how to get started?
What do you want to learn? What Ceph features are you interested in (RGW, CephFS, RBD)? The experience will greatly depend on the speed of the storage you plan to use. SBC with NVMe/SSD/SATA are preffered over SD card/eMMC. A 64 bit CPU is required for modern Ceph. Aarch64 should just work. A bunch of new Raspberry Pi's (8GB RAM) and a couple of USB3 connected disks should get you a fun cluster to play with. Gr. Stefan
Perhaps not SBCs, but I have 4x HP 6300s and have been running Kubernetes together with Ceph/Rook for more than 3 years. The HPs can be picked up around 80-120eu. I learned so much in 3 years, last time I had that was when I started using Linux. This was money well spent and still is, it runs nextcloud, home automation, Wifi controller (unifi controller), websites -- and all that safely on Ceph. Also a big plus of those HP typical business desktops is Intel AMT. It's kind of a poor-man's Remote Management Console (similar-ish to iDRAC and iLO). This integrates nicely with, in my case Ubuntu MAAS. So this allows me to programmatically spin up one of those nodes. Or take a node out for maintenance. The 4 nodes and network switch run around 100W which I think is pretty ok, but again, this is not SBC territory; though if you think the SBCs will run with a lot less power, think again. If you use spinners expect them to be the main power user in your setup. I never expected them to really do 6W when idle and that really adds up when you have multiple disks per node. If you don't need the sheer data capacity, choose SSDs. I migrated to SSDs for at least the OS disks and that got me into the 100W baseline. Also take at least 8GB per node, including 1 OSD, and an extra 4GB for every extra OSD. Hans
participants (7)
-
Anthony D'Atri
-
Frank Schilder
-
Hans van den Bogert
-
Lindsay Mathieson
-
Marc Roos
-
Philip Rhoades
-
Stefan Kooman