29 Sep
2020
29 Sep
'20
6:44 a.m.
Hi All :),
I would like to get your feedback about the components below to build a PoC
OSD Node (I will build 3 of these).
SSD for OS.
NVMe for cache.
HDD for storage.
The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
[image: image.png]
Thanks for your feedback!
--
Ignacio Ocampo
29 Sep
29 Sep
7:12 a.m.
On 2020-09-29 08:44, Ignacio Ocampo wrote:
Hi All :),
I would like to get your feedback about the components below to build a
PoC OSD Node (I will build 3 of these).
SSD for OS.
NVMe for cache.
^^ For RocksDB / WAL, or a dm-cache setup. Should work either way.
HDD for storage.
The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
image.png
Thanks for your feedback!
LGTM. The flash storage has power loss protection, so fsyncs should be
fast and durable.
Gr. Stefan
1 Oct
1 Oct
4:57 p.m.
What kind of cache configuration are you planning? Are you going to use
CephFS, RGW, and/or RBD?
On Tue, Sep 29, 2020 at 2:45 AM Ignacio Ocampo <nafiux(a)gmail.com> wrote:
Hi All :),
I would like to get your feedback about the components below to build a
PoC OSD Node (I will build 3 of these).
SSD for OS.
NVMe for cache.
HDD for storage.
The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
[image: image.png]
Thanks for your feedback!
--
Ignacio Ocampo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:34 p.m.
RGW and RBD primarily, CephFS in less capacity.
On 1 Oct 2020, at 9:58, Nathan Fish
<lordcirth(a)gmail.com> wrote:
What kind of cache configuration are you planning? Are you going to use CephFS, RGW,
and/or RBD?
> On Tue, Sep 29, 2020 at 2:45 AM Ignacio Ocampo <nafiux(a)gmail.com> wrote:
> Hi All :),
>
> I would like to get your feedback about the components below to build a PoC OSD Node
(I will build 3 of these).
>
> SSD for OS.
> NVMe for cache.
> HDD for storage.
>
> The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
>
> <image.png>
>
>
> Thanks for your feedback!
>
> --
> Ignacio Ocampo
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:35 p.m.
Welcome to Ceph!
I think better questions to start with are “what are your objectives in your study?” Is it
just seeing Ceph run with many disks, or are you trying to see how much performance you
can get out of it with distributed disk? What is your budget? Do you want to try different
combinations of storage devices to learn how they differ in performance or do you just
want to jump to the fastest things out there?
One often doesn’t need a bunch of machines to determine that Ceph is a really versatile
and robust solution. I pretty regularly deploy Ceph on a single node using Kubernetes and
Rook. Some would ask “why would one ever do that, just use direct storage!”. The answer is
when I want to expand a cluster, I am willing to have traded initial performance overhead
for letting Ceph distribute data at a later date. And the overhead is far lower than one
might think when there’s not a network bottleneck to deal with. I do use direct storage on
LVM when I have distributed workloads such as Kafka that abstract storage that a service
instance depends on. It doesn’t make much sense in my mind for Kafka or Cassandra to use
Ceph because I can afford to lose nodes using those services.
In other words, Ceph is virtualized storage. You have likely come to it because your
workloads need to be able to come up anywhere on your network and reach that storage. How
do you see those workloads exercising the capabilities of Ceph? That’s where your
interesting use cases come from, and can help you better decide what the best lab platform
is to get started.
Hope that helps, Brian
On Sep 29, 2020, at 12:44 AM, Ignacio Ocampo
<nafiux(a)gmail.com> wrote:
Hi All :),
I would like to get your feedback about the components below to build a PoC OSD Node (I
will build 3 of these).
SSD for OS.
NVMe for cache.
HDD for storage.
The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
<image.png>
Thanks for your feedback!
--
Ignacio Ocampo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Oct
2 Oct
6:31 a.m.
Hi Brian,
Here more context about what I want to accomplish: I've migrated a bunch of
services from AWS to a local server, but having everything in a single
server is not safe, and instead of investing in RAID, I would like to start
setting up a small Ceph Cluster to have redundancy and a robust mechanism
in case any component fails.
Also, in the mid-term, I do have plans to deploy a small OpenStack Cluster.
Because of that, I would like to set up the first small Ceph Cluster that
can scale as my needs grow, the idea is to have 3 OSD nodes with the same
characteristics and add additional HDDs as needed, up to 5 HDD per OSD
node, starting with 1 HDD per node.
Thanks!
On Thu, Oct 1, 2020 at 11:35 AM Brian Topping <brian.topping(a)gmail.com>
wrote:
Welcome to Ceph!
I think better questions to start with are “what are your objectives in
your study?” Is it just seeing Ceph run with many disks, or are you trying
to see how much performance you can get out of it with distributed disk?
What is your budget? Do you want to try different combinations of storage
devices to learn how they differ in performance or do you just want to jump
to the fastest things out there?
One often doesn’t need a bunch of machines to determine that Ceph is a
really versatile and robust solution. I pretty regularly deploy Ceph on a
single node using Kubernetes and Rook. Some would ask “why would one ever
do that, just use direct storage!”. The answer is when I want to expand a
cluster, I am willing to have traded initial performance overhead for
letting Ceph distribute data at a later date. And the overhead is far lower
than one might think when there’s not a network bottleneck to deal with. I
do use direct storage on LVM when I have distributed workloads such as
Kafka that abstract storage that a service instance depends on. It doesn’t
make much sense in my mind for Kafka or Cassandra to use Ceph because I can
afford to lose nodes using those services.
In other words, Ceph is virtualized storage. You have likely come to it
because your workloads need to be able to come up anywhere on your network
and reach that storage. How do you see those workloads exercising the
capabilities of Ceph? That’s where your interesting use cases come from,
and can help you better decide what the best lab platform is to get started.
Hope that helps, Brian
On Sep 29, 2020, at 12:44 AM, Ignacio Ocampo <nafiux(a)gmail.com> wrote:
Hi All :),
I would like to get your feedback about the components below to build a
PoC OSD Node (I will build 3 of these).
SSD for OS.
NVMe for cache.
HDD for storage.
The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
<image.png>
Thanks for your feedback!
--
Ignacio Ocampo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Ignacio Ocampo
7:38 a.m.
For private projects, you can search small 1U servers with up to 4 3.5"
disk slots and some e3-1230 v3/4/5 cpu. They can be bought for 250-350€
(used) and then you just plug in a disk.
They are also good for SATA SSDs and work quite well. You can mix both
drives in the same system as well.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Fr., 2. Okt. 2020 um 08:32 Uhr schrieb Ignacio Ocampo <nafiux(a)gmail.com>om>:
Hi Brian,
Here more context about what I want to accomplish: I've migrated a bunch of
services from AWS to a local server, but having everything in a single
server is not safe, and instead of investing in RAID, I would like to start
setting up a small Ceph Cluster to have redundancy and a robust mechanism
in case any component fails.
Also, in the mid-term, I do have plans to deploy a small OpenStack Cluster.
Because of that, I would like to set up the first small Ceph Cluster that
can scale as my needs grow, the idea is to have 3 OSD nodes with the same
characteristics and add additional HDDs as needed, up to 5 HDD per OSD
node, starting with 1 HDD per node.
Thanks!
On Thu, Oct 1, 2020 at 11:35 AM Brian Topping <brian.topping(a)gmail.com>
wrote:
Welcome to Ceph!
I think better questions to start with are “what are your objectives in
your study?” Is it just seeing Ceph run with many disks, or are you
trying
to see how much performance you can get out of it
with distributed disk?
What is your budget? Do you want to try different combinations of storage
devices to learn how they differ in performance or do you just want to
jump
to the fastest things out there?
One often doesn’t need a bunch of machines to determine that Ceph is a
really versatile and robust solution. I pretty regularly deploy Ceph on a
single node using Kubernetes and Rook. Some would ask “why would one ever
do that, just use direct storage!”. The answer is when I want to expand a
cluster, I am willing to have traded initial performance overhead for
letting Ceph distribute data at a later date. And the overhead is far
lower
than one might think when there’s not a network
bottleneck to deal with.
I
do use direct storage on LVM when I have
distributed workloads such as
Kafka that abstract storage that a service instance depends on. It
doesn’t
make much sense in my mind for Kafka or Cassandra
to use Ceph because I
can
afford to lose nodes using those services.
In other words, Ceph is virtualized storage. You have likely come to it
because your workloads need to be able to come up anywhere on your
network
and reach that storage. How do you see those
workloads exercising the
capabilities of Ceph? That’s where your interesting use cases come from,
and can help you better decide what the best lab platform is to get
started.
Hope that helps, Brian
On Sep 29, 2020, at 12:44 AM, Ignacio Ocampo <nafiux(a)gmail.com> wrote:
Hi All :),
I would like to get your feedback about the components below to build a
PoC OSD Node (I will build 3 of these).
SSD for OS.
NVMe for cache.
HDD for storage.
The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
<image.png>
Thanks for your feedback!
--
Ignacio Ocampo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Ignacio Ocampo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:03 a.m.
What about the network cards? The motherboard I’m looking for has 2 x 10Gbe, with that and
the CPU frequency, I think the bottleneck will be the HDD. Is that overkill? Thanks!
Ignacio Ocampo
On 2 Oct 2020, at 0:38, Martin Verges
<martin.verges(a)croit.io> wrote:
For private projects, you can search small 1U servers with up to 4 3.5" disk slots
and some e3-1230 v3/4/5 cpu. They can be bought for 250-350€ (used) and then you just plug
in a disk.
They are also good for SATA SSDs and work quite well. You can mix both drives in the same
system as well.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
> Am Fr., 2. Okt. 2020 um 08:32 Uhr schrieb Ignacio Ocampo <nafiux(a)gmail.com>om>:
> Hi Brian,
>
> Here more context about what I want to accomplish: I've migrated a bunch of
> services from AWS to a local server, but having everything in a single
> server is not safe, and instead of investing in RAID, I would like to start
> setting up a small Ceph Cluster to have redundancy and a robust mechanism
> in case any component fails.
>
> Also, in the mid-term, I do have plans to deploy a small OpenStack Cluster.
>
> Because of that, I would like to set up the first small Ceph Cluster that
> can scale as my needs grow, the idea is to have 3 OSD nodes with the same
> characteristics and add additional HDDs as needed, up to 5 HDD per OSD
> node, starting with 1 HDD per node.
>
> Thanks!
>
> On Thu, Oct 1, 2020 at 11:35 AM Brian Topping <brian.topping(a)gmail.com>
> wrote:
>
> > Welcome to Ceph!
> >
> > I think better questions to start with are “what are your objectives in
> > your study?” Is it just seeing Ceph run with many disks, or are you trying
> > to see how much performance you can get out of it with distributed disk?
> > What is your budget? Do you want to try different combinations of storage
> > devices to learn how they differ in performance or do you just want to jump
> > to the fastest things out there?
> >
> > One often doesn’t need a bunch of machines to determine that Ceph is a
> > really versatile and robust solution. I pretty regularly deploy Ceph on a
> > single node using Kubernetes and Rook. Some would ask “why would one ever
> > do that, just use direct storage!”. The answer is when I want to expand a
> > cluster, I am willing to have traded initial performance overhead for
> > letting Ceph distribute data at a later date. And the overhead is far lower
> > than one might think when there’s not a network bottleneck to deal with. I
> > do use direct storage on LVM when I have distributed workloads such as
> > Kafka that abstract storage that a service instance depends on. It doesn’t
> > make much sense in my mind for Kafka or Cassandra to use Ceph because I can
> > afford to lose nodes using those services.
> >
> > In other words, Ceph is virtualized storage. You have likely come to it
> > because your workloads need to be able to come up anywhere on your network
> > and reach that storage. How do you see those workloads exercising the
> > capabilities of Ceph? That’s where your interesting use cases come from,
> > and can help you better decide what the best lab platform is to get started.
> >
> > Hope that helps, Brian
> >
> > On Sep 29, 2020, at 12:44 AM, Ignacio Ocampo <nafiux(a)gmail.com> wrote:
> >
> > Hi All :),
> >
> > I would like to get your feedback about the components below to build a
> > PoC OSD Node (I will build 3 of these).
> >
> > SSD for OS.
> > NVMe for cache.
> > HDD for storage.
> >
> > The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
> >
> > <image.png>
> >
> > Thanks for your feedback!
> >
> > --
> > Ignacio Ocampo
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> >
> >
>
> --
> Ignacio Ocampo
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
4 Oct
4 Oct
5:31 p.m.
Hi Ignacio, apologies I missed your responses here.
I would agree with Martin about buying used hardware for as cheap as possible, but also
understand the desire to have hardware you can promote into future OpenStack usage.
Regarding networking, I started to use SFP+ cables like https://amzn.to/36sHZo1. These run
with less energy consumption, so are far cooler, and can be easily swapped out with Fiber
or 10GBase-T modules if / when I need them. https://amzn.to/34s6Ohj is the switch that I
am currently using and will be moving to a SONiC based EdgeCore when I get some time to
fix the crashes i’m having in the software. If the motherboards you found are built with
10GBase-T, it may be cheaper now to just stick with that in your switch, but long term,
you’ll probably save money going with SFP+, so it’s worth considering.
Regarding the MikroTik switch, I bought it on price since I thought I was just getting a
switch. I turns out the thing is a full-fledged router on the level of a Cisco Catalyst.
It has a very complete and mature CLI configuration language, as well as a web UI that is
a bit overwhelming at first but very useful once one gets the hang of it. I thought it was
something I’d be putting back on eBay pretty rapidly and I ended up growing quite fond of
it.
Regarding bottlenecks, there’s always going to be a bottleneck. Parts are never completely
balanced. Spindles still have good characteristics for certain workloads, and I’d say it’s
doubtful you’d exceed them with a three node experimentation cluster. The question I’d ask
is whether you can use the spindles in a long term solution for things like log storage
and backups. If the answer is yes, then having them now will allow you to get a better
feel for the differences. It’s a lot easier to discover that way than it is to describe
it. (In the old days, this was like people comparing statistics about stereo equipment -
today people don’t care about all that, they care about whether they can find the song
they want and headphones are more than good enough…)
Brian
On Oct 2, 2020, at 2:03 AM, Ignacio Ocampo
<nafiux(a)gmail.com> wrote:
What about the network cards? The motherboard I’m looking for has 2 x 10Gbe, with that
and the CPU frequency, I think the bottleneck will be the HDD. Is that overkill? Thanks!
Ignacio Ocampo
> On 2 Oct 2020, at 0:38, Martin Verges <martin.verges(a)croit.io> wrote:
>
>
> For private projects, you can search small 1U servers with up to 4 3.5" disk
slots and some e3-1230 v3/4/5 cpu. They can be bought for 250-350€ (used) and then you
just plug in a disk.
> They are also good for SATA SSDs and work quite well. You can mix both drives in the
same system as well.
>
> --
> Martin Verges
> Managing director
>
> Mobile: +49 174 9335695
> E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io>
> Chat: https://t.me/MartinVerges <https://t.me/MartinVerges>
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
>
> Web: https://croit.io <https://croit.io/>
> YouTube: https://goo.gl/PGE1Bx <https://goo.gl/PGE1Bx>
>
>
> Am Fr., 2. Okt. 2020 um 08:32 Uhr schrieb Ignacio Ocampo <nafiux(a)gmail.com
<mailto:nafiux@gmail.com>>:
> Hi Brian,
>
> Here more context about what I want to accomplish: I've migrated a bunch of
> services from AWS to a local server, but having everything in a single
> server is not safe, and instead of investing in RAID, I would like to start
> setting up a small Ceph Cluster to have redundancy and a robust mechanism
> in case any component fails.
>
> Also, in the mid-term, I do have plans to deploy a small OpenStack Cluster.
>
> Because of that, I would like to set up the first small Ceph Cluster that
> can scale as my needs grow, the idea is to have 3 OSD nodes with the same
> characteristics and add additional HDDs as needed, up to 5 HDD per OSD
> node, starting with 1 HDD per node.
>
> Thanks!
>
> On Thu, Oct 1, 2020 at 11:35 AM Brian Topping <brian.topping(a)gmail.com
<mailto:brian.topping@gmail.com>>
> wrote:
>
> > Welcome to Ceph!
> >
> > I think better questions to start with are “what are your objectives in
> > your study?” Is it just seeing Ceph run with many disks, or are you trying
> > to see how much performance you can get out of it with distributed disk?
> > What is your budget? Do you want to try different combinations of storage
> > devices to learn how they differ in performance or do you just want to jump
> > to the fastest things out there?
> >
> > One often doesn’t need a bunch of machines to determine that Ceph is a
> > really versatile and robust solution. I pretty regularly deploy Ceph on a
> > single node using Kubernetes and Rook. Some would ask “why would one ever
> > do that, just use direct storage!”. The answer is when I want to expand a
> > cluster, I am willing to have traded initial performance overhead for
> > letting Ceph distribute data at a later date. And the overhead is far lower
> > than one might think when there’s not a network bottleneck to deal with. I
> > do use direct storage on LVM when I have distributed workloads such as
> > Kafka that abstract storage that a service instance depends on. It doesn’t
> > make much sense in my mind for Kafka or Cassandra to use Ceph because I can
> > afford to lose nodes using those services.
> >
> > In other words, Ceph is virtualized storage. You have likely come to it
> > because your workloads need to be able to come up anywhere on your network
> > and reach that storage. How do you see those workloads exercising the
> > capabilities of Ceph? That’s where your interesting use cases come from,
> > and can help you better decide what the best lab platform is to get started.
> >
> > Hope that helps, Brian
> >
> > On Sep 29, 2020, at 12:44 AM, Ignacio Ocampo <nafiux(a)gmail.com
<mailto:nafiux@gmail.com>> wrote:
> >
> > Hi All :),
> >
> > I would like to get your feedback about the components below to build a
> > PoC OSD Node (I will build 3 of these).
> >
> > SSD for OS.
> > NVMe for cache.
> > HDD for storage.
> >
> > The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
> >
> > <image.png>
> >
> > Thanks for your feedback!
> >
> > --
> > Ignacio Ocampo
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
> >
> >
> >
>
> --
> Ignacio Ocampo
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
8:27 p.m.
Hi Brian and Martin,
Physical space isn't a constraint at this point, the only requirement I've
in mind is to maintain a *low level of noise* (since the equipment will be
in my office) and *if possible low energy consumption*.
Based on my limited experience, the only downside with used hardware is the
level of noise and power consumption. I think that building a new one with
a non-fancy big box
<https://www.newegg.com/black-fractal-design-node-micro-atx-cube-case/p/N82E16811352047>
and a typical CPU fan <https://www.amazon.com/gp/product/B01KBXKP8W/> can
help. But I'm not quite convinced this is a good idea, I would rather
prefer to purchase old hardware that works and add new HDD, SSD, NVMe just
I don't know which one to choice.
*If you guys have any suggestions about used hardware that can be a good
fit considering mainly low noise, please let me know.*
Your feedback has been extremely helpful, I will actually modify the plan
to utilize *SFP+ instead of ethernet*, the MikroTik Router Switch is
wonderful, I want it.
For the spindles (Are you referring to HDD right?) I selected a 6TB HDD
(without any prior experience, I've read that it's like a good balance
between a big disk and a small disk, that will impact the recovery time),
my plan is to have 5 HDD per Node, with a total capacity of 30 TB per node.
Regarding the kind of workloads I've in mind, in order of priority are: *Block
Storage for OpenStack *(it will be a mix of apps: databases, web servers,
etc), *Object Storage* (to develop apps compatible with S3), I don't think
I will need File Storage at this point.
I know HDDs are slow and provide a low IOPS, but since I will have 15 HDD
spread across the 3-nodes Ceph cluster, I thought it will help a little bit
with the performance, and if needed, I will slowly increase the number of
OSD nodes to continue linearly scaling the performance and capacity of the
cluster. Should I consider enterprise-grade SSD instead of HDD from day 1?
Thanks for your support!
On Sun, Oct 4, 2020 at 10:31 AM Brian Topping <brian.topping(a)gmail.com>
wrote:
Hi Ignacio, apologies I missed your responses here.
I would agree with Martin about buying used hardware for as cheap as
possible, but also understand the desire to have hardware you can promote
into future OpenStack usage.
Regarding networking, I started to use SFP+ cables like
https://amzn.to/36sHZo1. These run with less energy consumption, so are
far cooler, and can be easily swapped out with Fiber or 10GBase-T modules
if / when I need them. https://amzn.to/34s6Ohj is the switch that I am
currently using and will be moving to a SONiC based EdgeCore when I get
some time to fix the crashes i’m having in the software. If the
motherboards you found are built with 10GBase-T, it may be cheaper now to
just stick with that in your switch, but long term, you’ll probably save
money going with SFP+, so it’s worth considering.
Regarding the MikroTik switch, I bought it on price since I thought I was
just getting a switch. I turns out the thing is a full-fledged router on
the level of a Cisco Catalyst. It has a very complete and mature CLI
configuration language, as well as a web UI that is a bit overwhelming at
first but very useful once one gets the hang of it. I thought it was
something I’d be putting back on eBay pretty rapidly and I ended up growing
quite fond of it.
Regarding bottlenecks, there’s always going to be a bottleneck. Parts are
never completely balanced. Spindles still have good characteristics for
certain workloads, and I’d say it’s doubtful you’d exceed them with a three
node experimentation cluster. The question I’d ask is whether you can use
the spindles in a long term solution for things like log storage and
backups. If the answer is yes, then having them now will allow you to get a
better feel for the differences. It’s a lot easier to discover that way
than it is to describe it. (In the old days, this was like people comparing
statistics about stereo equipment - today people don’t care about all that,
they care about whether they can find the song they want and headphones are
more than good enough…)
Brian
On Oct 2, 2020, at 2:03 AM, Ignacio Ocampo <nafiux(a)gmail.com> wrote:
What about the network cards? The motherboard I’m looking for has 2 x
10Gbe, with that and the CPU frequency, I think the bottleneck will be the
HDD. Is that overkill? Thanks!
*Ignacio Ocampo*
On 2 Oct 2020, at 0:38, Martin Verges <martin.verges(a)croit.io> wrote:
For private projects, you can search small 1U servers with up to 4 3.5"
disk slots and some e3-1230 v3/4/5 cpu. They can be bought for 250-350€
(used) and then you just plug in a disk.
They are also good for SATA SSDs and work quite well. You can mix both
drives in the same system as well.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Fr., 2. Okt. 2020 um 08:32 Uhr schrieb Ignacio Ocampo <nafiux(a)gmail.com
--
Ignacio Ocampo
:
Hi Brian,
Here more context about what I want to accomplish: I've migrated a bunch
of
services from AWS to a local server, but having everything in a single
server is not safe, and instead of investing in RAID, I would like to
start
setting up a small Ceph Cluster to have redundancy and a robust mechanism
in case any component fails.
Also, in the mid-term, I do have plans to deploy a small OpenStack
Cluster.
Because of that, I would like to set up the first small Ceph Cluster that
can scale as my needs grow, the idea is to have 3 OSD nodes with the same
characteristics and add additional HDDs as needed, up to 5 HDD per OSD
node, starting with 1 HDD per node.
Thanks!
On Thu, Oct 1, 2020 at 11:35 AM Brian Topping <brian.topping(a)gmail.com>
wrote:
Welcome to Ceph!
I think better questions to start with are “what are your objectives in
your study?” Is it just seeing Ceph run with many disks, or are you
trying
to see how much performance you can get out of it
with distributed disk?
What is your budget? Do you want to try different combinations of
storage
devices to learn how they differ in performance
or do you just want to
jump
to the fastest things out there?
One often doesn’t need a bunch of machines to determine that Ceph is a
really versatile and robust solution. I pretty regularly deploy Ceph on
a
single node using Kubernetes and Rook. Some would
ask “why would one
ever
do that, just use direct storage!”. The answer is
when I want to expand
a
cluster, I am willing to have traded initial
performance overhead for
letting Ceph distribute data at a later date. And the overhead is far
lower
than one might think when there’s not a network
bottleneck to deal
with. I
do use direct storage on LVM when I have
distributed workloads such as
Kafka that abstract storage that a service instance depends on. It
doesn’t
make much sense in my mind for Kafka or Cassandra
to use Ceph because I
can
afford to lose nodes using those services.
In other words, Ceph is virtualized storage. You have likely come to it
because your workloads need to be able to come up anywhere on your
network
and reach that storage. How do you see those
workloads exercising the
capabilities of Ceph? That’s where your interesting use cases come from,
and can help you better decide what the best lab platform is to get
started.
Hope that helps, Brian
On Sep 29, 2020, at 12:44 AM, Ignacio Ocampo <nafiux(a)gmail.com> wrote:
Hi All :),
I would like to get your feedback about the components below to build a
PoC OSD Node (I will build 3 of these).
SSD for OS.
NVMe for cache.
HDD for storage.
The Supermicro motherboard has 2 10Gb cards, and I will use ECC
memories.
<image.png>
Thanks for your feedback!
--
Ignacio Ocampo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Ignacio Ocampo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:07 p.m.
Comments inline
On Oct 4, 2020, at 2:27 PM, Ignacio Ocampo
<nafiux(a)gmail.com> wrote:
Physical space isn't a constraint at this point, the only requirement I've in
mind is to maintain a low level of noise (since the equipment will be in my office) and if
possible low energy consumption.
Based on my limited experience, the only downside with used hardware is the level of
noise and power consumption. I think that building a new one with a non-fancy big box
<https://www.newegg.com/black-fractal-design-node-micro-atx-cube-case/p/N82E16811352047>
and a typical CPU fan <https://www.amazon.com/gp/product/B01KBXKP8W/> can help. But
I'm not quite convinced this is a good idea, I would rather prefer to purchase old
hardware that works and add new HDD, SSD, NVMe just I don't know which one to choice.
If you guys have any suggestions about used hardware
that can be a good fit considering mainly low noise, please let me know.
So we didn’t get these requirements initially, there’s no way for us to help you when the
requirements aren’t available for us to consider, even if we had time to consider every
one of them for you.
As you might imagine this question lab kit comes up a lot, and we aren’t just answering it
for you, but also for others who might have the forethought to search the list as well.
Things are changing constantly as well. What’s “best” today might not be so tomorrow. We
all started somewhere, so it’s a pleasure to help, but there are always hidden
requirements and the folks on the list here are usually not working with the same
requirements. For instance, most aren’t concerned about noise because it’s in a big
insulated room. They won’t have much to contribute in that regard.
If noise is the primary requirement, I would probably get a bunch of Raspberry Pis and add
SSDs on the USB ports. I hope you see what I mean by now :-) There are too many tradeoffs
to help each person find a perfect fit.
Your feedback has been extremely helpful, I will
actually modify the plan to utilize SFP+ instead of ethernet, the MikroTik Router Switch
is wonderful, I want it.
For the spindles (Are you referring to HDD right?) I selected a 6TB HDD (without any
prior experience, I've read that it's like a good balance between a big disk and a
small disk, that will impact the recovery time), my plan is to have 5 HDD per Node, with a
total capacity of 30 TB per node.
Regarding the kind of workloads I've in mind, in order of priority are: Block Storage
for OpenStack (it will be a mix of apps: databases, web servers, etc), Object Storage (to
develop apps compatible with S3), I don't think I will need File Storage at this
point.
I know HDDs are slow and provide a low IOPS, but since I will have 15 HDD spread across
the 3-nodes Ceph cluster, I thought it will help a little bit with the performance, and if
needed, I will slowly increase the number of OSD nodes to continue linearly scaling the
performance and capacity of the cluster. Should I consider enterprise-grade SSD instead of
HDD from day 1?
Thanks for your support!
As earlier, if you are trying to learn about the behaviors of the devices, I would take a
mix of both, with not too many of any type that you regret it if you don’t use them. Can
you use 90TB of storage in the next two years? If not, I would buy less.
I have personally been working with Ceph for two years, but I am not (and never will be) a
storage professional at the level of most people on the list here. We all have different
professional focus, but the most important thing is putting *something* together and
playing with it first. If you plan on eventually buying three SSDs for your kit and what
you’ve currently decided on is going to cost $10,000, buying just the SSDs now and three
Raspberry Pis is a rounding error for throwaway kit if you decide to work with a different
shopping list after getting experience. It may also save you a lot of money if you catch
the change earlier than later.
Lastly, don’t underestimate learning about all this using KVM or some other virtualizer on
your laptop. Especially if you have skills in something like Ansible, deploying to new
hardware as you get experience and need is trivial – the scripts you develop will work
there too. In-memory networking between VMs is as fast as you can get. Feeling like you
want physical devices? Plug them into USB and map them to the VMs. Then start doing things
like breaking the network, removing disks and still recovering your data. That’s far more
important to know than how to plug together big devices.
You don’t need to spend a lot of money to gain this experience.
Brian
On Sun, Oct 4, 2020 at 10:31 AM Brian Topping <brian.topping(a)gmail.com
<mailto:brian.topping@gmail.com>> wrote:
Hi Ignacio, apologies I missed your responses here.
I would agree with Martin about buying used hardware for as cheap as possible, but also
understand the desire to have hardware you can promote into future OpenStack usage.
Regarding networking, I started to use SFP+ cables like https://amzn.to/36sHZo1
<https://amzn.to/36sHZo1>. These run with less energy consumption, so are far
cooler, and can be easily swapped out with Fiber or 10GBase-T modules if / when I need
them. https://amzn.to/34s6Ohj <https://amzn.to/34s6Ohj> is the switch that I am
currently using and will be moving to a SONiC based EdgeCore when I get some time to fix
the crashes i’m having in the software. If the motherboards you found are built with
10GBase-T, it may be cheaper now to just stick with that in your switch, but long term,
you’ll probably save money going with SFP+, so it’s worth considering.
Regarding the MikroTik switch, I bought it on price since I thought I was just getting a
switch. I turns out the thing is a full-fledged router on the level of a Cisco Catalyst.
It has a very complete and mature CLI configuration language, as well as a web UI that is
a bit overwhelming at first but very useful once one gets the hang of it. I thought it was
something I’d be putting back on eBay pretty rapidly and I ended up growing quite fond of
it.
Regarding bottlenecks, there’s always going to be a bottleneck. Parts are never
completely balanced. Spindles still have good characteristics for certain workloads, and
I’d say it’s doubtful you’d exceed them with a three node experimentation cluster. The
question I’d ask is whether you can use the spindles in a long term solution for things
like log storage and backups. If the answer is yes, then having them now will allow you to
get a better feel for the differences. It’s a lot easier to discover that way than it is
to describe it. (In the old days, this was like people comparing statistics about stereo
equipment - today people don’t care about all that, they care about whether they can find
the song they want and headphones are more than good enough…)
Brian
On Oct 2, 2020, at 2:03 AM, Ignacio Ocampo
<nafiux(a)gmail.com <mailto:nafiux@gmail.com>> wrote:
What about the network cards? The motherboard I’m looking for has 2 x 10Gbe, with that
and the CPU frequency, I think the bottleneck will be the HDD. Is that overkill? Thanks!
Ignacio Ocampo
> On 2 Oct 2020, at 0:38, Martin Verges <martin.verges(a)croit.io
<mailto:martin.verges@croit.io>> wrote:
>
>
> For private projects, you can search small 1U servers with up to 4 3.5" disk
slots and some e3-1230 v3/4/5 cpu. They can be bought for 250-350€ (used) and then you
just plug in a disk.
> They are also good for SATA SSDs and work quite well. You can mix both drives in the
same system as well.
>
> --
> Martin Verges
> Managing director
>
> Mobile: +49 174 9335695
> E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io>
> Chat: https://t.me/MartinVerges <https://t.me/MartinVerges>
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
>
> Web: https://croit.io <https://croit.io/>
> YouTube: https://goo.gl/PGE1Bx <https://goo.gl/PGE1Bx>
>
>
> Am Fr., 2. Okt. 2020 um 08:32 Uhr schrieb Ignacio Ocampo <nafiux(a)gmail.com
<mailto:nafiux@gmail.com>>:
> Hi Brian,
>
> Here more context about what I want to accomplish: I've migrated a bunch of
> services from AWS to a local server, but having everything in a single
> server is not safe, and instead of investing in RAID, I would like to start
> setting up a small Ceph Cluster to have redundancy and a robust mechanism
> in case any component fails.
>
> Also, in the mid-term, I do have plans to deploy a small OpenStack Cluster.
>
> Because of that, I would like to set up the first small Ceph Cluster that
> can scale as my needs grow, the idea is to have 3 OSD nodes with the same
> characteristics and add additional HDDs as needed, up to 5 HDD per OSD
> node, starting with 1 HDD per node.
>
> Thanks!
>
> On Thu, Oct 1, 2020 at 11:35 AM Brian Topping <brian.topping(a)gmail.com
<mailto:brian.topping@gmail.com>>
> wrote:
>
> > Welcome to Ceph!
> >
> > I think better questions to start with are “what are your objectives in
> > your study?” Is it just seeing Ceph run with many disks, or are you trying
> > to see how much performance you can get out of it with distributed disk?
> > What is your budget? Do you want to try different combinations of storage
> > devices to learn how they differ in performance or do you just want to jump
> > to the fastest things out there?
> >
> > One often doesn’t need a bunch of machines to determine that Ceph is a
> > really versatile and robust solution. I pretty regularly deploy Ceph on a
> > single node using Kubernetes and Rook. Some would ask “why would one ever
> > do that, just use direct storage!”. The answer is when I want to expand a
> > cluster, I am willing to have traded initial performance overhead for
> > letting Ceph distribute data at a later date. And the overhead is far lower
> > than one might think when there’s not a network bottleneck to deal with. I
> > do use direct storage on LVM when I have distributed workloads such as
> > Kafka that abstract storage that a service instance depends on. It doesn’t
> > make much sense in my mind for Kafka or Cassandra to use Ceph because I can
> > afford to lose nodes using those services.
> >
> > In other words, Ceph is virtualized storage. You have likely come to it
> > because your workloads need to be able to come up anywhere on your network
> > and reach that storage. How do you see those workloads exercising the
> > capabilities of Ceph? That’s where your interesting use cases come from,
> > and can help you better decide what the best lab platform is to get started.
> >
> > Hope that helps, Brian
> >
> > On Sep 29, 2020, at 12:44 AM, Ignacio Ocampo <nafiux(a)gmail.com
<mailto:nafiux@gmail.com>> wrote:
> >
> > Hi All :),
> >
> > I would like to get your feedback about the components below to build a
> > PoC OSD Node (I will build 3 of these).
> >
> > SSD for OS.
> > NVMe for cache.
> > HDD for storage.
> >
> > The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories.
> >
> > <image.png>
> >
> > Thanks for your feedback!
> >
> > --
> > Ignacio Ocampo
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
> >
> >
> >
>
> --
> Ignacio Ocampo
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
--
Ignacio Ocampo
11:01 p.m.
If you guys
have any suggestions about used hardware that can be a good fit considering mainly low
noise, please let me know.
So we didn’t get these requirements initially, there’s no way for us to help you when the
requirements aren’t available for us to consider, even if we had time to consider every
one of them for you.
11:25 p.m.
A cluster to serve actual workloads.
I think raspberry pi and Virtual machine are out of scope, this is not just for learning.
Thanks!
Ignacio Ocampo
On 4 Oct 2020, at 16:01, Anthony D'Atri
<anthony.datri(a)gmail.com> wrote:
I think this has gone down a bit of a rabbit hole. Perhaps it would be beneficial to
step back and get clear about what exactly the OP is trying to accomplish: a cluster to
just play around with, or a cluster to serve actual workloads.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
If you
guys have any suggestions about used hardware that can be a good fit considering mainly
low noise, please let me know.
So we didn’t get these requirements initially, there’s no way for us to help you when the
requirements aren’t available for us to consider, even if we had time to consider every
one of them for you.
1299
days inactive
1304
days old
12 comments
6 participants
participants (6)
-
Anthony D'Atri -
Brian Topping -
Ignacio Ocampo -
Martin Verges -
Nathan Fish -
Stefan Kooman