Hardware feedback before purchasing for a PoC
Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase: The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*. System configuration: https://pcpartpicker.com/list/cfDpDx ($400 USD per node) Some notes about the configuration: - 4-core processor for 4 OSD daemons - 8GB RAM for the first 4TB of storage, that will increase to 16GB of RAM when 16TB of storage. - Motherboard: - 4 x SATA 6 Gb/s (one per each OSD disk) - 2 x PCI-E x1 Slots (1 will be used for an additional Gigabit Ethernet) - 1 x M.2 Slots for the host OS - Ram can increase up-to 32 GB, and another SATA 6b/s controller can be added on PCI-E x1 for growth up to *32TB* As noted, the plan is to deploy nodes with *4TB* and gradually add *12TB* as needed, memory also should be increased to *16GB* after *8TB* threshold. Edit Questions to validate before the purchase 1. Does the hardware components make sense for the *16TB* growth projection? 2. Is it easy to gradually add more capacity to each node (*4TB* each time per node)? Thanks for your support! -- Ignacio Ocampo
Hola Ignacio, El 9/3/20 a las 3:00, Ignacio Ocampo escribió:
Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase:
The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*.
System configuration: https://pcpartpicker.com/list/cfDpDx ($400 USD per node)
Some notes about the configuration:
- 4-core processor for 4 OSD daemons - 8GB RAM for the first 4TB of storage, that will increase to 16GB of RAM when 16TB of storage. - Motherboard: - 4 x SATA 6 Gb/s (one per each OSD disk) - 2 x PCI-E x1 Slots (1 will be used for an additional Gigabit Ethernet) - 1 x M.2 Slots for the host OS - Ram can increase up-to 32 GB, and another SATA 6b/s controller can be added on PCI-E x1 for growth up to *32TB*
As noted, the plan is to deploy nodes with *4TB* and gradually add *12TB* as needed, memory also should be increased to *16GB* after *8TB* threshold. Edit Questions to validate before the purchase
1. Does the hardware components make sense for the *16TB* growth projection?
2. Is it easy to gradually add more capacity to each node (*4TB* each time per node)? Thanks for your support!
You may find that having only one disk per node, will make the storage quite slow. I think you're low on RAM to use bluestore, specially when adding new disks, please check: https://docs.ceph.com/docs/master/start/hardware-recommendations/ It is easy to add additional disks, yes. You may also want to try some solution for easier deployment for such a small cluster, I can recommend Proxmox VE (https://www.proxmox.com) but there are others. Cheers and good luck Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es
Hi Eneko, I was considering 1 GB per TB, but I will switch that to 4 GB per TB to consider blue storage. Regarding the number of devices, Will 2 disks per node help with the cluster speed? My ultimate goal is to provide support to a second project which will be an OpenStack deployment. Oliver Audrey, I prefer to invest in hardware rather than renting it, since I want to create a private cloud. Thanks! Ignacio Ocampo
On Mar 9, 2020, at 4:12 AM, Eneko Lacunza <elacunza@binovo.es> wrote: Hola Ignacio,
El 9/3/20 a las 3:00, Ignacio Ocampo escribió: Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase:
The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*.
System configuration: https://pcpartpicker.com/list/cfDpDx ($400 USD per node)
Some notes about the configuration:
- 4-core processor for 4 OSD daemons - 8GB RAM for the first 4TB of storage, that will increase to 16GB of RAM when 16TB of storage. - Motherboard: - 4 x SATA 6 Gb/s (one per each OSD disk) - 2 x PCI-E x1 Slots (1 will be used for an additional Gigabit Ethernet) - 1 x M.2 Slots for the host OS - Ram can increase up-to 32 GB, and another SATA 6b/s controller can be added on PCI-E x1 for growth up to *32TB*
As noted, the plan is to deploy nodes with *4TB* and gradually add *12TB* as needed, memory also should be increased to *16GB* after *8TB* threshold. Edit Questions to validate before the purchase
1. Does the hardware components make sense for the *16TB* growth projection?
2. Is it easy to gradually add more capacity to each node (*4TB* each time per node)? Thanks for your support!
You may find that having only one disk per node, will make the storage quite slow.
I think you're low on RAM to use bluestore, specially when adding new disks, please check: https://docs.ceph.com/docs/master/start/hardware-recommendations/
It is easy to add additional disks, yes.
You may also want to try some solution for easier deployment for such a small cluster, I can recommend Proxmox VE (https://www.proxmox.com) but there are others.
Cheers and good luck Eneko
-- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Ignacio, El 9/3/20 a las 15:19, Ignacio Ocampo escribió:
I was considering 1 GB per TB, but I will switch that to 4 GB per TB to consider blue storage. Much better. :)
Regarding the number of devices, Will 2 disks per node help with the cluster speed? Yes, it will help. Generally speaking, more disks=better performance. You may consider using the server's OS m.2 slot disk for Bluestore's db to improve performance; if you do so, make sure to use enterprise SSDs (like Intel D3-S4510 for example). You may need to get a better AM4 CPU too, specially if using SSD for Bluestore:
https://cpu.userbenchmark.com/Compare/AMD-Ryzen-5-2600-vs-AMD-A8-9600-APU-20... https://cpu.userbenchmark.com/Compare/AMD-Ryzen-5-3600-vs-AMD-A8-9600-APU-20...
My ultimate goal is to provide support to a second project which will be an OpenStack deployment.
It will be a small deployment right? :-) Because the config you referenced is quite underpowered. How many VMs? What level of IO? (IO per second?) You'll be able to perform about 100-150 IOPS with one HDD per server... Cheers Eneko
Oliver Audrey, I prefer to invest in hardware rather than renting it, since I want to create a private cloud.
Thanks!
*Ignacio Ocampo*
On Mar 9, 2020, at 4:12 AM, Eneko Lacunza <elacunza@binovo.es> wrote:
Hola Ignacio,
El 9/3/20 a las 3:00, Ignacio Ocampo escribió:
Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase:
The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*.
System configuration: https://pcpartpicker.com/list/cfDpDx ($400 USD per node)
Some notes about the configuration:
- 4-core processor for 4 OSD daemons - 8GB RAM for the first 4TB of storage, that will increase to 16GB of RAM when 16TB of storage. - Motherboard: - 4 x SATA 6 Gb/s (one per each OSD disk) - 2 x PCI-E x1 Slots (1 will be used for an additional Gigabit Ethernet) - 1 x M.2 Slots for the host OS - Ram can increase up-to 32 GB, and another SATA 6b/s controller can be added on PCI-E x1 for growth up to *32TB*
As noted, the plan is to deploy nodes with *4TB* and gradually add *12TB* as needed, memory also should be increased to *16GB* after *8TB* threshold. Edit Questions to validate before the purchase
1. Does the hardware components make sense for the *16TB* growth projection?
2. Is it easy to gradually add more capacity to each node (*4TB* each time per node)? Thanks for your support!
You may find that having only one disk per node, will make the storage quite slow.
I think you're low on RAM to use bluestore, specially when adding new disks, please check: https://docs.ceph.com/docs/master/start/hardware-recommendations/
It is easy to add additional disks, yes.
You may also want to try some solution for easier deployment for such a small cluster, I can recommend Proxmox VE (https://www.proxmox.com) but there are others.
Cheers and good luck Eneko
-- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es
hello if it's only for a poc you can try to rent devices for some month. for less that 100€ per month you can get this kind of device at ovh.com - 32go RAM - Intel Xeon-E 2274G - 4 c/8 t - 4 GHz/4.9 GHz - 3× 4 To HDD SATA Soft RAID or 2× 960 Go SSD NVMe Soft RAID - 2 x 2Gbps network Personnaly I use bunch of them for my kubernetes prod (the one with 2× 960 Go SSD NVMe) and it's working pretty well. You can tweak to get more or different disk setup, more ram, more network speed. Everything is here : https://www.ovh.com/fr/serveurs_dedies/infra/tarifs/ The device I'm talking about is INFRA-1 oau Le dimanche 08 mars 2020 à 19:00 -0700, Ignacio Ocampo a écrit :
Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase:
The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*.
System configuration: https://pcpartpicker.com/list/cfDpDx ($400 USD per node)
Some notes about the configuration:
- 4-core processor for 4 OSD daemons - 8GB RAM for the first 4TB of storage, that will increase to 16GB of RAM when 16TB of storage. - Motherboard: - 4 x SATA 6 Gb/s (one per each OSD disk) - 2 x PCI-E x1 Slots (1 will be used for an additional Gigabit Ethernet) - 1 x M.2 Slots for the host OS - Ram can increase up-to 32 GB, and another SATA 6b/s controller can be added on PCI-E x1 for growth up to *32TB*
As noted, the plan is to deploy nodes with *4TB* and gradually add *12TB* as needed, memory also should be increased to *16GB* after *8TB* threshold. Edit Questions to validate before the purchase
1. Does the hardware components make sense for the *16TB* growth projection?
2. Is it easy to gradually add more capacity to each node (*4TB* each time per node)? Thanks for your support!
Ignacio; Personally, I like to use hardware for a proof of concept that I can roll over into the final system, or repurpose if the project is denied. As such, I would recommend these: Supermicro 5019A-12TN4 Barebones<https://www.supermicro.com/products/system/1U/5019/SYS-5019A-12TN4.cfm> I built our PoC around three of them (2 x 12TB Seagate Ironwolf drives, plus 1x 256 GB Intel M.2 SSDs), then they got turned into MONs for the production cluster. The production cluster ended up with larger M.2s, and smaller spinners, as I was concerned about recovery time for 24 - 36TB per node, with only 4 x 1Gb network. Just my 2 cents. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Ignacio Ocampo [mailto:nafiux@gmail.com] Sent: Sunday, March 08, 2020 7:00 PM To: ceph-users@ceph.io Subject: [ceph-users] Hardware feedback before purchasing for a PoC Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase: The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*. System configuration: https://pcpartpicker.com/list/cfDpDx ($400 USD per node) Some notes about the configuration: - 4-core processor for 4 OSD daemons - 8GB RAM for the first 4TB of storage, that will increase to 16GB of RAM when 16TB of storage. - Motherboard: - 4 x SATA 6 Gb/s (one per each OSD disk) - 2 x PCI-E x1 Slots (1 will be used for an additional Gigabit Ethernet) - 1 x M.2 Slots for the host OS - Ram can increase up-to 32 GB, and another SATA 6b/s controller can be added on PCI-E x1 for growth up to *32TB* As noted, the plan is to deploy nodes with *4TB* and gradually add *12TB* as needed, memory also should be increased to *16GB* after *8TB* threshold. Edit Questions to validate before the purchase 1. Does the hardware components make sense for the *16TB* growth projection? 2. Is it easy to gradually add more capacity to each node (*4TB* each time per node)? Thanks for your support! -- Ignacio Ocampo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
DHilsbos＠performair.com
-
Eneko Lacunza
-
Ignacio Ocampo
-
Olivier AUDRY