31 Jan
2020
31 Jan
'20
12:58 p.m.
Looking to role out a all flash Ceph cluster. Wanted to see if anyone else was using
Micron drives along with some basic input on my design so far?
Basic Config
Ceph OSD Nodes
8x Supermicro A+ Server 2113S-WTRT
- AMD EPYC 7601 32 Core 2.2Ghz
- 256G Ram
- AOC-S3008L-L8e HBA
- 10GB SFP+ for client network
- 40GB QSFP+ for ceph cluster network
OSD
10x Micron 5300 PRO 7.68TB in each ceph node
- 80 total drives across the 8 nodes
WAL/DB
5x Micron 7300 MAX NVMe 800GB per Ceph Node
- Plan on dedicating 1 for each 2 OSD's
Still thinking out a external monitor node as I have a lot of options, but this is a
pretty good start. Open to suggestions as well!
31 Jan
31 Jan
1:05 p.m.
Hmm change 40Gbps to 100Gbps networking.
40Gbps technology its just a bond of 4x10 Links with some latency due link aggregation.
100 Gbps and 25Gbps got less latency and Good performance. In ceph a 50% of the latency
comes from Network commits and the other 50% from disk commits.
A fast graph :
https://blog.mellanox.com/wp-content/uploads/John-Kim-030416-Fig-3a-1024x74…
Article:
https://blog.mellanox.com/2016/03/25-is-the-new-10-50-is-the-new-40-100-is-…
Micron got their own Whitepaper for CEPH and looks like performs fine.
https://www.micron.com/-/media/client/global/documents/products/other-docum…
AS your Budget is high, please buy 3 x 1.5K $ nodes for your monitors and you Will sleep
better. They just need 4 cores / 16GB RAM and 2x128GB SSD or NVME M2 .
-----Mensaje original-----
De: Adam Boyhan <adamb(a)medent.com>
Enviado el: viernes, 31 de enero de 2020 13:59
Para: ceph-users <ceph-users(a)ceph.io>
Asunto: [ceph-users] Micron SSD/Basic Config
Looking to role out a all flash Ceph cluster. Wanted to see if anyone else was using
Micron drives along with some basic input on my design so far?
Basic Config
Ceph OSD Nodes
8x Supermicro A+ Server 2113S-WTRT
- AMD EPYC 7601 32 Core 2.2Ghz
- 256G Ram
- AOC-S3008L-L8e HBA
- 10GB SFP+ for client network
- 40GB QSFP+ for ceph cluster network
OSD
10x Micron 5300 PRO 7.68TB in each ceph node
- 80 total drives across the 8 nodes
WAL/DB
5x Micron 7300 MAX NVMe 800GB per Ceph Node
- Plan on dedicating 1 for each 2 OSD's
Still thinking out a external monitor node as I have a lot of options, but this is a
pretty good start. Open to suggestions as well!
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
2:03 p.m.
Hello Adam,
Can you describe what performance values you want to gain out of your
cluster?
What's the use case?
EC oder Replica?
In general, more disks are preferred over bigger ones.
As Micron has not provided us with demo hardware, we can't say how fast
these disks are in reality. Before I think about 40 vs 25/50/100 GbE, I
would reduce latency of these disks.
--
Martin Verges
Managing director
Hint: Secure one of the last slots in the upcoming 4-day Ceph Intensive
Training at https://croit.io/training/4-days-ceph-in-depth-training.
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Fr., 31. Jan. 2020 um 13:58 Uhr schrieb Adam Boyhan <adamb(a)medent.com>om>:
Looking to role out a all flash Ceph cluster. Wanted
to see if anyone else
was using Micron drives along with some basic input on my design so far?
Basic Config
Ceph OSD Nodes
8x Supermicro A+ Server 2113S-WTRT
- AMD EPYC 7601 32 Core 2.2Ghz
- 256G Ram
- AOC-S3008L-L8e HBA
- 10GB SFP+ for client network
- 40GB QSFP+ for ceph cluster network
OSD
10x Micron 5300 PRO 7.68TB in each ceph node
- 80 total drives across the 8 nodes
WAL/DB
5x Micron 7300 MAX NVMe 800GB per Ceph Node
- Plan on dedicating 1 for each 2 OSD's
Still thinking out a external monitor node as I have a lot of options, but
this is a pretty good start. Open to suggestions as well!
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:39 p.m.
The use case is for KVM RBD volumes.
Our enviroment will be 80% random reads/writes probably 40/60 or 30/70 is a good estimate.
All 4k-8k IO sizes. We currently run on a Nimble Hybrid array which runs in the 5k-15k
IOPS range with some spikes up to 20-25k IOPS (Capable of 100k iops per Nimble).
Its also worth mentioning that I plan on splitting this cluster between our two sites. We
have a dedicated dark fiber connection which is 8x 10GB links between the two sites
(roughly .4 or .5 ms latency). I was leaning towards 40G because our existing HP 5900
switch's which we use to light our dark fiber also have 4x 40GB ports which I could
use to hang the Ceph cluster network from.
This would be a 4/2 setup, keeping 2 copies at each site.
I currently have a much lower end ceph cluster using all spinning rust with 3 nodes at
each site (48 OSD's). So far that is working out really well, but the data going on
that cluster is written than almost never touched again. The above spinning rust cluster
is running just simple 10G SFP+.
1551
days inactive
1551
days old
3 comments
4 participants
participants (4)
-
Adam Boyhan -
adamb＠medent.com
-
EDH - Manuel Rios -
Martin Verges