31 Jan
2020
31 Jan
'20
1:59 p.m.
I think 800 GB NVMe per 2 SSDs is an overkill. 1 OSD usually only
requires 30 GB block.db, so 400 GB per an OSD is a lot. On the other
hand, does 7300 have twice the iops of 5300? In fact, I'm not sure if a
7300 + 5300 OSD will perform better than just a 5300 OSD at all.
It would be interesting if you could benchmark & compare it though :)
> Hmm change 40Gbps to 100Gbps networking.
>
> 40Gbps technology its just a bond of 4x10 Links with some latency due
> link aggregation.
> 100 Gbps and 25Gbps got less latency and Good performance. In ceph a
> 50% of the latency comes from Network commits and the other 50% from
> disk commits.
>
> A fast graph :
> https://blog.mellanox.com/wp-content/uploads/John-Kim-030416-Fig-3a-1024x74…
> Article:
>
https://blog.mellanox.com/2016/03/25-is-the-new-10-50-is-the-new-40-100-is-…
>
> Micron got their own Whitepaper for CEPH and looks like performs fine.
>
https://www.micron.com/-/media/client/global/documents/products/other-docum…
>
>
> AS your Budget is high, please buy 3 x 1.5K $ nodes for your monitors
> and you Will sleep better. They just need 4 cores / 16GB RAM and
> 2x128GB SSD or NVME M2 .
>
> -----Mensaje original-----
> De: Adam Boyhan <adamb(a)medent.com>
> Enviado el: viernes, 31 de enero de 2020 13:59
> Para: ceph-users <ceph-users(a)ceph.io>
> Asunto: [ceph-users] Micron SSD/Basic Config
>
> Looking to role out a all flash Ceph cluster. Wanted to see if anyone
> else was using Micron drives along with some basic input on my design
> so far?
>
> Basic Config
> Ceph OSD Nodes
> 8x Supermicro A+ Server 2113S-WTRT
> - AMD EPYC 7601 32 Core 2.2Ghz
> - 256G Ram
> - AOC-S3008L-L8e HBA
> - 10GB SFP+ for client network
> - 40GB QSFP+ for ceph cluster network
>
> OSD
> 10x Micron 5300 PRO 7.68TB in each ceph node
> - 80 total drives across the 8 nodes
>
> WAL/DB
> 5x Micron 7300 MAX NVMe 800GB per Ceph Node
> - Plan on dedicating 1 for each 2 OSD's
>
> Still thinking out a external monitor node as I have a lot of options,
> but this is a pretty good start. Open to suggestions as well!
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
31 Jan
31 Jan
2:14 p.m.
New subject: Micron SSD/Basic Config
vitalif＠yourcmc.ru wrote:
I think 800 GB NVMe per 2 SSDs is an overkill. 1 OSD
usually only
requires 30 GB block.db, so 400 GB per an OSD is a lot. On the other
hand, does 7300 have twice the iops of 5300? In fact, I'm not sure if a
7300 + 5300 OSD will perform better than just a 5300 OSD at all.
It would be interesting if you could benchmark & compare it though :)
The documentation I read said it was 4% of the block device. Also been told the rule of
thumb is basically 3/30/300.
The 7.68TB 5300 pro does 11k random write IOPS, the 800GB 7300 MAX NVMe does 60k random
write IOPS. The micron white paper is using 9200 MAX's with the 5210 SATA SSD's.
Only reason I am going for the 5300's is for a bit more write endurance.
2:22 p.m.
New subject: Micron SSD/Basic Config
The RocksDB rings are 256MB, 2.5GB, 25GB, and 250GB. Unless you have a workload that uses
a lot of metadata, taking care of the first 3 and providing room for compaction should be
fine. To allow for compaction room, 60GB should be sufficient. Add 4GB to accommodate
WAL and you're at a nice multiple of 2, 64GB.
David Byte
Sr. Technology Strategist
SCE Enterprise Linux
SCE Enterprise Storage
Alliances and SUSE Embedded
dbyte(a)suse.com
918.528.4422
On 1/31/20, 8:16 AM, "adamb(a)medent.com" <adamb(a)medent.com> wrote:
vitalif＠yourcmc.ru wrote:
I think 800 GB NVMe per 2 SSDs is an overkill. 1 OSD
usually only
requires 30 GB block.db, so 400 GB per an OSD is a lot. On the other
hand, does 7300 have twice the iops of 5300? In fact, I'm not sure if a
7300 + 5300 OSD will perform better than just a 5300 OSD at all.
It would be interesting if you could benchmark & compare it though :)
The documentation I read said it was 4% of the block device. Also been told the rule
of thumb is basically 3/30/300.
The 7.68TB 5300 pro does 11k random write IOPS, the 800GB 7300 MAX NVMe does 60k
random write IOPS. The micron white paper is using 9200 MAX's with the 5210 SATA
SSD's. Only reason I am going for the 5300's is for a bit more write endurance.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1572
days inactive
1572
days old
2 comments
3 participants
participants (3)
-
adamb＠medent.com -
David Byte -
vitalif＠yourcmc.ru