On Fri, Jan 31, 2020 at 2:06 PM EDH - Manuel Rios <mriosfer@easydatahost.com> wrote:
Hmm change 40Gbps to 100Gbps networking.
40Gbps technology its just a bond of 4x10 Links with some latency due link aggregation. 100 Gbps and 25Gbps got less latency and Good performance. In ceph a 50% of the latency comes from Network commits and the other 50% from disk commits.
40G ethernet is not the same as 4x 10G bond. A bond load balances on a per-packet (or well, per flow usually) basis. A 40G link uses all four links even for a single packet. 100G is "just" 4x 25G I also wouldn't agree that network and disk latency is a 50/50 split in Ceph unless you have some NVRAM disks or something. Even for the network speed the processing and queuing in the network stack dominates over the serialization delay from a 40G/100G difference (4kb at 100G is 320ns, and 800ns at 40G for the serialization; I don't have any figures for processing times on 40/100G ethernet, but 10G fiber is at 300ns, 10G base-t at 2300 nanoseconds) Paul
A fast graph : https://blog.mellanox.com/wp-content/uploads/John-Kim-030416-Fig-3a-1024x747... Article: https://blog.mellanox.com/2016/03/25-is-the-new-10-50-is-the-new-40-100-is-t...
Micron got their own Whitepaper for CEPH and looks like performs fine. https://www.micron.com/-/media/client/global/documents/products/other-docume...
AS your Budget is high, please buy 3 x 1.5K $ nodes for your monitors and you Will sleep better. They just need 4 cores / 16GB RAM and 2x128GB SSD or NVME M2 .
-----Mensaje original----- De: Adam Boyhan <adamb@medent.com> Enviado el: viernes, 31 de enero de 2020 13:59 Para: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Micron SSD/Basic Config
Looking to role out a all flash Ceph cluster. Wanted to see if anyone else was using Micron drives along with some basic input on my design so far?
Basic Config Ceph OSD Nodes 8x Supermicro A+ Server 2113S-WTRT - AMD EPYC 7601 32 Core 2.2Ghz - 256G Ram - AOC-S3008L-L8e HBA - 10GB SFP+ for client network - 40GB QSFP+ for ceph cluster network
OSD 10x Micron 5300 PRO 7.68TB in each ceph node - 80 total drives across the 8 nodes
WAL/DB 5x Micron 7300 MAX NVMe 800GB per Ceph Node - Plan on dedicating 1 for each 2 OSD's
Still thinking out a external monitor node as I have a lot of options, but this is a pretty good start. Open to suggestions as well!
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io