Suitable 10G Switches for ceph storage - any recommendations?
Dear Ceph users, I am currently constructing a small hyperconverged Proxmox cluster with ceph as storage. So far I always had 3 nodes, which I directly linked together via 2 bonded 10G network interfaces for the Ceph storage, so I never needed any switching devices. This new cluster has more nodes, so I am considering using a 10G switch for the storage network. As I have no experience with such a setup, I wonder if there are any specific issues that I should think of (latency...)? As the whole cluster should be not too expensive, I am currently thinking of the following solution: 2* CRS317-1G-16s+RM switches: https://mikrotik.com/product/crs317_1g_16s_rm#fndtn-testresults SFP+ Cables like these: https://www.fs.com/de/products/48883.html Some network interface for each node with two SFP+ ports, e.g.: https://ark.intel.com/content/www/de/de/ark/products/39776/intel-ethernet-co... Connect each port with each switch and configure master/slave configuration so that the switches are redundant. What do you think of this setup - or is there any information / recommendation for an optimized setup of a 10G storage network? Best Regards, Hermann -- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers)
The CRS3xx series is no-frills but works. I have a 6-node cluster running just fine on them. Bonding should work also for 2x10G. Interestingly enough, MLAG was introduced in today's routeros v7 beta, but I'm skeptical that it will be stable yet. On Wed, May 19, 2021 at 1:22 AM Hermann Himmelbauer <hermann@qwer.tk> wrote:
Dear Ceph users, I am currently constructing a small hyperconverged Proxmox cluster with ceph as storage. So far I always had 3 nodes, which I directly linked together via 2 bonded 10G network interfaces for the Ceph storage, so I never needed any switching devices.
This new cluster has more nodes, so I am considering using a 10G switch for the storage network. As I have no experience with such a setup, I wonder if there are any specific issues that I should think of (latency...)?
As the whole cluster should be not too expensive, I am currently thinking of the following solution:
2* CRS317-1G-16s+RM switches: https://mikrotik.com/product/crs317_1g_16s_rm#fndtn-testresults
SFP+ Cables like these: https://www.fs.com/de/products/48883.html
Some network interface for each node with two SFP+ ports, e.g.:
https://ark.intel.com/content/www/de/de/ark/products/39776/intel-ethernet-co...
Connect each port with each switch and configure master/slave configuration so that the switches are redundant.
What do you think of this setup - or is there any information / recommendation for an optimized setup of a 10G storage network?
Best Regards, Hermann
-- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jeremy Austin jhaustin@gmail.com
participants (2)
-
Hermann Himmelbauer
-
Jeremy Austin