small cluster HW upgrade
We have a small ceph cluster running built from components that were phased out from compute applications. the current cluster consists of i7-860s. 6 disks (5TB, 7200RPM) per node and 8 nodes totaling 48 OSDs. A compute cluster will be discontinued, which will make Ryzen 5-1600 hardware available (8 nodes with 16GB RAM each) with which to replace the CPUs of the current setup. How could we best distribute the OSDs (keeping existing disks for storage) to the Ryzen systems to get a good performance improvement? Unfortunately the interconnect is still only 1Gb/s so expected to be a limiting factor. Would it make sense to create fewer bigger nodes, e.g., to use 6 nodes with 8 disks each or even more condensed? We would like to move the Luminous cluster to Nautilus/Bluestore and can get SSDs for each of the nodes, as it appears to be essential to get performance. Can actually benefit from improvements in the OSDs if the network is so limited? Would bonding of network interfaces be a workaround until we can get a network update or are we overestimating the power of the upgraded OSD nodes? What strategy would you suggest with these resources? Any comments and suggestions would be highly welcome :) Thanks in advance Philipp
Hi Philipp, More nodes is better, more availability, more CPU and more RAM. But, I'm agree that your 1GbE link will be most limiting factor, especially if there are some SSDs. I suggest you upgrade your networking to 10GbE (or 25GbE since it will cost you nearly same with 10GbE). Upgrading your networking is better than using bonding since bonding cannot have 100% of total links bandwidth. Best regards,
Osd's do not even use bonding effenciently. If it were to use 2 links concurrently it would be a lot better. https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html -----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Re: small cluster HW upgrade Hi Philipp, More nodes is better, more availability, more CPU and more RAM. But, I'm agree that your 1GbE link will be most limiting factor, especially if there are some SSDs. I suggest you upgrade your networking to 10GbE (or 25GbE since it will cost you nearly same with 10GbE). Upgrading your networking is better than using bonding since bonding cannot have 100% of total links bandwidth. Best regards, _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This is a natural condition of bonding, it has little to do with ceph-osd. Make sure your hash policy is set appropriatelly, so that you even have a chance of using both links. https://support.packet.com/kb/articles/lacp-bonding The larger the set of destinations, the more likely you are to spread traffic across both links.
Osd's do not even use bonding effenciently. If it were to use 2 links concurrently it would be a lot better.
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html
-----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Re: small cluster HW upgrade
Hi Philipp,
More nodes is better, more availability, more CPU and more RAM. But, I'm agree that your 1GbE link will be most limiting factor, especially if there are some SSDs. I suggest you upgrade your networking to 10GbE (or 25GbE since it will cost you nearly same with 10GbE). Upgrading your networking is better than using bonding since bonding cannot have 100% of total links bandwidth.
Best regards, _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You can optimize ceph-osd for this of course. It would benefit people that like to use the 1Gbit connections. I can understand putting time into it now does not make sense because of the availability of 10Gbit. However, I do not get why this was not optimized already 5 or 10 years ago. -----Original Message----- Cc: ceph-users; mrxlazuardin Subject: *****SPAM***** Re: [ceph-users] Re: small cluster HW upgrade This is a natural condition of bonding, it has little to do with ceph-osd. Make sure your hash policy is set appropriatelly, so that you even have a chance of using both links. https://support.packet.com/kb/articles/lacp-bonding The larger the set of destinations, the more likely you are to spread traffic across both links.
Osd's do not even use bonding effenciently. If it were to use 2 links concurrently it would be a lot better.
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg35474.html
-----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Re: small cluster HW upgrade
Hi Philipp,
More nodes is better, more availability, more CPU and more RAM. But, I'm agree that your 1GbE link will be most limiting factor, especially
if there are some SSDs. I suggest you upgrade your networking to 10GbE
(or 25GbE since it will cost you nearly same with 10GbE). Upgrading your networking is better than using bonding since bonding cannot have
100% of total links bandwidth.
Best regards, _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Marc Roos
-
mrxlazuardin＠gmail.com
-
Philipp Schwaha