Hi all, The foundation has some money to spend on hardware. Given a budget of ~$200k, what would we buy? I see two main options: - More smithi-class machines. We would presumably take a closer look at the available machines to make a reasonable choice, since it's been a few years now... it might make sense to buy something a bit faster. I think ideally, though, they would go into the same machine pool/class, since that tends to lead to better overall utilization of the hardware. - Build machines. If we buy more beefy boxes, we could reduce our reliance on OVH cloud instances, which are averaging around $15k/month. It'll be a larger initial outlay, but the ongoing cost would come down (particularly since we're fortunate enough to have essentially free datacenter space). Thoughts? sage
On Thu, Sep 12, 2019 at 8:33 AM Sage Weil <sage@newdream.net> wrote:
Hi all,
The foundation has some money to spend on hardware. Given a budget of ~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at the available machines to make a reasonable choice, since it's been a few years now... it might make sense to buy something a bit faster. I think ideally, though, they would go into the same machine pool/class, since that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our reliance on OVH cloud instances, which are averaging around $15k/month. It'll be a larger initial outlay, but the ongoing cost would come down (particularly since we're fortunate enough to have essentially free datacenter space).
I'd also like to suggest developer build machines. Maybe this means re-provisioning existing hardware. I don't think we need state-of-the-art. The senta boxes are frequently out-of-space and packages get changed up all the time by builds of luminous/mimic/nautilus. I'd really like to see us with a private VM cloud (openstack?) where we could use hardware more effectively and not interfere with others' workspaces. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 9/12/19 8:51 AM, Patrick Donnelly wrote:
On Thu, Sep 12, 2019 at 8:33 AM Sage Weil <sage@newdream.net> wrote:
Hi all,
The foundation has some money to spend on hardware. Given a budget of ~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at the available machines to make a reasonable choice, since it's been a few years now... it might make sense to buy something a bit faster. I think ideally, though, they would go into the same machine pool/class, since that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our reliance on OVH cloud instances, which are averaging around $15k/month. It'll be a larger initial outlay, but the ongoing cost would come down (particularly since we're fortunate enough to have essentially free datacenter space).
I'd also like to suggest developer build machines. Maybe this means re-provisioning existing hardware. I don't think we need state-of-the-art. The senta boxes are frequently out-of-space and packages get changed up all the time by builds of luminous/mimic/nautilus. I'd really like to see us with a private VM cloud (openstack?) where we could use hardware more effectively and not interfere with others' workspaces.
For dev machines, we should be able to get some new ones to replace the aging rex etc. in LA. I'll follow up with David about that. on the existing senta, using the HDD (/slow) for build/{dev,out} directories would help the space problem.
Have we given up doing any real testing on hard drives in the lab? The remaining mira nodes keep being cannibalized into the LRC or as Jenkins slaves so the numbers there keep dropping, plus they're aging out. If we're not interested in that specifically (or maybe even if we are), I think we should explore some more dynamic allocation of machines into jenkins/build slaves and out of the pools before we commit to buying hardware. We've got hundreds of servers that are mostly statically assigned to tasks, and the ratios we're interested in are continuing to vary as we build up more jenkins pipelines in some areas of the code. -Greg On Thu, Sep 12, 2019 at 8:33 AM Sage Weil <sage@newdream.net> wrote:
Hi all,
The foundation has some money to spend on hardware. Given a budget of ~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at the available machines to make a reasonable choice, since it's been a few years now... it might make sense to buy something a bit faster. I think ideally, though, they would go into the same machine pool/class, since that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our reliance on OVH cloud instances, which are averaging around $15k/month. It'll be a larger initial outlay, but the ongoing cost would come down (particularly since we're fortunate enough to have essentially free datacenter space).
Thoughts? sage _______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
On Thu, 12 Sep 2019, Gregory Farnum wrote:
Have we given up doing any real testing on hard drives in the lab? The remaining mira nodes keep being cannibalized into the LRC or as Jenkins slaves so the numbers there keep dropping, plus they're aging out.
I don't have a strong opinion here about whether we need to retain an HDD pool for (presumably?) rados. FWIW I never run the rados suite on mira.
If we're not interested in that specifically (or maybe even if we are), I think we should explore some more dynamic allocation of machines into jenkins/build slaves and out of the pools before we commit to buying hardware. We've got hundreds of servers that are mostly statically assigned to tasks, and the ratios we're interested in are continuing to vary as we build up more jenkins pipelines in some areas of the code.
I think this is an orthogonal concern that's blocked by time and effort to fiddle with jenkins. Wnd we can't use these machines for the primary teuthology pool because the performance and configurations are too different, so making that investment won't reduce the delays we now getting teuthology test suites to complete. That said, yes, I think it would be great to do a pass over all of the old stuff in the lab and figure out how to make better use of it! sage
-Greg
On Thu, Sep 12, 2019 at 8:33 AM Sage Weil <sage@newdream.net> wrote:
Hi all,
The foundation has some money to spend on hardware. Given a budget of ~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at the available machines to make a reasonable choice, since it's been a few years now... it might make sense to buy something a bit faster. I think ideally, though, they would go into the same machine pool/class, since that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our reliance on OVH cloud instances, which are averaging around $15k/month. It'll be a larger initial outlay, but the ongoing cost would come down (particularly since we're fortunate enough to have essentially free datacenter space).
Thoughts? sage _______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
On 9/20/19 8:11 AM, Sage Weil wrote:
On Thu, 12 Sep 2019, Gregory Farnum wrote:
Have we given up doing any real testing on hard drives in the lab? The remaining mira nodes keep being cannibalized into the LRC or as Jenkins slaves so the numbers there keep dropping, plus they're aging out.
I don't have a strong opinion here about whether we need to retain an HDD pool for (presumably?) rados. FWIW I never run the rados suite on mira.
I don't think we're seeing as many race conditions only turn up on mira anymore. I'd be fine using the space for more fast nodes instead.
On 19/09/12 03:33PM, Sage Weil wrote:
Hi all,
The foundation has some money to spend on hardware. Given a budget of ~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at the available machines to make a reasonable choice, since it's been a few years now... it might make sense to buy something a bit faster. I think ideally, though, they would go into the same machine pool/class, since that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our reliance on OVH cloud instances, which are averaging around $15k/month. It'll be a larger initial outlay, but the ongoing cost would come down (particularly since we're fortunate enough to have essentially free datacenter space).
Although more test machines in the lab are always good, it begs the question of how much money would be saved by moving builds internally. All of those 15k? Just part of it? If we could save up all those 15k/month, then build machines make more fiscal sense I think. Delaying buying machines for the lab for about a year, and instead investing in the build machines, would have us breaking even the investment in about 14 months (assuming the load stays about the same). -Joao
On Fri, 13 Sep 2019, Joao Eduardo Luis wrote:
On 19/09/12 03:33PM, Sage Weil wrote:
Hi all,
The foundation has some money to spend on hardware. Given a budget of ~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at the available machines to make a reasonable choice, since it's been a few years now... it might make sense to buy something a bit faster. I think ideally, though, they would go into the same machine pool/class, since that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our reliance on OVH cloud instances, which are averaging around $15k/month. It'll be a larger initial outlay, but the ongoing cost would come down (particularly since we're fortunate enough to have essentially free datacenter space).
Although more test machines in the lab are always good, it begs the question of how much money would be saved by moving builds internally. All of those 15k? Just part of it?
If we could save up all those 15k/month, then build machines make more fiscal sense I think. Delaying buying machines for the lab for about a year, and instead investing in the build machines, would have us breaking even the investment in about 14 months (assuming the load stays about the same).
How about we price out both options and see what how the costs compare and how much we could get of each... we can can do a meaningful cost/benefit comparison. (I also wonder if new dedicated build machines could perform better than the cloud instances we're using now and speed up builds a bit...) sage
participants (5)
-
Gregory Farnum
-
Joao Eduardo Luis
-
Josh Durgin
-
Patrick Donnelly
-
Sage Weil