12 Sep
2019
12 Sep
'19
8:03 p.m.
Hi all,
The foundation has some money to spend on hardware. Given a budget of
~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at
the available machines to make a reasonable choice, since it's been a few
years now... it might make sense to buy something a bit faster. I think
ideally, though, they would go into the same machine pool/class, since
that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our
reliance on OVH cloud instances, which are averaging around $15k/month.
It'll be a larger initial outlay, but the ongoing cost would come down
(particularly since we're fortunate enough to have essentially free
datacenter space).
Thoughts?
sage
12 Sep
12 Sep
8:21 p.m.
On Thu, Sep 12, 2019 at 8:33 AM Sage Weil <sage(a)newdream.net> wrote:
Hi all,
The foundation has some money to spend on hardware. Given a budget of
~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at
the available machines to make a reasonable choice, since it's been a few
years now... it might make sense to buy something a bit faster. I think
ideally, though, they would go into the same machine pool/class, since
that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our
reliance on OVH cloud instances, which are averaging around $15k/month.
It'll be a larger initial outlay, but the ongoing cost would come down
(particularly since we're fortunate enough to have essentially free
datacenter space).
I'd also like to suggest developer build machines. Maybe this means
re-provisioning existing hardware. I don't think we need
state-of-the-art. The senta boxes are frequently out-of-space and
packages get changed up all the time by builds of
luminous/mimic/nautilus. I'd really like to see us with a private VM
cloud (openstack?) where we could use hardware more effectively and
not interfere with others' workspaces.
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
9:56 p.m.
On 9/12/19 8:51 AM, Patrick Donnelly wrote:
On Thu, Sep 12, 2019 at 8:33 AM Sage Weil
<sage(a)newdream.net> wrote:
For dev machines, we should be able to get some new ones to replace the
aging rex etc. in LA. I'll follow up with David about that.
on the existing senta, using the HDD (/slow) for build/{dev,out}
directories would help the space problem.
Hi all,
The foundation has some money to spend on hardware. Given a budget of
~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at
the available machines to make a reasonable choice, since it's been a few
years now... it might make sense to buy something a bit faster. I think
ideally, though, they would go into the same machine pool/class, since
that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our
reliance on OVH cloud instances, which are averaging around $15k/month.
It'll be a larger initial outlay, but the ongoing cost would come down
(particularly since we're fortunate enough to have essentially free
datacenter space).
I'd also like to suggest developer build machines. Maybe this means
re-provisioning existing hardware. I don't think we need
state-of-the-art. The senta boxes are frequently out-of-space and
packages get changed up all the time by builds of
luminous/mimic/nautilus. I'd really like to see us with a private VM
cloud (openstack?) where we could use hardware more effectively and
not interfere with others' workspaces. 
10:22 p.m.
Have we given up doing any real testing on hard drives in the lab?
The remaining mira nodes keep being cannibalized into the LRC or as
Jenkins slaves so the numbers there keep dropping, plus they're aging
out.
If we're not interested in that specifically (or maybe even if we
are), I think we should explore some more dynamic allocation of
machines into jenkins/build slaves and out of the pools before we
commit to buying hardware. We've got hundreds of servers that are
mostly statically assigned to tasks, and the ratios we're interested
in are continuing to vary as we build up more jenkins pipelines in
some areas of the code.
-Greg
On Thu, Sep 12, 2019 at 8:33 AM Sage Weil <sage(a)newdream.net> wrote:
>
> Hi all,
>
> The foundation has some money to spend on hardware. Given a budget of
> ~$200k, what would we buy?
>
> I see two main options:
>
> - More smithi-class machines. We would presumably take a closer look at
> the available machines to make a reasonable choice, since it's been a few
> years now... it might make sense to buy something a bit faster. I think
> ideally, though, they would go into the same machine pool/class, since
> that tends to lead to better overall utilization of the hardware.
>
> - Build machines. If we buy more beefy boxes, we could reduce our
> reliance on OVH cloud instances, which are averaging around $15k/month.
> It'll be a larger initial outlay, but the ongoing cost would come down
> (particularly since we're fortunate enough to have essentially free
> datacenter space).
>
> Thoughts?
> sage
> _______________________________________________
> Sepia mailing list -- sepia(a)ceph.io
> To unsubscribe send an email to sepia-leave(a)ceph.io
13 Sep
13 Sep
1:39 p.m.
On 19/09/12 03:33PM, Sage Weil wrote:
Hi all,
The foundation has some money to spend on hardware. Given a budget of
~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at
the available machines to make a reasonable choice, since it's been a few
years now... it might make sense to buy something a bit faster. I think
ideally, though, they would go into the same machine pool/class, since
that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our
reliance on OVH cloud instances, which are averaging around $15k/month.
It'll be a larger initial outlay, but the ongoing cost would come down
(particularly since we're fortunate enough to have essentially free
datacenter space).
Although more test machines in the lab are always good, it begs the question
of how much money would be saved by moving builds internally. All of those
15k? Just part of it?
If we could save up all those 15k/month, then build machines make more fiscal
sense I think. Delaying buying machines for the lab for about a year, and
instead investing in the build machines, would have us breaking even the
investment in about 14 months (assuming the load stays about the same).
-Joao
20 Sep
20 Sep
7:41 p.m.
On Thu, 12 Sep 2019, Gregory Farnum wrote:
Have we given up doing any real testing on hard drives
in the lab?
The remaining mira nodes keep being cannibalized into the LRC or as
Jenkins slaves so the numbers there keep dropping, plus they're aging
out.
I don't have a strong opinion here about whether we need to retain an HDD
pool for (presumably?) rados. FWIW I never run the rados suite on mira.
If we're not interested in that specifically (or
maybe even if we
are), I think we should explore some more dynamic allocation of
machines into jenkins/build slaves and out of the pools before we
commit to buying hardware. We've got hundreds of servers that are
mostly statically assigned to tasks, and the ratios we're interested
in are continuing to vary as we build up more jenkins pipelines in
some areas of the code.
I think this is an orthogonal concern that's blocked by time and effort to
fiddle with jenkins. Wnd we can't use these machines for the primary
teuthology pool because the performance and configurations are too
different, so making that investment won't reduce the delays we
now getting teuthology test suites to complete.
That said, yes, I think it would be great to do a pass over all of the old
stuff in the lab and figure out how to make better use of it!
sage
> -Greg
>
> On Thu, Sep 12, 2019 at 8:33 AM Sage Weil <sage(a)newdream.net> wrote:
> >
> > Hi all,
> >
> > The foundation has some money to spend on hardware. Given a budget of
> > ~$200k, what would we buy?
> >
> > I see two main options:
> >
> > - More smithi-class machines. We would presumably take a closer look at
> > the available machines to make a reasonable choice, since it's been a few
> > years now... it might make sense to buy something a bit faster. I think
> > ideally, though, they would go into the same machine pool/class, since
> > that tends to lead to better overall utilization of the hardware.
> >
> > - Build machines. If we buy more beefy boxes, we could reduce our
> > reliance on OVH cloud instances, which are averaging around $15k/month.
> > It'll be a larger initial outlay, but the ongoing cost would come down
> > (particularly since we're fortunate enough to have essentially free
> > datacenter space).
> >
> > Thoughts?
> > sage
> > _______________________________________________
> > Sepia mailing list -- sepia(a)ceph.io
> > To unsubscribe send an email to sepia-leave(a)ceph.io
>
>
7:43 p.m.
On Fri, 13 Sep 2019, Joao Eduardo Luis wrote:
On 19/09/12 03:33PM, Sage Weil wrote:
How about we price out both options and see what how the costs compare and
how much we could get of each... we can can do a meaningful cost/benefit
comparison. (I also wonder if new dedicated build machines could perform
better than the cloud instances we're using now and speed up builds a
bit...)
sage
Hi all,
The foundation has some money to spend on hardware. Given a budget of
~$200k, what would we buy?
I see two main options:
- More smithi-class machines. We would presumably take a closer look at
the available machines to make a reasonable choice, since it's been a few
years now... it might make sense to buy something a bit faster. I think
ideally, though, they would go into the same machine pool/class, since
that tends to lead to better overall utilization of the hardware.
- Build machines. If we buy more beefy boxes, we could reduce our
reliance on OVH cloud instances, which are averaging around $15k/month.
It'll be a larger initial outlay, but the ongoing cost would come down
(particularly since we're fortunate enough to have essentially free
datacenter space).
Although more test machines in the lab are always good, it begs the question
of how much money would be saved by moving builds internally. All of those
15k? Just part of it?
If we could save up all those 15k/month, then build machines make more fiscal
sense I think. Delaying buying machines for the lab for about a year, and
instead investing in the build machines, would have us breaking even the
investment in about 14 months (assuming the load stays about the same).
25 Sep
25 Sep
11:39 p.m.
On 9/20/19 8:11 AM, Sage Weil wrote:
On Thu, 12 Sep 2019, Gregory Farnum wrote:
I don't think we're seeing as many race conditions only turn up on mira
anymore. I'd be fine using the space for more fast nodes instead.
Have we given up doing any real testing on hard
drives in the lab?
The remaining mira nodes keep being cannibalized into the LRC or as
Jenkins slaves so the numbers there keep dropping, plus they're aging
out.
I don't have a strong opinion here about whether we need to retain an HDD
pool for (presumably?) rados. FWIW I never run the rados suite on mira.
1676
days inactive
1689
days old
7 comments
5 participants
participants (5)
-
Gregory Farnum -
Joao Eduardo Luis -
Josh Durgin -
Patrick Donnelly -
Sage Weil