Hi everyone, I'd really like to push this through and get something working this week. After poking around it seems clear that there aren't any other registries we should be using for our temp/test builds (unless we just spam dockerhub, but that seems unwise). So, let's just get over ourselves and run our own registry. 1- Where to put it? I'm assuming this should go in the same place that chacra is putting our other temp builds. This is in RDU, right? What machine(s) should we use? If we use the same retention policy as the debs/rpms then this will be an incremental increase in the storage needed. 2- What registry software to use? We don't need any fancy featuers whatsoever--just the ability to push, pull, and delete images. So, whatever is easiest to set up. 3- Jenkins integration. I think we need to have a child job linked to the centos build to do the ceph-container build and then push. Similarly, whatever it is that removes the old packages from the repos needs to also delete the image. 4- Chacra/shaman integration? Should the container build show up in shaman/chacra as well? Is there extra work needed to do that? Thanks! sage
+Dimitri Savineau for ceph-container related questions. I think it'd be nice to have the registry public so that everyone can access and test a build. Thanks! ––––––––– Sébastien Han Principal Software Engineer, Storage Architect "Always give 100%. Unless you're giving blood." On Mon, Jun 17, 2019 at 5:34 PM Sage Weil <sweil@redhat.com> wrote:
Hi everyone,
I'd really like to push this through and get something working this week. After poking around it seems clear that there aren't any other registries we should be using for our temp/test builds (unless we just spam dockerhub, but that seems unwise). So, let's just get over ourselves and run our own registry.
1- Where to put it? I'm assuming this should go in the same place that chacra is putting our other temp builds. This is in RDU, right? What machine(s) should we use? If we use the same retention policy as the debs/rpms then this will be an incremental increase in the storage needed.
2- What registry software to use? We don't need any fancy featuers whatsoever--just the ability to push, pull, and delete images. So, whatever is easiest to set up.
3- Jenkins integration. I think we need to have a child job linked to the centos build to do the ceph-container build and then push. Similarly, whatever it is that removes the old packages from the repos needs to also delete the image.
4- Chacra/shaman integration? Should the container build show up in shaman/chacra as well? Is there extra work needed to do that?
Thanks! sage
On Mon, 17 Jun 2019, Sebastien Han wrote:
+Dimitri Savineau for ceph-container related questions. I think it'd be nice to have the registry public so that everyone can access and test a build.
Definitely! sage
Thanks! ––––––––– Sébastien Han Principal Software Engineer, Storage Architect
"Always give 100%. Unless you're giving blood."
On Mon, Jun 17, 2019 at 5:34 PM Sage Weil <sweil@redhat.com> wrote:
Hi everyone,
I'd really like to push this through and get something working this week. After poking around it seems clear that there aren't any other registries we should be using for our temp/test builds (unless we just spam dockerhub, but that seems unwise). So, let's just get over ourselves and run our own registry.
1- Where to put it? I'm assuming this should go in the same place that chacra is putting our other temp builds. This is in RDU, right? What machine(s) should we use? If we use the same retention policy as the debs/rpms then this will be an incremental increase in the storage needed.
2- What registry software to use? We don't need any fancy featuers whatsoever--just the ability to push, pull, and delete images. So, whatever is easiest to set up.
3- Jenkins integration. I think we need to have a child job linked to the centos build to do the ceph-container build and then push. Similarly, whatever it is that removes the old packages from the repos needs to also delete the image.
4- Chacra/shaman integration? Should the container build show up in shaman/chacra as well? Is there extra work needed to do that?
Thanks! sage
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
I've been having good luck with experiments of creating the container as a tarball and shipping it around as an asset in parallel to the packages. This might ease the integration with teuthology and other things that consume shaman without having to worry about running a docker registry separately. On Mon, Jun 17, 2019, 9:06 AM Sage Weil <sweil@redhat.com> wrote:
On Mon, 17 Jun 2019, Sebastien Han wrote:
+Dimitri Savineau for ceph-container related questions. I think it'd be nice to have the registry public so that everyone can access and test a build.
Definitely!
sage
Thanks! ––––––––– Sébastien Han Principal Software Engineer, Storage Architect
"Always give 100%. Unless you're giving blood."
On Mon, Jun 17, 2019 at 5:34 PM Sage Weil <sweil@redhat.com> wrote:
Hi everyone,
I'd really like to push this through and get something working this
After poking around it seems clear that there aren't any other registries we should be using for our temp/test builds (unless we just spam dockerhub, but that seems unwise). So, let's just get over ourselves and run our own registry.
1- Where to put it? I'm assuming this should go in the same place that chacra is putting our other temp builds. This is in RDU, right? What machine(s) should we use? If we use the same retention policy as the debs/rpms then this will be an incremental increase in the storage needed.
2- What registry software to use? We don't need any fancy featuers whatsoever--just the ability to push, pull, and delete images. So, whatever is easiest to set up.
3- Jenkins integration. I think we need to have a child job linked to
week. the
centos build to do the ceph-container build and then push. Similarly, whatever it is that removes the old packages from the repos needs to also delete the image.
4- Chacra/shaman integration? Should the container build show up in shaman/chacra as well? Is there extra work needed to do that?
Thanks! sage
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io _______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
What's the status of registry.ceph.com? Has it been abandoned? Can we repurpose or kill it? It's currently in OVH's cloud (not Sepia). We can host a container registry (publicly) in the Sepia lab no problem. We could even use the LRC as backing storage. I kind of like the idea of including tarballs with existing chacra repos like what Dan mentions. I suspect that would minimize the amount of hacking needed of other tools. This isn't something we can accomplish this week but we can certainly get a game plan going. I would recommend we start with a nightly build before building every sha1. If we end up going with a simple container registry, teuthology will need some hacking. If we want to use shaman/chacra, the will both need some hacking along with teuthology. On 6/17/19 12:43 PM, Dan Mick wrote:
I've been having good luck with experiments of creating the container as a tarball and shipping it around as an asset in parallel to the packages. This might ease the integration with teuthology and other things that consume shaman without having to worry about running a docker registry separately.
On Mon, Jun 17, 2019, 9:06 AM Sage Weil <sweil@redhat.com <mailto:sweil@redhat.com>> wrote:
On Mon, 17 Jun 2019, Sebastien Han wrote: > +Dimitri Savineau for ceph-container related questions. > I think it'd be nice to have the registry public so that everyone can > access and test a build.
Definitely!
sage
> > Thanks! > ––––––––– > Sébastien Han > Principal Software Engineer, Storage Architect > > "Always give 100%. Unless you're giving blood." > > On Mon, Jun 17, 2019 at 5:34 PM Sage Weil <sweil@redhat.com <mailto:sweil@redhat.com>> wrote: > > > > Hi everyone, > > > > I'd really like to push this through and get something working this week. > > After poking around it seems clear that there aren't any other registries > > we should be using for our temp/test builds (unless we just spam > > dockerhub, but that seems unwise). So, let's just get over ourselves and > > run our own registry. > > > > 1- Where to put it? I'm assuming this should go in the same place that > > chacra is putting our other temp builds. This is in RDU, right? What > > machine(s) should we use? If we use the same retention policy as the > > debs/rpms then this will be an incremental increase in the storage needed. > > > > 2- What registry software to use? We don't need any fancy featuers > > whatsoever--just the ability to push, pull, and delete images. So, > > whatever is easiest to set up. > > > > 3- Jenkins integration. I think we need to have a child job linked to the > > centos build to do the ceph-container build and then push. Similarly, > > whatever it is that removes the old packages from the repos needs to also > > delete the image. > > > > 4- Chacra/shaman integration? Should the container build show up in > > shaman/chacra as well? Is there extra work needed to do that? > > > > Thanks! > > sage > > > _______________________________________________ > Sepia mailing list -- sepia@ceph.io <mailto:sepia@ceph.io> > To unsubscribe send an email to sepia-leave@ceph.io <mailto:sepia-leave@ceph.io> > _______________________________________________ Sepia mailing list -- sepia@ceph.io <mailto:sepia@ceph.io> To unsubscribe send an email to sepia-leave@ceph.io <mailto:sepia-leave@ceph.io>
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
Am 18.06.19 um 16:33 schrieb David Galloway:
What's the status of registry.ceph.com? Has it been abandoned? Can we repurpose or kill it? It's currently in OVH's cloud (not Sepia). We can host a container registry (publicly) in the Sepia lab no problem. We could even use the LRC as backing storage.
I kind of like the idea of including tarballs with existing chacra repos
Are these tarsthen consumable (pullable) like any other container image? Or differently, is Rook able to consume them?
like what Dan mentions. I suspect that would minimize the amount of hacking needed of other tools.
This isn't something we can accomplish this week but we can certainly get a game plan going. I would recommend we start with a nightly build before building every sha1.
If we end up going with a simple container registry, teuthology will need some hacking. If we want to use shaman/chacra, the will both need some hacking along with teuthology.
On 6/17/19 12:43 PM, Dan Mick wrote:
I've been having good luck with experiments of creating the container as a tarball and shipping it around as an asset in parallel to the packages. This might ease the integration with teuthology and other things that consume shaman without having to worry about running a docker registry separately.
On Mon, Jun 17, 2019, 9:06 AM Sage Weil <sweil@redhat.com <mailto:sweil@redhat.com>> wrote:
On Mon, 17 Jun 2019, Sebastien Han wrote: > +Dimitri Savineau for ceph-container related questions. > I think it'd be nice to have the registry public so that everyone can > access and test a build.
Definitely!
sage
> > Thanks! > ––––––––– > Sébastien Han > Principal Software Engineer, Storage Architect > > "Always give 100%. Unless you're giving blood." > > On Mon, Jun 17, 2019 at 5:34 PM Sage Weil <sweil@redhat.com <mailto:sweil@redhat.com>> wrote: > > > > Hi everyone, > > > > I'd really like to push this through and get something working this week. > > After poking around it seems clear that there aren't any other registries > > we should be using for our temp/test builds (unless we just spam > > dockerhub, but that seems unwise). So, let's just get over ourselves and > > run our own registry. > > > > 1- Where to put it? I'm assuming this should go in the same place that > > chacra is putting our other temp builds. This is in RDU, right? What > > machine(s) should we use? If we use the same retention policy as the > > debs/rpms then this will be an incremental increase in the storage needed. > > > > 2- What registry software to use? We don't need any fancy featuers > > whatsoever--just the ability to push, pull, and delete images. So, > > whatever is easiest to set up. > > > > 3- Jenkins integration. I think we need to have a child job linked to the > > centos build to do the ceph-container build and then push. Similarly, > > whatever it is that removes the old packages from the repos needs to also > > delete the image. > > > > 4- Chacra/shaman integration? Should the container build show up in > > shaman/chacra as well? Is there extra work needed to do that? > > > > Thanks! > > sage > > > _______________________________________________ > Sepia mailing list -- sepia@ceph.io <mailto:sepia@ceph.io> > To unsubscribe send an email to sepia-leave@ceph.io <mailto:sepia-leave@ceph.io> > _______________________________________________ Sepia mailing list -- sepia@ceph.io <mailto:sepia@ceph.io> To unsubscribe send an email to sepia-leave@ceph.io <mailto:sepia-leave@ceph.io>
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
-- SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
I kind of like the idea of including tarballs with existing chacra repos
Are these tarsthen consumable (pullable) like any other container image? Or differently, is Rook able to consume them?
I had the same question. Don't we still need a registry of some sort to use them?
This isn't something we can accomplish this week but we can certainly get a game plan going. I would recommend we start with a nightly build before building every sha1.
I like the idea of starting with a nightly build. Will we eventually want the same policy of every push to ceph-ci gets a container built as well?
If we end up going with a simple container registry, teuthology will need some hacking. If we want to use shaman/chacra, the will both need some hacking along with teuthology.
They would both need some work here. Maybe not chacra if we don't try to include containers in the existing dev repos though. - Andrew
On Tue, 18 Jun 2019, Andrew Schoen wrote:
This isn't something we can accomplish this week but we can certainly get a game plan going. I would recommend we start with a nightly build before building every sha1.
I like the idea of starting with a nightly build. Will we eventually want the same policy of every push to ceph-ci gets a container built as well?
I think we already have some ad hoc nightly builds. I suggest jumping straight to ceph-ci builds for every branch--that's the important part.
If we end up going with a simple container registry, teuthology will need some hacking. If we want to use shaman/chacra, the will both need some hacking along with teuthology.
They would both need some work here. Maybe not chacra if we don't try to include containers in the existing dev repos though.
Let's not worry about consuming htese from teuthology yet--that is its own can of worms. s
On 6/18/19 7:37 AM, Sebastian Wagner wrote:
Am 18.06.19 um 16:33 schrieb David Galloway:
What's the status of registry.ceph.com? Has it been abandoned? Can we repurpose or kill it? It's currently in OVH's cloud (not Sepia). We can host a container registry (publicly) in the Sepia lab no problem. We could even use the LRC as backing storage.
I kind of like the idea of including tarballs with existing chacra repos
Are these tarsthen consumable (pullable) like any other container image? Or differently, is Rook able to consume them?
Yes, but they have to be preloaded into Docker's (or podman's or whoever's) storage locally first; then the pull operation notices them locally in the same way it notices that a remote image has been cached. So it's a "transfer the tarball to the remote node, do "docker load" or the equivalent", and then docker pull (or Kubernetes' builtin pull) works as expected (assuming you ask for the right tag, of course). Sage and I talked some more about this, and the distribution to test nodes is definitely a wrinkle that will imply some sort of test framework (not hard for teuthology, but another manual step for things like manual Rook testing). We can continue to look at registries and registrieless while we work out the builds and see which difficulties are most annoying. I'm more worried about redundancy/availability and old-image rolloff with the registry server, but it does come with efficiencies too (like not having to transfer the layers that are already locally cached) and I'm not yet ready to claim that one is best.
like what Dan mentions. I suspect that would minimize the amount of hacking needed of other tools.
This isn't something we can accomplish this week but we can certainly get a game plan going. I would recommend we start with a nightly build before building every sha1.
If we end up going with a simple container registry, teuthology will need some hacking. If we want to use shaman/chacra, the will both need some hacking along with teuthology.
On 6/17/19 12:43 PM, Dan Mick wrote:
I've been having good luck with experiments of creating the container as a tarball and shipping it around as an asset in parallel to the packages. This might ease the integration with teuthology and other things that consume shaman without having to worry about running a docker registry separately.
On Mon, Jun 17, 2019, 9:06 AM Sage Weil <sweil@redhat.com <mailto:sweil@redhat.com>> wrote:
On Mon, 17 Jun 2019, Sebastien Han wrote: > +Dimitri Savineau for ceph-container related questions. > I think it'd be nice to have the registry public so that everyone can > access and test a build.
Definitely!
sage
> > Thanks! > ––––––––– > Sébastien Han > Principal Software Engineer, Storage Architect > > "Always give 100%. Unless you're giving blood." > > On Mon, Jun 17, 2019 at 5:34 PM Sage Weil <sweil@redhat.com <mailto:sweil@redhat.com>> wrote: > > > > Hi everyone, > > > > I'd really like to push this through and get something working this week. > > After poking around it seems clear that there aren't any other registries > > we should be using for our temp/test builds (unless we just spam > > dockerhub, but that seems unwise). So, let's just get over ourselves and > > run our own registry. > > > > 1- Where to put it? I'm assuming this should go in the same place that > > chacra is putting our other temp builds. This is in RDU, right? What > > machine(s) should we use? If we use the same retention policy as the > > debs/rpms then this will be an incremental increase in the storage needed. > > > > 2- What registry software to use? We don't need any fancy featuers > > whatsoever--just the ability to push, pull, and delete images. So, > > whatever is easiest to set up. > > > > 3- Jenkins integration. I think we need to have a child job linked to the > > centos build to do the ceph-container build and then push. Similarly, > > whatever it is that removes the old packages from the repos needs to also > > delete the image. > > > > 4- Chacra/shaman integration? Should the container build show up in > > shaman/chacra as well? Is there extra work needed to do that? > > > > Thanks! > > sage > > > _______________________________________________ > Sepia mailing list -- sepia@ceph.io <mailto:sepia@ceph.io> > To unsubscribe send an email to sepia-leave@ceph.io <mailto:sepia-leave@ceph.io> > _______________________________________________ Sepia mailing list -- sepia@ceph.io <mailto:sepia@ceph.io> To unsubscribe send an email to sepia-leave@ceph.io <mailto:sepia-leave@ceph.io>
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
On Tue, 18 Jun 2019, David Galloway wrote:
What's the status of registry.ceph.com? Has it been abandoned? Can we repurpose or kill it? It's currently in OVH's cloud (not Sepia). We can host a container registry (publicly) in the Sepia lab no problem.
Not sure.. dan, was this something you set up long ago?
We could even use the LRC as backing storage.
I suggest we use whatever backs the current packages (not sure if that's LRC or not).
I kind of like the idea of including tarballs with existing chacra repos like what Dan mentions. I suspect that would minimize the amount of hacking needed of other tools.
...as long as it is also fully consumable in other contexts without any weirdness.
This isn't something we can accomplish this week but we can certainly get a game plan going. I would recommend we start with a nightly build before building every sha1.
If we end up going with a simple container registry, teuthology will need some hacking. If we want to use shaman/chacra, the will both need some hacking along with teuthology.
On 6/17/19 12:43 PM, Dan Mick wrote:
I've been having good luck with experiments of creating the container as a tarball and shipping it around as an asset in parallel to the packages. This might ease the integration with teuthology and other things that consume shaman without having to worry about running a docker registry separately.
On Mon, Jun 17, 2019, 9:06 AM Sage Weil <sweil@redhat.com <mailto:sweil@redhat.com>> wrote:
On Mon, 17 Jun 2019, Sebastien Han wrote: > +Dimitri Savineau for ceph-container related questions. > I think it'd be nice to have the registry public so that everyone can > access and test a build.
Definitely!
sage
> > Thanks! > ––––––––– > Sébastien Han > Principal Software Engineer, Storage Architect > > "Always give 100%. Unless you're giving blood." > > On Mon, Jun 17, 2019 at 5:34 PM Sage Weil <sweil@redhat.com <mailto:sweil@redhat.com>> wrote: > > > > Hi everyone, > > > > I'd really like to push this through and get something working this week. > > After poking around it seems clear that there aren't any other registries > > we should be using for our temp/test builds (unless we just spam > > dockerhub, but that seems unwise). So, let's just get over ourselves and > > run our own registry. > > > > 1- Where to put it? I'm assuming this should go in the same place that > > chacra is putting our other temp builds. This is in RDU, right? What > > machine(s) should we use? If we use the same retention policy as the > > debs/rpms then this will be an incremental increase in the storage needed. > > > > 2- What registry software to use? We don't need any fancy featuers > > whatsoever--just the ability to push, pull, and delete images. So, > > whatever is easiest to set up. > > > > 3- Jenkins integration. I think we need to have a child job linked to the > > centos build to do the ceph-container build and then push. Similarly, > > whatever it is that removes the old packages from the repos needs to also > > delete the image. > > > > 4- Chacra/shaman integration? Should the container build show up in > > shaman/chacra as well? Is there extra work needed to do that? > > > > Thanks! > > sage > > > _______________________________________________ > Sepia mailing list -- sepia@ceph.io <mailto:sepia@ceph.io> > To unsubscribe send an email to sepia-leave@ceph.io <mailto:sepia-leave@ceph.io> > _______________________________________________ Sepia mailing list -- sepia@ceph.io <mailto:sepia@ceph.io> To unsubscribe send an email to sepia-leave@ceph.io <mailto:sepia-leave@ceph.io>
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
On Tue, Jun 18, 2019 at 9:09 AM Sage Weil <sweil@redhat.com> wrote:
On Tue, 18 Jun 2019, David Galloway wrote:
What's the status of registry.ceph.com? Has it been abandoned? Can we repurpose or kill it? It's currently in OVH's cloud (not Sepia). We can host a container registry (publicly) in the Sepia lab no problem.
Not sure.. dan, was this something you set up long ago?
I think I set that up a while back. It's definitely abandoned when I found out the pain this would involve to do things like updates, authentication, and sane garbage collection. In RH we have a few internal OpenShift instances for CI, and they prune all images older than two weeks. The command that the administrators run is: oc adm prune images --keep-tag-revisions=2 --keep-younger-than=720h --registry-url=docker-registry-default.cloud.registry.example.com --confirm They probably run that in cron or something. *However*, please take my advice as someone who has tried to operationalize my own registry in my spare time and failed: quay.io has pretty good garbage collection now. Eg. they prune an old image immediately once you delete or move the repository's tag. Red Hat is planning to use quay.io for more and more things, so it needs to scale. - Ken
On Tue, 18 Jun 2019, Ken Dreyer wrote:
On Tue, Jun 18, 2019 at 9:09 AM Sage Weil <sweil@redhat.com> wrote:
On Tue, 18 Jun 2019, David Galloway wrote:
What's the status of registry.ceph.com? Has it been abandoned? Can we repurpose or kill it? It's currently in OVH's cloud (not Sepia). We can host a container registry (publicly) in the Sepia lab no problem.
Not sure.. dan, was this something you set up long ago?
I think I set that up a while back. It's definitely abandoned when I found out the pain this would involve to do things like updates, authentication, and sane garbage collection.
In RH we have a few internal OpenShift instances for CI, and they prune all images older than two weeks. The command that the administrators run is:
oc adm prune images --keep-tag-revisions=2 --keep-younger-than=720h --registry-url=docker-registry-default.cloud.registry.example.com --confirm
They probably run that in cron or something.
That doesn't look or sounds that bad...
*However*, please take my advice as someone who has tried to operationalize my own registry in my spare time and failed: quay.io has pretty good garbage collection now. Eg. they prune an old image immediately once you delete or move the repository's tag. Red Hat is planning to use quay.io for more and more things, so it needs to scale.
I'm not sure I read this paragraph correctly. Are you saying we should use quay? Or that garbage collection is hard in general? Because I think the single oc adm prune command above is all that we need... we'll be publishing a zillion ci-built containers and trim them after 2 weeks. What's missing? sage
On Wed, Jun 19, 2019 at 12:03 PM Sage Weil <sweil@redhat.com> wrote:
I'm not sure I read this paragraph correctly. Are you saying we should use quay? Or that garbage collection is hard in general? Because I think the single oc adm prune command above is all that we need... we'll be publishing a zillion ci-built containers and trim them after 2 weeks. What's missing?
Yes, if I were doing this over, I would spend the time trying to make it all work with quay.io instead of running our own registry. Running that oc prune command in cron is just the tip of the iceberg trying to operationalize this. For example with OpenShift 3.6, there was a bug with the way OpenShift (or docker-distribution, or whatever) writes incoming files to NFS, so "docker push" just fails. There is no workaround, and the only fix is to upgrade to newer versions of OpenShift 3. We're already struggling to keep our Jenkins masters updated for security patches, and a public OpenShift/Kubernetes infrastructure will compound the problem. - Ken
On 6/19/19 3:49 PM, Ken Dreyer wrote:
On Wed, Jun 19, 2019 at 12:03 PM Sage Weil <sweil@redhat.com> wrote:
I'm not sure I read this paragraph correctly. Are you saying we should use quay? Or that garbage collection is hard in general? Because I think the single oc adm prune command above is all that we need... we'll be publishing a zillion ci-built containers and trim them after 2 weeks. What's missing?
Yes, if I were doing this over, I would spend the time trying to make it all work with quay.io instead of running our own registry.
Running that oc prune command in cron is just the tip of the iceberg trying to operationalize this. For example with OpenShift 3.6, there was a bug with the way OpenShift (or docker-distribution, or whatever) writes incoming files to NFS, so "docker push" just fails. There is no workaround, and the only fix is to upgrade to newer versions of OpenShift 3.
We're already struggling to keep our Jenkins masters updated for security patches, and a public OpenShift/Kubernetes infrastructure will compound the problem.
I'm all for *not* running yet-another-service. But are quay.io going to be okay with us creating containers for every sha1? Christina and I did a rough estimate and it'd be about 500GB worth of containers at one time. Should we ask quay ahead of time if this is okay? I'd rather get a definitive answer ahead of time instead of set everything up, have them get mad, and then we end up needing to run our own registry anyway.
On Wed, Jun 19, 2019 at 2:15 PM David Galloway <dgallowa@redhat.com> wrote:
On 6/19/19 3:49 PM, Ken Dreyer wrote:
On Wed, Jun 19, 2019 at 12:03 PM Sage Weil <sweil@redhat.com> wrote:
I'm not sure I read this paragraph correctly. Are you saying we should use quay? Or that garbage collection is hard in general? Because I think the single oc adm prune command above is all that we need... we'll be publishing a zillion ci-built containers and trim them after 2 weeks. What's missing?
Yes, if I were doing this over, I would spend the time trying to make it all work with quay.io instead of running our own registry.
Running that oc prune command in cron is just the tip of the iceberg trying to operationalize this. For example with OpenShift 3.6, there was a bug with the way OpenShift (or docker-distribution, or whatever) writes incoming files to NFS, so "docker push" just fails. There is no workaround, and the only fix is to upgrade to newer versions of OpenShift 3.
We're already struggling to keep our Jenkins masters updated for security patches, and a public OpenShift/Kubernetes infrastructure will compound the problem.
I'm all for *not* running yet-another-service. But are quay.io going to be okay with us creating containers for every sha1? Christina and I did a rough estimate and it'd be about 500GB worth of containers at one time.
Should we ask quay ahead of time if this is okay? I'd rather get a definitive answer ahead of time instead of set everything up, have them get mad, and then we end up needing to run our own registry anyway.
From what I heard, they're serving 10TB daily. If quay.io cannot handle this, I wonder how we think we're going to handle it with our current resources.
participants (8)
-
Andrew Schoen
-
Dan Mick
-
David Galloway
-
Ken Dreyer
-
Sage Weil
-
Sage Weil
-
Sebastian Wagner
-
Sebastien Han