On Mon, 17 Jun 2019, Sebastien Han wrote: Definitely! sage

What's the status of registry.ceph.com? Has it been abandoned? Can we repurpose or kill it? It's currently in OVH's cloud (not Sepia). We can host a container registry (publicly) in the Sepia lab no problem. We could even use the LRC as backing storage. I kind of like the idea of including tarballs with existing chacra repos like what Dan mentions. I suspect that would minimize the amount of hacking needed of other tools. This isn't something we can accomplish this week but we can certainly get a game plan going. I would recommend we start with a nightly build before building every sha1. If we end up going with a simple container registry, teuthology will need some hacking. If we want to use shaman/chacra, the will both need some hacking along with teuthology. On 6/17/19 12:43 PM, Dan Mick wrote: > I've been having good luck with experiments of creating the container as > a tarball and shipping it around as an asset in parallel to the > packages. This might ease the integration with teuthology and other > things that consume shaman without having to worry about running a > docker registry separately. > > On Mon, Jun 17, 2019, 9:06 AM Sage Weil &lt;sweil(a)redhat.com > <mailto:sweil@redhat.com>> wrote: > > On Mon, 17 Jun 2019, Sebastien Han wrote: > > +Dimitri Savineau  for ceph-container related questions. > > I think it'd be nice to have the registry public so that everyone can > > access and test a build. > > Definitely! > > sage > > > > > Thanks! > > ––––––––– > > Sébastien Han > > Principal Software Engineer, Storage Architect > > > > "Always give 100%. Unless you're giving blood." > > > > On Mon, Jun 17, 2019 at 5:34 PM Sage Weil &lt;sweil(a)redhat.com > <mailto:sweil@redhat.com>> wrote: > > > > > > Hi everyone, > > > > > > I'd really like to push this through and get something working > this week. > > > After poking around it seems clear that there aren't any other > registries > > > we should be using for our temp/test builds (unless we just spam > > > dockerhub, but that seems unwise).  So, let's just get over > ourselves and > > > run our own registry. > > > > > > 1- Where to put it?  I'm assuming this should go in the same > place that > > > chacra is putting our other temp builds.  This is in RDU, > right?  What > > > machine(s) should we use?  If we use the same retention policy > as the > > > debs/rpms then this will be an incremental increase in the > storage needed. > > > > > > 2- What registry software to use?  We don't need any fancy featuers > > > whatsoever--just the ability to push, pull, and delete images.  So, > > > whatever is easiest to set up. > > > > > > 3- Jenkins integration. I think we need to have a child job > linked to the > > > centos build to do the ceph-container build and then push.  > Similarly, > > > whatever it is that removes the old packages from the repos > needs to also > > > delete the image. > > > > > > 4- Chacra/shaman integration?  Should the container build show up in > > > shaman/chacra as well?  Is there extra work needed to do that? > > > > > > Thanks! > > > sage > > > > > _______________________________________________ > > Sepia mailing list -- sepia(a)ceph.io <mailto:sepia@ceph.io> > > To unsubscribe send an email to sepia-leave(a)ceph.io > <mailto:sepia-leave@ceph.io> > > _______________________________________________ > Sepia mailing list -- sepia(a)ceph.io <mailto:sepia@ceph.io> > To unsubscribe send an email to sepia-leave(a)ceph.io > <mailto:sepia-leave@ceph.io> > > > _______________________________________________ > Sepia mailing list -- sepia(a)ceph.io > To unsubscribe send an email to sepia-leave(a)ceph.io >

I had the same question. Don't we still need a registry of some sort to use them? I like the idea of starting with a nightly build. Will we eventually want the same policy of every push to ceph-ci gets a container built as well? They would both need some work here. Maybe not chacra if we don't try to include containers in the existing dev repos though. - Andrew

On 6/18/19 7:37 AM, Sebastian Wagner wrote: Yes, but they have to be preloaded into Docker's (or podman's or whoever's) storage locally first; then the pull operation notices them locally in the same way it notices that a remote image has been cached. So it's a "transfer the tarball to the remote node, do "docker load" or the equivalent", and then docker pull (or Kubernetes' builtin pull) works as expected (assuming you ask for the right tag, of course). Sage and I talked some more about this, and the distribution to test nodes is definitely a wrinkle that will imply some sort of test framework (not hard for teuthology, but another manual step for things like manual Rook testing). We can continue to look at registries and registrieless while we work out the builds and see which difficulties are most annoying. I'm more worried about redundancy/availability and old-image rolloff with the registry server, but it does come with efficiencies too (like not having to transfer the layers that are already locally cached) and I'm not yet ready to claim that one is best.

On Tue, Jun 18, 2019 at 9:09 AM Sage Weil &lt;sweil(a)redhat.com&gt; wrote: I think I set that up a while back. It's definitely abandoned when I found out the pain this would involve to do things like updates, authentication, and sane garbage collection. In RH we have a few internal OpenShift instances for CI, and they prune all images older than two weeks. The command that the administrators run is: oc adm prune images --keep-tag-revisions=2 --keep-younger-than=720h --registry-url=docker-registry-default.cloud.registry.example.com --confirm They probably run that in cron or something. *However*, please take my advice as someone who has tried to operationalize my own registry in my spare time and failed: quay.io has pretty good garbage collection now. Eg. they prune an old image immediately once you delete or move the repository's tag. Red Hat is planning to use quay.io for more and more things, so it needs to scale. - Ken

On Wed, Jun 19, 2019 at 12:03 PM Sage Weil &lt;sweil(a)redhat.com&gt; wrote: Yes, if I were doing this over, I would spend the time trying to make it all work with quay.io instead of running our own registry. Running that oc prune command in cron is just the tip of the iceberg trying to operationalize this. For example with OpenShift 3.6, there was a bug with the way OpenShift (or docker-distribution, or whatever) writes incoming files to NFS, so "docker push" just fails. There is no workaround, and the only fix is to upgrade to newer versions of OpenShift 3. We're already struggling to keep our Jenkins masters updated for security patches, and a public OpenShift/Kubernetes infrastructure will compound the problem. - Ken

On Wed, Jun 19, 2019 at 2:15 PM David Galloway &lt;dgallowa(a)redhat.com&gt; wrote: From what I heard, they're serving 10TB daily. If quay.io cannot handle this, I wonder how we think we're going to handle it with our current resources.