Re: [sepia] Testing mgr/rook orchestrator PRs
Am 30.04.19 um 15:34 schrieb Alfredo Deza:
On Tue, Apr 30, 2019 at 9:27 AM Sebastian Wagner <sebastian.wagner@suse.com> wrote:
Hi Alfredo,
Am 30.04.19 um 15:00 schrieb Alfredo Deza:
On Tue, Apr 30, 2019 at 8:52 AM Sebastian Wagner <sebastian.wagner@suse.com> wrote:
All,
I've been working on exercising some Rook orchestrator commands in an automated fashion (like deploying Ceph services). The concept itself works pretty well and now I'd like to integrate this into Sepia.
A part of this endeavor was to set up an empty Kubernetes cluster using local VMs and Terraform. As Sepia already runs a k8s cluster, it might make sense to just use this existing cluster, instead of creating a new cluster for every test run. One downside of re-using existing clusters is: Only one Test run can access a given cluster at a time and thus eliminating some possible parallelism.
There is another bummer: As far as I know, we're not building Ceph container images for Ceph PRs and https://hub.docker.com/r/ceph/ceph only contains stable Nautilus images. Testing Ceph images automatically after they're released to the public isn't going to fly.
Are there any plans to build Ceph container images in Shaman or from within Jenkins Jobs?
This has been discussed in the past , but it is a tremendous effort which has many moving pieces.
Indeed, it is.
On the other hand, if we want to make container first-class citizens, is not building them really a viable option?
I agree with you here, we should be building containers, regardless of how many repositories we produce a day
One of them is where to store the container images - I don't think it is OK to push to hub.docker.com since we build about 400 repositories per day.
Actually this would be a perfect use case for a private registry.
I agree again here. Would love to see if it there was a community-based effort for a registry so we could push images. As it stands right now, our very small team can't possibly take on running/maintaining another service, much less provide for the tremendous amount of infrastructure needed.
Out of my head, I could think of two alternatives that don't require any new services: 1. Maybe Docker hub could build a nightly image from latest master https://shaman.ceph.com/api/repos/ceph/master/latest/ using a static Dockerfile using a setup described at https://docs.docker.com/docker-hub/builds/ This wouldn't give us tests for PRs, though. 2. Or alternatively, Jenkins could build images locally without pushing them anywhere. (I'm not a big fan of this, as it would require a temporary private container registry while executing the test.)
Or asked in a different way: Are there any automatically build Octopus container images?
There isn't anything for any release at the moment.
Thanks for the clarification.
Best, Sebastia
-- SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
On Tue, 30 Apr 2019, Sebastian Wagner wrote:
Am 30.04.19 um 15:34 schrieb Alfredo Deza:
On Tue, Apr 30, 2019 at 9:27 AM Sebastian Wagner <sebastian.wagner@suse.com> wrote:
Hi Alfredo,
Am 30.04.19 um 15:00 schrieb Alfredo Deza:
On Tue, Apr 30, 2019 at 8:52 AM Sebastian Wagner <sebastian.wagner@suse.com> wrote:
All,
I've been working on exercising some Rook orchestrator commands in an automated fashion (like deploying Ceph services). The concept itself works pretty well and now I'd like to integrate this into Sepia.
A part of this endeavor was to set up an empty Kubernetes cluster using local VMs and Terraform. As Sepia already runs a k8s cluster, it might make sense to just use this existing cluster, instead of creating a new cluster for every test run. One downside of re-using existing clusters is: Only one Test run can access a given cluster at a time and thus eliminating some possible parallelism.
There is another bummer: As far as I know, we're not building Ceph container images for Ceph PRs and https://hub.docker.com/r/ceph/ceph only contains stable Nautilus images. Testing Ceph images automatically after they're released to the public isn't going to fly.
Are there any plans to build Ceph container images in Shaman or from within Jenkins Jobs?
This has been discussed in the past , but it is a tremendous effort which has many moving pieces.
Indeed, it is.
On the other hand, if we want to make container first-class citizens, is not building them really a viable option?
I agree with you here, we should be building containers, regardless of how many repositories we produce a day
One of them is where to store the container images - I don't think it is OK to push to hub.docker.com since we build about 400 repositories per day.
Actually this would be a perfect use case for a private registry.
I agree again here. Would love to see if it there was a community-based effort for a registry so we could push images. As it stands right now, our very small team can't possibly take on running/maintaining another service, much less provide for the tremendous amount of infrastructure needed.
Out of my head, I could think of two alternatives that don't require any new services:
1. Maybe Docker hub could build a nightly image from latest master
https://shaman.ceph.com/api/repos/ceph/master/latest/
using a static Dockerfile using a setup described at
https://docs.docker.com/docker-hub/builds/
This wouldn't give us tests for PRs, though.
2. Or alternatively, Jenkins could build images locally without pushing them anywhere. (I'm not a big fan of this, as it would require a temporary private container registry while executing the test.)
3. Make the normal jenkins jobs build a container and push it to docker hub. Then modify the shaman cleanup task to also remove it. Pushing is easy, but the cleanup is slightly tricky; see this stackoverflow: https://stackoverflow.com/questions/44209644/how-do-i-delete-a-docker-image-... sage
Or asked in a different way: Are there any automatically build Octopus container images?
There isn't anything for any release at the moment.
Thanks for the clarification.
Best, Sebastia
-- SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) _______________________________________________ Sepia mailing list Sepia@lists.ceph.com http://lists.ceph.com/listinfo.cgi/sepia-ceph.com
Am 30.04.19 um 16:36 schrieb Sage Weil:
On Tue, 30 Apr 2019, Sebastian Wagner wrote:
Am 30.04.19 um 15:34 schrieb Alfredo Deza:
On Tue, Apr 30, 2019 at 9:27 AM Sebastian Wagner <sebastian.wagner@suse.com> wrote:
Hi Alfredo,
Am 30.04.19 um 15:00 schrieb Alfredo Deza:
On Tue, Apr 30, 2019 at 8:52 AM Sebastian Wagner <sebastian.wagner@suse.com> wrote:
All,
I've been working on exercising some Rook orchestrator commands in an automated fashion (like deploying Ceph services). The concept itself works pretty well and now I'd like to integrate this into Sepia.
A part of this endeavor was to set up an empty Kubernetes cluster using local VMs and Terraform. As Sepia already runs a k8s cluster, it might make sense to just use this existing cluster, instead of creating a new cluster for every test run. One downside of re-using existing clusters is: Only one Test run can access a given cluster at a time and thus eliminating some possible parallelism.
There is another bummer: As far as I know, we're not building Ceph container images for Ceph PRs and https://hub.docker.com/r/ceph/ceph only contains stable Nautilus images. Testing Ceph images automatically after they're released to the public isn't going to fly.
Are there any plans to build Ceph container images in Shaman or from within Jenkins Jobs?
This has been discussed in the past , but it is a tremendous effort which has many moving pieces.
Indeed, it is.
On the other hand, if we want to make container first-class citizens, is not building them really a viable option?
I agree with you here, we should be building containers, regardless of how many repositories we produce a day
One of them is where to store the container images - I don't think it is OK to push to hub.docker.com since we build about 400 repositories per day.
Actually this would be a perfect use case for a private registry.
I agree again here. Would love to see if it there was a community-based effort for a registry so we could push images. As it stands right now, our very small team can't possibly take on running/maintaining another service, much less provide for the tremendous amount of infrastructure needed.
Out of my head, I could think of two alternatives that don't require any new services:
1. Maybe Docker hub could build a nightly image from latest master
https://shaman.ceph.com/api/repos/ceph/master/latest/
using a static Dockerfile using a setup described at
https://docs.docker.com/docker-hub/builds/
This wouldn't give us tests for PRs, though.
2. Or alternatively, Jenkins could build images locally without pushing them anywhere. (I'm not a big fan of this, as it would require a temporary private container registry while executing the test.)
3. Make the normal jenkins jobs build a container and push it to docker hub. Then modify the shaman cleanup task to also remove it. Pushing is easy, but the cleanup is slightly tricky; see this stackoverflow:
https://stackoverflow.com/questions/44209644/how-do-i-delete-a-docker-image-...
For a nightly master build, we don't need to clean up anything, as we can just replace the existing tag from yesterday (e.g. `latest`). Docker Hub will eventually remove old referenced images.
sage
Or asked in a different way: Are there any automatically build Octopus container images?
There isn't anything for any release at the moment.
Thanks for the clarification.
Best, Sebastia
-- SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
On Tue, 30 Apr 2019, Sebastian Wagner wrote:
Out of my head, I could think of two alternatives that don't require any new services:
1. Maybe Docker hub could build a nightly image from latest master
https://shaman.ceph.com/api/repos/ceph/master/latest/
using a static Dockerfile using a setup described at
https://docs.docker.com/docker-hub/builds/
This wouldn't give us tests for PRs, though.
2. Or alternatively, Jenkins could build images locally without pushing them anywhere. (I'm not a big fan of this, as it would require a temporary private container registry while executing the test.)
3. Make the normal jenkins jobs build a container and push it to docker hub. Then modify the shaman cleanup task to also remove it. Pushing is easy, but the cleanup is slightly tricky; see this stackoverflow:
https://stackoverflow.com/questions/44209644/how-do-i-delete-a-docker-image-...
For a nightly master build, we don't need to clean up anything, as we can just replace the existing tag from yesterday (e.g. `latest`). Docker Hub will eventually remove old referenced images.
I'm suggesting a build for every ceph-ci.git branch, like we do for packages. In that case it's not a 'latest' at all, right? And we want to clean them up after ~2 weeks, just like we do with the rpm/dev repos. sage
participants (2)
-
Sage Weil
-
Sebastian Wagner