Re: Containerized builds
reviving an old thread, but i'm still very interested in this idea. has any other progress been made here? are there simple steps we could take to start experimenting with this? On Thu, Aug 18, 2022 at 6:19 AM Ernesto Puerta <epuertat@redhat.com> wrote:
Hi John,
I'm not very jenkins savvy so I can't speak much to that part. I'll dig into your dockerfile a bit. One thing I note is that there's only a dockerfile for centos (stream). The build infrastructure I'm imagining let's one build ubuntu binaries & packages and/or centos binaries & packages regardless of the "real" OS/distro . What do you think about supporting multiple different dockefiles for each supported distro and selecting between them hypothetically based on a (distro, release, flavor, arch) style tuple like used in the builds?
Yeah, that was my idea too. We currently support 2 different distros in our CI (CentOS and Ubuntu), but we randomly test a PR only on one of them. With containerized (reproducible) builds we could start using a distributed ccache, and with the savings in build times we could afford running the PR pipeline for 2 different distros.
Regarding the Dockerfile, if you check it, there's only 1 line of code that is CentOS-specific (the "dnf install" for the EPEL repo). If we move that to the install-deps.sh, we could have a distro-neutral Dockerfile (the base layer would still have to be set via DISTRO build arg).
Kind Regards, Ernesto
On Fri, Aug 12, 2022 at 5:20 PM John Mulligan <phlogistonjohn@asynchrono.us> wrote:
On Thursday, August 11, 2022 7:17:19 PM EDT John Mulligan wrote:
On Thursday, August 11, 2022 5:34:40 PM EDT Josh Durgin wrote:
I think it's a great idea - it's related to ideas in this thread: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/UD43OL6YBN5A2QHLK
RU QLYQRXMHM5FKJ/
Indeed, I remember participating a little in that thread.
The main idea there is to make it simple to update containers so you
can
run teuthology tests against code you just changed with very little overhead (no need to wait for hours for package + container complete rebuilds).
Great, I've been thinking about this as well - I've been poking at cpatch here-and-there as well and had started working on a python version that I believe will be easier to hack on further. I need to polish it up a
little
and share it.
I spent some time today to make my branch presentable and filed an RFC PR for the python version of cpatch I mentioned above. Posting it here for context: https://github.com/ceph/ceph/pull/47573
Zack Cerza has made a lot of progress on the teuthology side of this - running the tests locally using an existing container image. The 2nd
half,
of making it easy and fast to update a container image, is still TBD.
That's great! I only got Teuthology in Sepia access recently and I would love to try out this version too. Is there a link to a WIP PR or
something
along that line? I'd be interested in trying it out a bit.
In the short term, I'll try to put together a proof-of-concept PR for
some
of the build container ideas I'm thinking of. It seems like there's a
fair
amount of interest.
Thanks!
Josh
On Thu, Aug 11, 2022 at 12:09 PM Tom R <
precision.automobilia@gmail.com>
wrote: > John > > I think your proposal to separate the build process such that the
user
> can > select an os flavor of their liking is a fantastic idea. > > I'm not familiar with the process to assist but would love to follow
if
> this proposal is accepted. > > > > On Thu, Aug 11, 2022, 1:37 PM John Mulligan > <phlogistonjohn@asynchrono.us> > > wrote: >> On the user's list one thread about the packages took a turn into >> discussing >> building in containers [1]. This is a topic that I have had some
idle
>> conversations about with Adam King and so I figured that I would
raise
>> this to >> a wider audience. >> >> My thought is to use container images specifically for building
ceph -
>> and not >> just it's container images. The builds may continue to produce >> packages, >> but a >> container would be used as an abstraction between the actual OS and
the
>> build >> process of (do_cmake.sh, etc.). >> >> Builder images would be be available for use both by the build
system
>> (jenkins >> builders) as well as individual users. One advantage of this is >> that >> the >> user >> can build the packages for distros that don't match the local
distro.
>> I've >> also found it advantageous for my own builds to use the container >> to >> limit >> memory and CPU for the build. >> >> I'm curious if anyone has discussed this before. Does it interest >> anyone? I >> am willing to volunteer some time to help as well. >> >> Thanks for reading! >> >> >> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/
>> VR3ZKP4T2PLZ6BJ23GPZAG3KBV6AI3LA/ >> <
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/VR3ZK
>> P4 >> T2PLZ6BJ23GPZAG3KBV6AI3LA/> >> >> >> _______________________________________________ >> Dev mailing list -- dev@ceph.io >> To unsubscribe send an email to dev-leave@ceph.io > > _______________________________________________ > Dev mailing list -- dev@ceph.io > To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Thanks for reviving this thread Casey! I'd love to pick up these discussions again. I did a fair amount of work earlier this year investigating how we might improve our build system, both in the context of CI builds and also individuals working locally. My work was focused on improving reliability and performance, and involved building Ceph in a container - but not yet on building a container _of_ Ceph. Much of the performance work consisted of experimenting with sccache[0], a ccache-like project created by Mozilla. I was interested in that tool because it can use shared distributed storage for cached artifacts - meaning individuals can benefit from the cache created by CI builds. Ceph main has initial CMake support[1][2] for this now; we'll want to backport that if we end up liking sccache. I created a Containerfile[3] and image[4] that work with podman and likely also docker. I set up a proof-of-concept build job[5] on BuildKite using a cluster I built in the Sepia lab; it builds on pushes and PRs in ~23min currently. I've invited folks in this thread to the ceph org on BuildKite so they can poke around a bit. Happy to invite more. To access the sccache cluster used by the BuildKite job, VPN access is required. For individuals to write to it we'd want a way to manage credentials. It's possible to use it as a read-only cache anonymously though[6]. I'm nearly finished with an Ansible Galaxy role that can deploy new sccache clusters, but that's not published yet. Some unresolved issues: * ceph-build.git support is missing * During package builds, we use two single-threaded tools that limit potential improvements: bzip2, and dwz * I haven't attempted to build a Ceph container from the built artifacts [0] https://github.com/mozilla/sccache/ [1] https://github.com/ceph/ceph/pull/56734 [2] https://github.com/ceph/ceph/pull/56865 [3] https://github.com/zmc/ceph-sccache/blob/main/Containerfile.builder [4] https://quay.io/repository/ceph-infra/ceph-sccache [5] https://buildkite.com/ceph/build-in-container [6] https://github.com/zmc/ceph-sccache/blob/main/sccache_anon_s3.conf On Thu, Jul 18, 2024 at 1:01 PM Casey Bodley <cbodley@redhat.com> wrote:
reviving an old thread, but i'm still very interested in this idea. has any other progress been made here? are there simple steps we could take to start experimenting with this?
On Thu, Aug 18, 2022 at 6:19 AM Ernesto Puerta <epuertat@redhat.com> wrote:
Hi John,
I'm not very jenkins savvy so I can't speak much to that part. I'll dig into your dockerfile a bit. One thing I note is that there's only a dockerfile for centos (stream). The build infrastructure I'm imagining let's one build ubuntu binaries & packages and/or centos binaries & packages regardless of the "real" OS/distro . What do you think about supporting multiple different dockefiles for each supported distro and selecting between them hypothetically based on a (distro, release, flavor, arch) style tuple like used in the builds?
Yeah, that was my idea too. We currently support 2 different distros in our CI (CentOS and Ubuntu), but we randomly test a PR only on one of them. With containerized (reproducible) builds we could start using a distributed ccache, and with the savings in build times we could afford running the PR pipeline for 2 different distros.
Regarding the Dockerfile, if you check it, there's only 1 line of code that is CentOS-specific (the "dnf install" for the EPEL repo). If we move that to the install-deps.sh, we could have a distro-neutral Dockerfile (the base layer would still have to be set via DISTRO build arg).
Kind Regards, Ernesto
On Fri, Aug 12, 2022 at 5:20 PM John Mulligan <phlogistonjohn@asynchrono.us> wrote:
On Thursday, August 11, 2022 7:17:19 PM EDT John Mulligan wrote:
On Thursday, August 11, 2022 5:34:40 PM EDT Josh Durgin wrote: > I think it's a great idea - it's related to ideas in this thread: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/UD43OL6YBN5A2QHLK
> RU QLYQRXMHM5FKJ/
Indeed, I remember participating a little in that thread.
> The main idea there is to make it simple to update containers so you
can
> run teuthology tests against code you just changed with very little > overhead (no need to wait for hours for package + container complete > rebuilds).
Great, I've been thinking about this as well - I've been poking at cpatch here-and-there as well and had started working on a python version that I believe will be easier to hack on further. I need to polish it up a
little
and share it.
I spent some time today to make my branch presentable and filed an RFC PR for the python version of cpatch I mentioned above. Posting it here for context: https://github.com/ceph/ceph/pull/47573
> Zack Cerza has made a lot of progress on the teuthology side of this - > running the tests locally using an existing container image. The 2nd
half,
> of making it easy and fast to update a container image, is still TBD.
That's great! I only got Teuthology in Sepia access recently and I would love to try out this version too. Is there a link to a WIP PR or
something
along that line? I'd be interested in trying it out a bit.
In the short term, I'll try to put together a proof-of-concept PR for
some
of the build container ideas I'm thinking of. It seems like there's a
fair
amount of interest.
Thanks!
> Josh > > On Thu, Aug 11, 2022 at 12:09 PM Tom R <
precision.automobilia@gmail.com>
> wrote: > > John > > > > I think your proposal to separate the build process such that the
user
> > can > > select an os flavor of their liking is a fantastic idea. > > > > I'm not familiar with the process to assist but would love to follow
if
> > this proposal is accepted. > > > > > > > > On Thu, Aug 11, 2022, 1:37 PM John Mulligan > > <phlogistonjohn@asynchrono.us> > > > > wrote: > >> On the user's list one thread about the packages took a turn into > >> discussing > >> building in containers [1]. This is a topic that I have had some
idle
> >> conversations about with Adam King and so I figured that I would
raise
> >> this to > >> a wider audience. > >> > >> My thought is to use container images specifically for building
ceph -
> >> and not > >> just it's container images. The builds may continue to produce > >> packages, > >> but a > >> container would be used as an abstraction between the actual OS and
the
> >> build > >> process of (do_cmake.sh, etc.). > >> > >> Builder images would be be available for use both by the build
system
> >> (jenkins > >> builders) as well as individual users. One advantage of this is > >> that > >> the > >> user > >> can build the packages for distros that don't match the local
distro.
> >> I've > >> also found it advantageous for my own builds to use the container > >> to > >> limit > >> memory and CPU for the build. > >> > >> I'm curious if anyone has discussed this before. Does it interest > >> anyone? I > >> am willing to volunteer some time to help as well. > >> > >> Thanks for reading! > >> > >> > >> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/
> >> VR3ZKP4T2PLZ6BJ23GPZAG3KBV6AI3LA/ > >> <
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/VR3ZK
> >> P4 > >> T2PLZ6BJ23GPZAG3KBV6AI3LA/> > >> > >> > >> _______________________________________________ > >> Dev mailing list -- dev@ceph.io > >> To unsubscribe send an email to dev-leave@ceph.io > > > > _______________________________________________ > > Dev mailing list -- dev@ceph.io > > To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Thursday, July 18, 2024 2:53:37 PM EDT Casey Bodley wrote:
reviving an old thread, but i'm still very interested in this idea. has any other progress been made here? are there simple steps we could take to start experimenting with this?
Hi there, Everything I have with regards to this is available to look at via https://github.com/ceph/ceph/pull/56711 I periodically rebase it and occasionally use it to build ceph. However, lately I have been too busy working on the smb support to spend much time looking at this. Happy to continue discussing this effort and sharing my experiences and thoughts but I won't have a lot of time to spend on it directly for a while - probably a month or so.
On Thu, Aug 18, 2022 at 6:19 AM Ernesto Puerta <epuertat@redhat.com> wrote:
Hi John,
I'm not very jenkins savvy so I can't speak much to that part. I'll dig into
your dockerfile a bit. One thing I note is that there's only a
dockerfile for centos (stream). The build infrastructure I'm imagining let's one build ubuntu binaries & packages and/or centos binaries & packages regardless of the "real" OS/distro . What do you think about supporting multiple different dockefiles for each supported distro and selecting between them hypothetically based on a (distro, release, flavor, arch) style tuple like used in the builds?>
Yeah, that was my idea too. We currently support 2 different distros in our CI (CentOS and Ubuntu), but we randomly test a PR only on one of them. With containerized (reproducible) builds we could start using a distributed ccache, and with the savings in build times we could afford running the PR pipeline for 2 different distros.
Regarding the Dockerfile, if you check it, there's only 1 line of code that is CentOS-specific (the "dnf install" for the EPEL repo). If we move that to the install-deps.sh, we could have a distro-neutral Dockerfile (the base layer would still have to be set via DISTRO build arg).
Kind Regards, Ernesto
On Fri, Aug 12, 2022 at 5:20 PM John Mulligan <phlogistonjohn@asynchrono.us>
wrote:
On Thursday, August 11, 2022 7:17:19 PM EDT John Mulligan wrote:
On Thursday, August 11, 2022 5:34:40 PM EDT Josh Durgin wrote:
> I think it's a great idea - it's related to ideas in this > thread:
https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/UD43OL6YBN5A 2QHLK
> RU QLYQRXMHM5FKJ/
Indeed, I remember participating a little in that thread.
> The main idea there is to make it simple to update containers so > you
can
> run teuthology tests against code you just changed with very > little > overhead (no need to wait for hours for package + container > complete > rebuilds).
Great, I've been thinking about this as well - I've been poking at cpatch here-and-there as well and had started working on a python version that I believe will be easier to hack on further. I need to polish it up a
little
and share it.
I spent some time today to make my branch presentable and filed an RFC PR for the python version of cpatch I mentioned above. Posting it here for context: https://github.com/ceph/ceph/pull/47573
> Zack Cerza has made a lot of progress on the teuthology side of > this - > running the tests locally using an existing container image. The > 2nd
half,
> of making it easy and fast to update a container image, is still > TBD.
That's great! I only got Teuthology in Sepia access recently and I would love to try out this version too. Is there a link to a WIP PR or
something
along that line? I'd be interested in trying it out a bit.
In the short term, I'll try to put together a proof-of-concept PR for
some
of the build container ideas I'm thinking of. It seems like there's a
fair
amount of interest.
Thanks!
> Josh > > > > On Thu, Aug 11, 2022 at 12:09 PM Tom R <
precision.automobilia@gmail.com>
> wrote: > > > John > > > > > > > > I think your proposal to separate the build process such that > > the
user
> > can > > select an os flavor of their liking is a fantastic idea. > > > > > > > > I'm not familiar with the process to assist but would love to > > follow
if
> > this proposal is accepted. > > > > > > > > > > > > On Thu, Aug 11, 2022, 1:37 PM John Mulligan > > <phlogistonjohn@asynchrono.us> > > > > > > > > wrote: > > > >> On the user's list one thread about the packages took a turn > >> into > >> discussing > >> building in containers [1]. This is a topic that I have had > >> some
idle
> >> conversations about with Adam King and so I figured that I > >> would
raise
> >> this to > >> a wider audience. > >> > >> > >> > >> My thought is to use container images specifically for > >> building
ceph -
> >> and not > >> just it's container images. The builds may continue to > >> produce > >> packages, > >> but a > >> container would be used as an abstraction between the actual > >> OS and
the
> >> build > >> process of (do_cmake.sh, etc.). > >> > >> > >> > >> Builder images would be be available for use both by the > >> build
system
> >> (jenkins > >> builders) as well as individual users. One advantage of this > >> is > >> that > >> the > >> user > >> can build the packages for distros that don't match the local
distro.
> >> I've > >> also found it advantageous for my own builds to use the > >> container > >> to > >> limit > >> memory and CPU for the build. > >> > >> > >> > >> I'm curious if anyone has discussed this before. Does it > >> interest > >> anyone? I > >> am willing to volunteer some time to help as well. > >> > >> > >> > >> Thanks for reading! > >> > >> > >> > >> > >> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/
> >> VR3ZKP4T2PLZ6BJ23GPZAG3KBV6AI3LA/ > >> <
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/VR3Z K
> >> P4 > >> T2PLZ6BJ23GPZAG3KBV6AI3LA/> > >> > >> > >> > >> > >> _______________________________________________ > >> Dev mailing list -- dev@ceph.io > >> To unsubscribe send an email to dev-leave@ceph.io > > > > > > > > _______________________________________________ > > Dev mailing list -- dev@ceph.io > > To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (3)
-
Casey Bodley
-
John Mulligan
-
Zack Cerza