Proposal: Limiting builds for branches pushed to ceph-ci
There are occasions when I would like a branch pushed to ceph-ci to only trigger one build, say centos9+x86_64. I created a PR for that specific case[1], but I’m wondering how feasible it would be to create a more generalized solution. It feels wasteful to launch, say, 7 builds when you’re only interested in 1, and it may slow down other developers. My thoughts aren’t fully concrete yet, in part I don’t the details of the code that interprets those .yml files, but here’s a sketch… 1. Those .yml files look for a tag in the branch name along the lines of “build-file” using the existing regex mechanism already leveraged in ceph-dev-new-trigger.yml. 2. When that tag is found in the branch, use wget to pull a specific file from the top-level of that branch. Say the file is called “ceph-build.config”, I believe a command along the lines of the following would pull that one file without cloning the whole repository: wget https://raw.githubusercontent.com/ceph/ceph-ci/${GIT_BRANCH}/ceph-build.config 3. That file would then define the values of DISTROS, ARCHS, FLAVOR, and maybe also FORCE. BRANCH would be defined by the existing variable ${GIT_BRANCH} variable. I don’t know which format for that file would be easiest — .yml, key/value pairs, etc. 4. The build process would then continue as it does now. Perhaps there are hurdles that I’m not seeing. But I thought it’d be worth putting the idea out there. Thanks for considering, Eric [1] https://github.com/ceph/ceph-build/pull/2104
This seems like a great idea. ...I don't actually know what bit of code handles these builds, though. I thought it was shaman, but looking at the github repo I'm not sure? -Greg On Thu, Feb 2, 2023 at 9:34 AM J. Eric Ivancich <ivancich@redhat.com> wrote:
There are occasions when I would like a branch pushed to ceph-ci to only trigger one build, say centos9+x86_64. I created a PR for that specific case[1], but I’m wondering how feasible it would be to create a more generalized solution. It feels wasteful to launch, say, 7 builds when you’re only interested in 1, and it may slow down other developers.
My thoughts aren’t fully concrete yet, in part I don’t the details of the code that interprets those .yml files, but here’s a sketch…
1. Those .yml files look for a tag in the branch name along the lines of “build-file” using the existing regex mechanism already leveraged in ceph-dev-new-trigger.yml.
2. When that tag is found in the branch, use wget to pull a specific file from the top-level of that branch. Say the file is called “ceph-build.config”, I believe a command along the lines of the following would pull that one file without cloning the whole repository:
wget https://raw.githubusercontent.com/ceph/ceph-ci/${GIT_BRANCH}/ceph-build.config
3. That file would then define the values of DISTROS, ARCHS, FLAVOR, and maybe also FORCE. BRANCH would be defined by the existing variable ${GIT_BRANCH} variable. I don’t know which format for that file would be easiest — .yml, key/value pairs, etc.
4. The build process would then continue as it does now.
Perhaps there are hurdles that I’m not seeing. But I thought it’d be worth putting the idea out there. Thanks for considering,
Eric
[1] https://github.com/ceph/ceph-build/pull/2104 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Sounds like a great idea Eric! Greg, Eric's already made a change in the ceph-build PR linked, that's where this logic occurs. A yaml file in the branch seems like a reasonable way to go. On Thu, Feb 2, 2023 at 9:40 AM Gregory Farnum <gfarnum@redhat.com> wrote:
This seems like a great idea.
...I don't actually know what bit of code handles these builds, though. I thought it was shaman, but looking at the github repo I'm not sure? -Greg
On Thu, Feb 2, 2023 at 9:34 AM J. Eric Ivancich <ivancich@redhat.com> wrote:
There are occasions when I would like a branch pushed to ceph-ci to only
trigger one build, say centos9+x86_64. I created a PR for that specific case[1], but I’m wondering how feasible it would be to create a more generalized solution. It feels wasteful to launch, say, 7 builds when you’re only interested in 1, and it may slow down other developers.
My thoughts aren’t fully concrete yet, in part I don’t the details of
the code that interprets those .yml files, but here’s a sketch…
1. Those .yml files look for a tag in the branch name along the lines of
“build-file” using the existing regex mechanism already leveraged in ceph-dev-new-trigger.yml.
2. When that tag is found in the branch, use wget to pull a specific
file from the top-level of that branch. Say the file is called “ceph-build.config”, I believe a command along the lines of the following would pull that one file without cloning the whole repository:
wget
https://raw.githubusercontent.com/ceph/ceph-ci/${GIT_BRANCH}/ceph-build.config
3. That file would then define the values of DISTROS, ARCHS, FLAVOR, and
maybe also FORCE. BRANCH would be defined by the existing variable ${GIT_BRANCH} variable. I don’t know which format for that file would be easiest — .yml, key/value pairs, etc.
4. The build process would then continue as it does now.
Perhaps there are hurdles that I’m not seeing. But I thought it’d be
worth putting the idea out there. Thanks for considering,
Eric
[1] https://github.com/ceph/ceph-build/pull/2104 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Thu, Feb 2, 2023 at 9:50 AM Josh Durgin <jdurgin@redhat.com> wrote:
Greg, Eric's already made a change in the ceph-build PR linked, that's where this logic occurs.
Durr, I skipped over the footnote and ceph-build is where I went first but its short description is "Helper scripts for building the official Ceph packages" so I thought I was in the wrong place. :D
It seems there must be some other repo involved with the processing scripts that use these .yml files. For example, when I look in ceph-build for a non-yml file that has some of the keys in the .yml files (e.g., "condition-kind"), I can’t find any. So that would seem to suggest that there’s some other repo containing scripts that I haven’t found yet. Eric (he/him)
On Feb 2, 2023, at 1:01 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
On Thu, Feb 2, 2023 at 9:50 AM Josh Durgin <jdurgin@redhat.com> wrote:
Greg, Eric's already made a change in the ceph-build PR linked, that's where this logic occurs.
Durr, I skipped over the footnote and ceph-build is where I went first but its short description is "Helper scripts for building the official Ceph packages" so I thought I was in the wrong place. :D
Great idea, Eric! Definitely that'd certainly save a lot of resources. A small suggestion if I may: based upon your use-case, this will be mostly used to pick 1 or max 2 target releases, right? So rather than relying on a new file inside some repo, what about defining this into the branch name? It feels like a bit hacky at first, but we already recommend a conventional branch name for ceph-ci (mostly for informational purposes), so we could simply extend that convention. The current syntax is: wip-<username>-testing-<date>-<branch> (e.g.: wip-spongebob-testing-2023-01-30-1500-quincy) The proposed syntax would be (we have upto 250 chars to play with branch names): wip-<username>-testing-<date>-<branch>*-build_<distro>* (e.g.: wip-spongebob -testing-2023-01-30-1500-quincy-*build_centos9stream,debian12*) The main benefits of this are: - Can be easily added to the "build-integration-branch" script (which basically generates a branch name by appending stuff), - Jenkins allows to trigger jobs based on branch name patterns, - In Shaman/Jenkins/Pulpito/etc., it would also be immediate to understand what releases a job/branch is building (no need to check out the branch code). - Could be easily implemented in teuthology-suite to detect the available Shaman packages and run only those. Kind Regards, Ernesto On Thu, Feb 2, 2023 at 7:19 PM J. Eric Ivancich <ivancich@redhat.com> wrote:
It seems there must be some other repo involved with the processing scripts that use these .yml files. For example, when I look in ceph-build for a non-yml file that has some of the keys in the .yml files (e.g., "condition-kind"), I can’t find any. So that would seem to suggest that there’s some other repo containing scripts that I haven’t found yet.
Eric (he/him)
On Feb 2, 2023, at 1:01 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
On Thu, Feb 2, 2023 at 9:50 AM Josh Durgin <jdurgin@redhat.com> wrote:
Greg, Eric's already made a change in the ceph-build PR linked, that's
where this logic occurs.
Durr, I skipped over the footnote and ceph-build is where I went first but its short description is "Helper scripts for building the official Ceph packages" so I thought I was in the wrong place. :D
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (4)
-
Ernesto Puerta
-
Gregory Farnum
-
J. Eric Ivancich
-
Josh Durgin