8 Feb
2021
8 Feb
'21
8:48 p.m.
Hi Folks,
I am working on GitHub action based automation for running teuthology
on some batch of PRs.
Following step describes this workflow:
Step 1: Once any PR passes necessary checks( make etc) a newly
introduced label `next` can be assigned to PR.
Step 2: The `next` labeled PRs will be batched together and scheduled
for the next teuthology run.
The collected labels from all the batched PRs will be used for
selecting the teuthology suite.
Having said that, there are two options to batch and schedule a tethology run:
1. Batch all patch together
2. Batch patch component wise
Both of the above options have their pros and cons one being the high
number of teuthology jobs per run if batched together and more number
teuthology runs if batched component wise.
Please, let me know which option is good to go ahead with and if there
are ways to improve it.
/sunny
9 Feb
9 Feb
3:46 a.m.
Hey Sunny,
Happy to hear about this effort!
Regarding the label/s, there's already this `needs-qa` label, which is
basically to ask QA folks to run that PR through teuthology. Could we reuse
it?
Additionally, this other 'skip-teuthology' label (we could rename it as
'skip-qa' for consistency) is to make it explicit that a PR doesn't need to
go through it.
On the batching process, my vote is for the component-wise grouping, but I
understand this might mean a significant increase in runs, right? Are you
planning to introduce some 'wait' period to allow more PRs enter the batch?
Kind Regards,
Ernesto
On Mon, Feb 8, 2021 at 12:50 PM Sunny Kumar <sunkumar(a)redhat.com> wrote:
Hi Folks,
I am working on GitHub action based automation for running teuthology
on some batch of PRs.
Following step describes this workflow:
Step 1: Once any PR passes necessary checks( make etc) a newly
introduced label `next` can be assigned to PR.
Step 2: The `next` labeled PRs will be batched together and scheduled
for the next teuthology run.
The collected labels from all the batched PRs will be used for
selecting the teuthology suite.
Having said that, there are two options to batch and schedule a tethology
run:
1. Batch all patch together
2. Batch patch component wise
Both of the above options have their pros and cons one being the high
number of teuthology jobs per run if batched together and more number
teuthology runs if batched component wise.
Please, let me know which option is good to go ahead with and if there
are ways to improve it.
/sunny
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
8:37 p.m.
Hey Ernesto,
Thanks!
On Mon, Feb 8, 2021 at 6:46 PM Ernesto Puerta <epuertat(a)redhat.com> wrote:
Hey Sunny,
Happy to hear about this effort!
Regarding the label/s, there's already this `needs-qa` label, which is basically to
ask QA folks to run that PR through teuthology. Could we reuse it?
Additionally, this other 'skip-teuthology' label (we could rename it as
'skip-qa' for consistency) is to make it explicit that a PR doesn't need to go
through it.
Yes sure we can use any existing labels, I was considering `next` to
imply this will be marked for the next batch.
There is no need for skip-label as this action will be only triggered
for the PRs marked with label `next`.
In addition to `next` there also will be a list of required labels
(like needs-qa) and conditions(all mandatory checks passed) to be met
before getting included in the next batch.
If any PR(s) will not satisfy these criteria a comment with proper
explanation will be made.
On the batching process, my vote is for the
component-wise grouping, but I understand this might mean a significant increase in runs,
right? Are you planning to introduce some 'wait' period to allow more PRs enter
the batch?
Yes! component-wise will increase the number of runs.
I am working to make it more like a schedule say run once or twice a
day. Longer the duration, more PRs can enter the next batch.
Kind Regards,
Ernesto
On Mon, Feb 8, 2021 at 12:50 PM Sunny Kumar <sunkumar(a)redhat.com> wrote:
>
> Hi Folks,
>
> I am working on GitHub action based automation for running teuthology
> on some batch of PRs.
>
> Following step describes this workflow:
>
> Step 1: Once any PR passes necessary checks( make etc) a newly
> introduced label `next` can be assigned to PR.
>
> Step 2: The `next` labeled PRs will be batched together and scheduled
> for the next teuthology run.
>
> The collected labels from all the batched PRs will be used for
> selecting the teuthology suite.
> Having said that, there are two options to batch and schedule a tethology run:
> 1. Batch all patch together
> 2. Batch patch component wise
>
> Both of the above options have their pros and cons one being the high
> number of teuthology jobs per run if batched together and more number
> teuthology runs if batched component wise.
>
> Please, let me know which option is good to go ahead with and if there
> are ways to improve it.
>
> /sunny
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
>
11 Feb
11 Feb
10:15 a.m.
On 2/9/21 3:37 AM, Sunny Kumar wrote:
Hey Ernesto,
Thanks!
On Mon, Feb 8, 2021 at 6:46 PM Ernesto Puerta <epuertat(a)redhat.com> wrote:
This seems like a good place to start, we can always adjust the batching
later if needed.
Josh
Hey Sunny,
Happy to hear about this effort!
Regarding the label/s, there's already this `needs-qa` label, which is basically to
ask QA folks to run that PR through teuthology. Could we reuse it?
Additionally, this other 'skip-teuthology' label (we could rename it as
'skip-qa' for consistency) is to make it explicit that a PR doesn't need to go
through it.
Yes sure we can use any existing labels, I was considering `next` to
imply this will be marked for the next batch.
There is no need for skip-label as this action will be only triggered
for the PRs marked with label `next`.
In addition to `next` there also will be a list of required labels
(like needs-qa) and conditions(all mandatory checks passed) to be met
before getting included in the next batch.
If any PR(s) will not satisfy these criteria a comment with proper
explanation will be made.
On the batching process, my vote is for the
component-wise grouping, but I understand this might mean a significant increase in runs,
right? Are you planning to introduce some 'wait' period to allow more PRs enter
the batch?
Yes! component-wise will increase the number of runs.
I am working to make it more like a schedule say run once or twice a
day. Longer the duration, more PRs can enter the next batch. 
10 Feb
10 Feb
5 a.m.
On Mon, Feb 8, 2021 at 3:50 AM Sunny Kumar <sunkumar(a)redhat.com> wrote:
Hi Folks,
I am working on GitHub action based automation for running teuthology
on some batch of PRs.
Following step describes this workflow:
Step 1: Once any PR passes necessary checks( make etc) a newly
introduced label `next` can be assigned to PR.
Step 2: The `next` labeled PRs will be batched together and scheduled
for the next teuthology run.
The collected labels from all the batched PRs will be used for
selecting the teuthology suite.
Having said that, there are two options to batch and schedule a tethology run:
1. Batch all patch together
2. Batch patch component wise
Both of the above options have their pros and cons one being the high
number of teuthology jobs per run if batched together and more number
teuthology runs if batched component wise.
It's not really the number of scheduled suite runs that matter, but
the total number of jobs. I'm inclined to think runs should be batched
together based on the component/suites they need to run through, since
we try to cross-pollinate the QA to catch obvious errors which cross
component boundaries, and non-obvious ones can be caught in master
integration.
I'm curious about the intention here, though -- do we expect all QA
runs to eventually be scheduled this way? If so, that sounds GREAT,
but for ease of error bisection we probably want to limit the number
of PRs which are batched together, or have some way to differentiate
between low- and high-risk changes so that we can push simpler things
through ASAP without them getting stuck behind more invasive patches
that cause test issues.
-Greg
Please, let me know which option is good to go ahead with and if there
are ways to improve it.
/sunny
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
11 Feb
11 Feb
10:13 a.m.
On 2/9/21 12:00 PM, Gregory Farnum wrote:
On Mon, Feb 8, 2021 at 3:50 AM Sunny Kumar
<sunkumar(a)redhat.com> wrote:
Yes, using this to replace manual scheduling is the idea - that will
save a lot of developer time and lower the probability of accindents
like running the wrong suite(s).
If we can stabilize the suites enough (I hope we can with all the bug
fixing for pacific) we could even use these runs as a gate for merging
PRs.
Josh
Hi Folks,
I am working on GitHub action based automation for running teuthology
on some batch of PRs.
Following step describes this workflow:
Step 1: Once any PR passes necessary checks( make etc) a newly
introduced label `next` can be assigned to PR.
Step 2: The `next` labeled PRs will be batched together and scheduled
for the next teuthology run.
The collected labels from all the batched PRs will be used for
selecting the teuthology suite.
Having said that, there are two options to batch and schedule a tethology run:
1. Batch all patch together
2. Batch patch component wise
Both of the above options have their pros and cons one being the high
number of teuthology jobs per run if batched together and more number
teuthology runs if batched component wise.
It's not really the number of scheduled suite runs that matter, but
the total number of jobs. I'm inclined to think runs should be batched
together based on the component/suites they need to run through, since
we try to cross-pollinate the QA to catch obvious errors which cross
component boundaries, and non-obvious ones can be caught in master
integration.
I'm curious about the intention here, though -- do we expect all QA
runs to eventually be scheduled this way? If so, that sounds GREAT,
but for ease of error bisection we probably want to limit the number
of PRs which are batched together, or have some way to differentiate
between low- and high-risk changes so that we can push simpler things
through ASAP without them getting stuck behind more invasive patches
that cause test issues.
1173
days inactive
1176
days old
5 comments
4 participants
participants (4)
-
Ernesto Puerta -
Gregory Farnum -
Josh Durgin -
Sunny Kumar