20 Apr
2021
20 Apr
'21
9:42 p.m.
Hi everyone,
You may have noticed some unusual activity in the backport PRs in the
past week, namely force pushes to the base branch and temporary changes
of the same, resulting in humongous changesets/diffstats being shown.
This was done to work around some deficiencies in the release process,
apologies for the inconvenience.
Now that 14.2.20, 15.2.11 and 16.2.1 are out the door, everything has
been restored. Jenkins is currently backed up processing "make check"
and "ceph API tests" jobs, but it should clear up by tomorrow. Please
retrigger with "jenkins test make check", "jenkins test api", etc if
needed.
Some of you have clicked the unhelpful "Update branch" button, which
generated an unneeded merge commit. Please get rid of it, either by
rebasing or simply rolling back to the parent. I went through the PRs
and commented on those that need action, but please double check that
there are no merge commits or unrelated changes in the "Commits" list
before merging.
Thanks,
Ilya
14 May
14 May
8:47 p.m.
Hi Ilya,
This process seems to have led to mistakes and wasted cycles by some (many?) contributors,
such as me, and some of which you make reference to. For example, after the initial
force-push on a base branch I unknowingly did a rebase on a backport branch, due to an
automated email, only to have to undo the rebase once the base branch was restored. Some
of my backports generated lots of extraneous emails, now list many additional
contributors, along with some additional invites for reviewers.
First, did I miss an announcement ahead of time that this would take place and how I
should handle the noise?
And will this process be used for future releases? I don’t know what the deficiencies of
our release process are and why this work-around was the preferred option, but I’m left
imagining that there must be a better way that’s far less noisy and disruptive.
Thanks for considering,
Eric
On Apr 20, 2021, at 3:42 PM, Ilya Dryomov
<idryomov(a)gmail.com> wrote:
Hi everyone,
You may have noticed some unusual activity in the backport PRs in the
past week, namely force pushes to the base branch and temporary changes
of the same, resulting in humongous changesets/diffstats being shown.
This was done to work around some deficiencies in the release process,
apologies for the inconvenience.
Now that 14.2.20, 15.2.11 and 16.2.1 are out the door, everything has
been restored. Jenkins is currently backed up processing "make check"
and "ceph API tests" jobs, but it should clear up by tomorrow. Please
retrigger with "jenkins test make check", "jenkins test api", etc if
needed.
Some of you have clicked the unhelpful "Update branch" button, which
generated an unneeded merge commit. Please get rid of it, either by
rebasing or simply rolling back to the parent. I went through the PRs
and commented on those that need action, but please double check that
there are no merge commits or unrelated changes in the "Commits" list
before merging.
Thanks,
Ilya
9:55 p.m.
On Fri, May 14, 2021 at 11:48 AM J. Eric Ivancich <ivancich(a)redhat.com> wrote:
Hi Ilya,
This process seems to have led to mistakes and wasted cycles by some (many?)
contributors, such as me, and some of which you make reference to. For example, after the
initial force-push on a base branch I unknowingly did a rebase on a backport branch, due
to an automated email, only to have to undo the rebase once the base branch was restored.
Some of my backports generated lots of extraneous emails, now list many additional
contributors, along with some additional invites for reviewers.
Yes, it's a real mess and everyone hates this process.
The point of the process is to avoid having to rewrite (rebase) any
commits. The -saved branch is used for the current tip of whatever
release branch we're hotfixing. The release branch is then hard reset
to the last minor release, hotfix patches applied, and then a new
release is generated. That last part relies on the branch name as part
of the release and is the cause of all these gymnastics. Once the new
release is done, the -saved branch merges the new minor release (one
new merge commit) and then the release branch hard resets to the
-saved branch.
This process causes github to freak out and add a number of commits to
all of the backport PRs. The "fix" for that is to change the base
branch for every backport PR to -saved and back which results in the
spam you see. We don't know a better way around that, sorry.
Ultimately, the release scripts need to be fixed to no longer require
these gymnastics.
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
16 May
16 May
4:02 p.m.
On Fri, May 14, 2021 at 8:47 PM J. Eric Ivancich <ivancich(a)redhat.com> wrote:
Hi Ilya,
This process seems to have led to mistakes and wasted cycles by some (many?)
contributors, such as me, and some of which you make reference to. For example, after the
initial force-push on a base branch I unknowingly did a rebase on a backport branch, due
to an automated email, only to have to undo the rebase once the base branch was restored.
Some of my backports generated lots of extraneous emails, now list many additional
contributors, along with some additional invites for reviewers.
Right, see Patrick's reply for why this happens.
First, did I miss an announcement ahead of time that this would take place and how I
should handle the noise?
There was no separate announcement but you may have seen hotfix
release emails going by. master and release branches are never
rebased or force pushed to otherwise, so if you see it happen it is
because a hotfix release is being built and someone from the release
team is handling it. Just as a confirmation, another indication is
that the base branch would be "locked" by requiring six reviews.
The best course of action is to ignore it and let the release team
sort out the affected PRs. If you are in hurry, changing the base
branch from e.g. octopus to octopus-saved and then back to octopus
for an octopus PR will get rid of extraneous commits and contributors
(but not labels or review requests, unfortunately).
And will this process be used for future releases? I don’t know what the deficiencies of
our release process are and why this work-around was the preferred option, but I’m left
imagining that there must be a better way that’s far less noisy and disruptive.
Yes, this process will be used for hotfix releases until the release
tooling is able to build, sign and get the release out the door from
an arbitrarily named branch. This came up at the last CLT meeting [1]
and a bunch of times before that, hopefully it will get fixed soon!
[1]
https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/AQDOCJGUVXCN7Y5OBN…
Thanks,
Ilya
>
> Thanks for considering,
>
> Eric
>
> > On Apr 20, 2021, at 3:42 PM, Ilya Dryomov <idryomov(a)gmail.com> wrote:
> >
> > Hi everyone,
> >
> > You may have noticed some unusual activity in the backport PRs in the
> > past week, namely force pushes to the base branch and temporary changes
> > of the same, resulting in humongous changesets/diffstats being shown.
> > This was done to work around some deficiencies in the release process,
> > apologies for the inconvenience.
> >
> > Now that 14.2.20, 15.2.11 and 16.2.1 are out the door, everything has
> > been restored. Jenkins is currently backed up processing "make
check"
> > and "ceph API tests" jobs, but it should clear up by tomorrow.
Please
> > retrigger with "jenkins test make check", "jenkins test
api", etc if
> > needed.
> >
> > Some of you have clicked the unhelpful "Update branch" button, which
> > generated an unneeded merge commit. Please get rid of it, either by
> > rebasing or simply rolling back to the parent. I went through the PRs
> > and commented on those that need action, but please double check that
> > there are no merge commits or unrelated changes in the "Commits" list
> > before merging.
> >
> > Thanks,
> >
> > Ilya
>
1069
days inactive
1095
days old
3 comments
3 participants
participants (3)
-
Ilya Dryomov -
J. Eric Ivancich -
Patrick Donnelly