nautilus, octopus and pacific branches restored
Hi everyone, You may have noticed some unusual activity in the backport PRs in the past week, namely force pushes to the base branch and temporary changes of the same, resulting in humongous changesets/diffstats being shown. This was done to work around some deficiencies in the release process, apologies for the inconvenience. Now that 14.2.20, 15.2.11 and 16.2.1 are out the door, everything has been restored. Jenkins is currently backed up processing "make check" and "ceph API tests" jobs, but it should clear up by tomorrow. Please retrigger with "jenkins test make check", "jenkins test api", etc if needed. Some of you have clicked the unhelpful "Update branch" button, which generated an unneeded merge commit. Please get rid of it, either by rebasing or simply rolling back to the parent. I went through the PRs and commented on those that need action, but please double check that there are no merge commits or unrelated changes in the "Commits" list before merging. Thanks, Ilya
Hi Ilya, This process seems to have led to mistakes and wasted cycles by some (many?) contributors, such as me, and some of which you make reference to. For example, after the initial force-push on a base branch I unknowingly did a rebase on a backport branch, due to an automated email, only to have to undo the rebase once the base branch was restored. Some of my backports generated lots of extraneous emails, now list many additional contributors, along with some additional invites for reviewers. First, did I miss an announcement ahead of time that this would take place and how I should handle the noise? And will this process be used for future releases? I don’t know what the deficiencies of our release process are and why this work-around was the preferred option, but I’m left imagining that there must be a better way that’s far less noisy and disruptive. Thanks for considering, Eric
On Apr 20, 2021, at 3:42 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
Hi everyone,
You may have noticed some unusual activity in the backport PRs in the past week, namely force pushes to the base branch and temporary changes of the same, resulting in humongous changesets/diffstats being shown. This was done to work around some deficiencies in the release process, apologies for the inconvenience.
Now that 14.2.20, 15.2.11 and 16.2.1 are out the door, everything has been restored. Jenkins is currently backed up processing "make check" and "ceph API tests" jobs, but it should clear up by tomorrow. Please retrigger with "jenkins test make check", "jenkins test api", etc if needed.
Some of you have clicked the unhelpful "Update branch" button, which generated an unneeded merge commit. Please get rid of it, either by rebasing or simply rolling back to the parent. I went through the PRs and commented on those that need action, but please double check that there are no merge commits or unrelated changes in the "Commits" list before merging.
Thanks,
Ilya
On Fri, May 14, 2021 at 11:48 AM J. Eric Ivancich <ivancich@redhat.com> wrote:
Hi Ilya,
This process seems to have led to mistakes and wasted cycles by some (many?) contributors, such as me, and some of which you make reference to. For example, after the initial force-push on a base branch I unknowingly did a rebase on a backport branch, due to an automated email, only to have to undo the rebase once the base branch was restored. Some of my backports generated lots of extraneous emails, now list many additional contributors, along with some additional invites for reviewers.
Yes, it's a real mess and everyone hates this process. The point of the process is to avoid having to rewrite (rebase) any commits. The -saved branch is used for the current tip of whatever release branch we're hotfixing. The release branch is then hard reset to the last minor release, hotfix patches applied, and then a new release is generated. That last part relies on the branch name as part of the release and is the cause of all these gymnastics. Once the new release is done, the -saved branch merges the new minor release (one new merge commit) and then the release branch hard resets to the -saved branch. This process causes github to freak out and add a number of commits to all of the backport PRs. The "fix" for that is to change the base branch for every backport PR to -saved and back which results in the spam you see. We don't know a better way around that, sorry. Ultimately, the release scripts need to be fixed to no longer require these gymnastics. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Fri, May 14, 2021 at 8:47 PM J. Eric Ivancich <ivancich@redhat.com> wrote:
Hi Ilya,
This process seems to have led to mistakes and wasted cycles by some (many?) contributors, such as me, and some of which you make reference to. For example, after the initial force-push on a base branch I unknowingly did a rebase on a backport branch, due to an automated email, only to have to undo the rebase once the base branch was restored. Some of my backports generated lots of extraneous emails, now list many additional contributors, along with some additional invites for reviewers.
Right, see Patrick's reply for why this happens.
First, did I miss an announcement ahead of time that this would take place and how I should handle the noise?
There was no separate announcement but you may have seen hotfix release emails going by. master and release branches are never rebased or force pushed to otherwise, so if you see it happen it is because a hotfix release is being built and someone from the release team is handling it. Just as a confirmation, another indication is that the base branch would be "locked" by requiring six reviews. The best course of action is to ignore it and let the release team sort out the affected PRs. If you are in hurry, changing the base branch from e.g. octopus to octopus-saved and then back to octopus for an octopus PR will get rid of extraneous commits and contributors (but not labels or review requests, unfortunately).
And will this process be used for future releases? I don’t know what the deficiencies of our release process are and why this work-around was the preferred option, but I’m left imagining that there must be a better way that’s far less noisy and disruptive.
Yes, this process will be used for hotfix releases until the release tooling is able to build, sign and get the release out the door from an arbitrarily named branch. This came up at the last CLT meeting [1] and a bunch of times before that, hopefully it will get fixed soon! [1] https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/AQDOCJGUVXCN7Y5OBNO... Thanks, Ilya
Thanks for considering,
Eric
On Apr 20, 2021, at 3:42 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
Hi everyone,
You may have noticed some unusual activity in the backport PRs in the past week, namely force pushes to the base branch and temporary changes of the same, resulting in humongous changesets/diffstats being shown. This was done to work around some deficiencies in the release process, apologies for the inconvenience.
Now that 14.2.20, 15.2.11 and 16.2.1 are out the door, everything has been restored. Jenkins is currently backed up processing "make check" and "ceph API tests" jobs, but it should clear up by tomorrow. Please retrigger with "jenkins test make check", "jenkins test api", etc if needed.
Some of you have clicked the unhelpful "Update branch" button, which generated an unneeded merge commit. Please get rid of it, either by rebasing or simply rolling back to the parent. I went through the PRs and commented on those that need action, but please double check that there are no merge commits or unrelated changes in the "Commits" list before merging.
Thanks,
Ilya
participants (3)
-
Ilya Dryomov
-
J. Eric Ivancich
-
Patrick Donnelly