Ceph Release Branch Management
So we've started doing hotfix releases (v14.2.6 & v14.2.7), yay! Unfortunately, this process is involving some kind of rebase which is wrecking the release commit history. Exhibit A: https://github.com/ceph/ceph/pull/32602 I'm in the process of fixing my backports but for those looking after I'm done, my backport PR now has 156 commits including merges where only 2 should exist. This must not continue. Asking backporters to rebase every backport PR whenever we do a hotfix is a non-starter, IMHO. There's been discussions about using some various branching strategies to make this work better but that would complicate the existing release process and/or muck up upgrade tests. (In particular, it's been suggested that the release branch _only_ point to a tagged release.) Without entering into a drawn out discussion on how we could fix the release process, I'm going to suggest the following git workflow to avoid rebases within the current limitation of the release branch always pointing to the cutting edge state: (1) Hotfix need identified: save current branch state: git checkout $release git branch $release-save git push upstream $release-save # push branch save to GitHub for disaster recovery (2) In GitHub, restrict branch push rights on $release. (3) Hard reset to last tagged release: git reset --hard vX.Y.Z git push --force upstream HEAD:$release (4) Do hotfix merges. (5) Tag/test release. (6) Merge the saved branch onto the release branch: git merge $release-save git push upstream $release # no --force should be necessary! No history is lost. The commit history reflects the reality of what happened. No backport rebasing. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 2/4/20 3:13 PM, Patrick Donnelly wrote:
So we've started doing hotfix releases (v14.2.6 & v14.2.7), yay! Unfortunately, this process is involving some kind of rebase which is wrecking the release commit history. Exhibit A:
https://github.com/ceph/ceph/pull/32602
I'm in the process of fixing my backports but for those looking after I'm done, my backport PR now has 156 commits including merges where only 2 should exist.
This must not continue. Asking backporters to rebase every backport PR whenever we do a hotfix is a non-starter, IMHO.
There's been discussions about using some various branching strategies to make this work better but that would complicate the existing release process and/or muck up upgrade tests. (In particular, it's been suggested that the release branch _only_ point to a tagged release.)
Without entering into a drawn out discussion on how we could fix the release process, I'm going to suggest the following git workflow to avoid rebases within the current limitation of the release branch always pointing to the cutting edge state:
(1) Hotfix need identified: save current branch state:
git checkout $release git branch $release-save git push upstream $release-save # push branch save to GitHub for disaster recovery
(2) In GitHub, restrict branch push rights on $release.
(3) Hard reset to last tagged release:
git reset --hard vX.Y.Z git push --force upstream HEAD:$release
(4) Do hotfix merges.
(5) Tag/test release.
(6) Merge the saved branch onto the release branch:
git merge $release-save git push upstream $release # no --force should be necessary!
No history is lost. The commit history reflects the reality of what happened. No backport rebasing.
This was exactly the procedure we used - if you check via the cli you'll see all those commits are in fact still in the nautilus branch. There was no rebasing. For example, git log origin/nautilus..FETCH_HEAD after fetching that PR shows only 2 commits. I'm not sure why github did not update the list of commits in PRs after the nautilus branch was restored - it seems like a bug. I'm curious if anyone knows how to make github refresh its view again. Josh
On Tue, Feb 4, 2020 at 4:13 PM Josh Durgin <jdurgin@redhat.com> wrote:
On 2/4/20 3:13 PM, Patrick Donnelly wrote:
So we've started doing hotfix releases (v14.2.6 & v14.2.7), yay! Unfortunately, this process is involving some kind of rebase which is wrecking the release commit history. Exhibit A:
https://github.com/ceph/ceph/pull/32602
I'm in the process of fixing my backports but for those looking after I'm done, my backport PR now has 156 commits including merges where only 2 should exist.
This must not continue. Asking backporters to rebase every backport PR whenever we do a hotfix is a non-starter, IMHO.
There's been discussions about using some various branching strategies to make this work better but that would complicate the existing release process and/or muck up upgrade tests. (In particular, it's been suggested that the release branch _only_ point to a tagged release.)
Without entering into a drawn out discussion on how we could fix the release process, I'm going to suggest the following git workflow to avoid rebases within the current limitation of the release branch always pointing to the cutting edge state:
(1) Hotfix need identified: save current branch state:
git checkout $release git branch $release-save git push upstream $release-save # push branch save to GitHub for disaster recovery
(2) In GitHub, restrict branch push rights on $release.
(3) Hard reset to last tagged release:
git reset --hard vX.Y.Z git push --force upstream HEAD:$release
(4) Do hotfix merges.
(5) Tag/test release.
(6) Merge the saved branch onto the release branch:
git merge $release-save git push upstream $release # no --force should be necessary!
No history is lost. The commit history reflects the reality of what happened. No backport rebasing.
This was exactly the procedure we used - if you check via the cli you'll see all those commits are in fact still in the nautilus branch. There was no rebasing. For example, git log origin/nautilus..FETCH_HEAD after fetching that PR shows only 2 commits.
My apologies Josh. You're quite right. Shame on me for not actually looking at the commit history!
I'm not sure why github did not update the list of commits in PRs after the nautilus branch was restored - it seems like a bug. I'm curious if anyone knows how to make github refresh its view again.
I don't know except to do the rebase :( -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Tue, 4 Feb 2020, Patrick Donnelly wrote:
On Tue, Feb 4, 2020 at 4:13 PM Josh Durgin <jdurgin@redhat.com> wrote:
On 2/4/20 3:13 PM, Patrick Donnelly wrote:
So we've started doing hotfix releases (v14.2.6 & v14.2.7), yay! Unfortunately, this process is involving some kind of rebase which is wrecking the release commit history. Exhibit A:
https://github.com/ceph/ceph/pull/32602
I'm in the process of fixing my backports but for those looking after I'm done, my backport PR now has 156 commits including merges where only 2 should exist.
This must not continue. Asking backporters to rebase every backport PR whenever we do a hotfix is a non-starter, IMHO.
There's been discussions about using some various branching strategies to make this work better but that would complicate the existing release process and/or muck up upgrade tests. (In particular, it's been suggested that the release branch _only_ point to a tagged release.)
Without entering into a drawn out discussion on how we could fix the release process, I'm going to suggest the following git workflow to avoid rebases within the current limitation of the release branch always pointing to the cutting edge state:
(1) Hotfix need identified: save current branch state:
git checkout $release git branch $release-save git push upstream $release-save # push branch save to GitHub for disaster recovery
(2) In GitHub, restrict branch push rights on $release.
(3) Hard reset to last tagged release:
git reset --hard vX.Y.Z git push --force upstream HEAD:$release
(4) Do hotfix merges.
(5) Tag/test release.
(6) Merge the saved branch onto the release branch:
git merge $release-save git push upstream $release # no --force should be necessary!
No history is lost. The commit history reflects the reality of what happened. No backport rebasing.
This was exactly the procedure we used - if you check via the cli you'll see all those commits are in fact still in the nautilus branch. There was no rebasing. For example, git log origin/nautilus..FETCH_HEAD after fetching that PR shows only 2 commits.
My apologies Josh. You're quite right. Shame on me for not actually looking at the commit history!
I'm not sure why github did not update the list of commits in PRs after the nautilus branch was restored - it seems like a bug. I'm curious if anyone knows how to make github refresh its view again.
I don't know except to do the rebase :(
I suspect a commit --amend on the top commit would do the trick? But in any case, I don't think it matters that much what commits github shows (except that it is confusing)--the reality in git is the same, right? sage
On Tue, Feb 4, 2020 at 7:14 PM Sage Weil <sweil@redhat.com> wrote:
On Tue, 4 Feb 2020, Patrick Donnelly wrote:
On Tue, Feb 4, 2020 at 4:13 PM Josh Durgin <jdurgin@redhat.com> wrote:
On 2/4/20 3:13 PM, Patrick Donnelly wrote:
So we've started doing hotfix releases (v14.2.6 & v14.2.7), yay! Unfortunately, this process is involving some kind of rebase which is wrecking the release commit history. Exhibit A:
https://github.com/ceph/ceph/pull/32602
I'm in the process of fixing my backports but for those looking after I'm done, my backport PR now has 156 commits including merges where only 2 should exist.
This must not continue. Asking backporters to rebase every backport PR whenever we do a hotfix is a non-starter, IMHO.
There's been discussions about using some various branching strategies to make this work better but that would complicate the existing release process and/or muck up upgrade tests. (In particular, it's been suggested that the release branch _only_ point to a tagged release.)
Without entering into a drawn out discussion on how we could fix the release process, I'm going to suggest the following git workflow to avoid rebases within the current limitation of the release branch always pointing to the cutting edge state:
(1) Hotfix need identified: save current branch state:
git checkout $release git branch $release-save git push upstream $release-save # push branch save to GitHub for disaster recovery
(2) In GitHub, restrict branch push rights on $release.
(3) Hard reset to last tagged release:
git reset --hard vX.Y.Z git push --force upstream HEAD:$release
(4) Do hotfix merges.
(5) Tag/test release.
(6) Merge the saved branch onto the release branch:
git merge $release-save git push upstream $release # no --force should be necessary!
No history is lost. The commit history reflects the reality of what happened. No backport rebasing.
This was exactly the procedure we used - if you check via the cli you'll see all those commits are in fact still in the nautilus branch. There was no rebasing. For example, git log origin/nautilus..FETCH_HEAD after fetching that PR shows only 2 commits.
My apologies Josh. You're quite right. Shame on me for not actually looking at the commit history!
I'm not sure why github did not update the list of commits in PRs after the nautilus branch was restored - it seems like a bug. I'm curious if anyone knows how to make github refresh its view again.
I don't know except to do the rebase :(
I suspect a commit --amend on the top commit would do the trick?
Yes, but I suppose at that point you might as well rebase anyway.
But in any case, I don't think it matters that much what commits github shows (except that it is confusing)--the reality in git is the same, right?
I don't think it matters but it was confusing to some folks reviewing the backport PRs. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
I experimented with https://github.com/ceph/ceph/pull/32997, changing the base branch to nautilus-saved and then back to nautilus(using the edit button) fixes things. Neha On Tue, Feb 4, 2020 at 7:31 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Tue, Feb 4, 2020 at 7:14 PM Sage Weil <sweil@redhat.com> wrote:
On Tue, 4 Feb 2020, Patrick Donnelly wrote:
On Tue, Feb 4, 2020 at 4:13 PM Josh Durgin <jdurgin@redhat.com> wrote:
On 2/4/20 3:13 PM, Patrick Donnelly wrote:
So we've started doing hotfix releases (v14.2.6 & v14.2.7), yay! Unfortunately, this process is involving some kind of rebase which is wrecking the release commit history. Exhibit A:
https://github.com/ceph/ceph/pull/32602
I'm in the process of fixing my backports but for those looking after I'm done, my backport PR now has 156 commits including merges where only 2 should exist.
This must not continue. Asking backporters to rebase every backport PR whenever we do a hotfix is a non-starter, IMHO.
There's been discussions about using some various branching strategies to make this work better but that would complicate the existing release process and/or muck up upgrade tests. (In particular, it's been suggested that the release branch _only_ point to a tagged release.)
Without entering into a drawn out discussion on how we could fix the release process, I'm going to suggest the following git workflow to avoid rebases within the current limitation of the release branch always pointing to the cutting edge state:
(1) Hotfix need identified: save current branch state:
git checkout $release git branch $release-save git push upstream $release-save # push branch save to GitHub for disaster recovery
(2) In GitHub, restrict branch push rights on $release.
(3) Hard reset to last tagged release:
git reset --hard vX.Y.Z git push --force upstream HEAD:$release
(4) Do hotfix merges.
(5) Tag/test release.
(6) Merge the saved branch onto the release branch:
git merge $release-save git push upstream $release # no --force should be necessary!
No history is lost. The commit history reflects the reality of what happened. No backport rebasing.
This was exactly the procedure we used - if you check via the cli you'll see all those commits are in fact still in the nautilus branch. There was no rebasing. For example, git log origin/nautilus..FETCH_HEAD after fetching that PR shows only 2 commits.
My apologies Josh. You're quite right. Shame on me for not actually looking at the commit history!
I'm not sure why github did not update the list of commits in PRs after the nautilus branch was restored - it seems like a bug. I'm curious if anyone knows how to make github refresh its view again.
I don't know except to do the rebase :(
I suspect a commit --amend on the top commit would do the trick?
Yes, but I suppose at that point you might as well rebase anyway.
But in any case, I don't think it matters that much what commits github shows (except that it is confusing)--the reality in git is the same, right?
I don't think it matters but it was confusing to some folks reviewing the backport PRs.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Neha Ojha <nojha@redhat.com> writes:
I experimented with https://github.com/ceph/ceph/pull/32997, changing the base branch to nautilus-saved and then back to nautilus(using the edit button) fixes things.
I just scripted something to go change every other pr targetting the old sha of nautilus as well, and do the branch change dance. Hopefully all the prs targetting nautilus are now targetting the correct branch. For reference the script is here https://gist.github.com/theanalyst/210f60b621e0fd965f5033b150d736a2 (I just made it target the current sha of nautilus c15b0b4124104a3ffdfda8666cdc419fb592a1f8)
Neha
On Tue, Feb 4, 2020 at 7:31 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Tue, Feb 4, 2020 at 7:14 PM Sage Weil <sweil@redhat.com> wrote:
On Tue, 4 Feb 2020, Patrick Donnelly wrote:
On Tue, Feb 4, 2020 at 4:13 PM Josh Durgin <jdurgin@redhat.com> wrote:
On 2/4/20 3:13 PM, Patrick Donnelly wrote:
So we've started doing hotfix releases (v14.2.6 & v14.2.7), yay! Unfortunately, this process is involving some kind of rebase which is wrecking the release commit history. Exhibit A:
https://github.com/ceph/ceph/pull/32602
I'm in the process of fixing my backports but for those looking after I'm done, my backport PR now has 156 commits including merges where only 2 should exist.
This must not continue. Asking backporters to rebase every backport PR whenever we do a hotfix is a non-starter, IMHO.
There's been discussions about using some various branching strategies to make this work better but that would complicate the existing release process and/or muck up upgrade tests. (In particular, it's been suggested that the release branch _only_ point to a tagged release.)
Without entering into a drawn out discussion on how we could fix the release process, I'm going to suggest the following git workflow to avoid rebases within the current limitation of the release branch always pointing to the cutting edge state:
(1) Hotfix need identified: save current branch state:
git checkout $release git branch $release-save git push upstream $release-save # push branch save to GitHub for disaster recovery
(2) In GitHub, restrict branch push rights on $release.
(3) Hard reset to last tagged release:
git reset --hard vX.Y.Z git push --force upstream HEAD:$release
(4) Do hotfix merges.
(5) Tag/test release.
(6) Merge the saved branch onto the release branch:
git merge $release-save git push upstream $release # no --force should be necessary!
No history is lost. The commit history reflects the reality of what happened. No backport rebasing.
This was exactly the procedure we used - if you check via the cli you'll see all those commits are in fact still in the nautilus branch. There was no rebasing. For example, git log origin/nautilus..FETCH_HEAD after fetching that PR shows only 2 commits.
My apologies Josh. You're quite right. Shame on me for not actually looking at the commit history!
I'm not sure why github did not update the list of commits in PRs after the nautilus branch was restored - it seems like a bug. I'm curious if anyone knows how to make github refresh its view again.
I don't know except to do the rebase :(
I suspect a commit --amend on the top commit would do the trick?
Yes, but I suppose at that point you might as well rebase anyway.
But in any case, I don't think it matters that much what commits github shows (except that it is confusing)--the reality in git is the same, right?
I don't think it matters but it was confusing to some folks reviewing the backport PRs.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
On 2/5/20 9:15 AM, Abhishek Lekshmanan wrote:
Neha Ojha <nojha@redhat.com> writes:
I experimented with https://github.com/ceph/ceph/pull/32997, changing the base branch to nautilus-saved and then back to nautilus(using the edit button) fixes things.
I just scripted something to go change every other pr targetting the old sha of nautilus as well, and do the branch change dance. Hopefully all the prs targetting nautilus are now targetting the correct branch.
For reference the script is here https://gist.github.com/theanalyst/210f60b621e0fd965f5033b150d736a2
(I just made it target the current sha of nautilus c15b0b4124104a3ffdfda8666cdc419fb592a1f8)
Thanks, looks like that did the trick!
participants (5)
-
Abhishek Lekshmanan
-
Josh Durgin
-
Neha Ojha
-
Patrick Donnelly
-
Sage Weil