Here we go https://hub.docker.com/r/ceph/ceph/tags?page=1&name=v15.2 Regards, Dimitri On Tue, Mar 24, 2020 at 8:31 AM Dimitri Savineau <dsavinea@redhat.com> wrote:

Hi Sage,

The arm container is already built so we're only waiting for the x86_64 container image. I just triggered the job manually so we should have the v15.2/v15.2.0 tags in the next hour or so.

Regards,

Dimitri

On Mon, Mar 23, 2020 at 9:12 PM Sage Weil <sweil@redhat.com> wrote:

...

Hi Abhi,

15.2.0 is pushed to download.ceph.com. I merged the last 2 doc PRs targetted for octopus and merged that back into master. Your PR targets master, so that'll need a cherry-pick/backport to octopus branch after it merges.

The v15.2[.0] tags aren't up on docker hub yet.. I'm not sure when the nightly cron for that runs.

I think the only other thing left is to push out the email and blogs... want to go ahead and do that in the morning?

Yay! sage

I think we've tried this in the past, and certainly discussed it. The problems were: 1) We really, really, REALLY don't want to have regressions in new releases compared to old ones. Developing against a stable branch and then needing to forward port makes that way more likely. 2) When we make changes that are difficult to port across, having to forward-port is a pain. It's a pain going backwards too, but at least we have a well-defined process for doing that which makes it hard to lose track of regressions and makes the incompatibilities clear. 2a) Repeated merges into a diverged master from a stable branch across difficult changes doesn't necessarily make it easier to track or fix just because we use the merging mechanic — cherry-picks can be fixed as you go, whereas merges need to fix all divergences within the merge commit. 3) Forward-merging breaks down pretty badly once fixes need to be applied across multiple releases. 4) We need to deal with bugs which are fixed incidentally in master but require their own patches in older releases, and we don't necessarily want those fixes showing up in HEAD. In terms of what we do now, we've had a few problems but that's going to happen in any process, and given how much we backport I'd actually argue it's pretty effective. I don't handle a huge number of downstream customer cases personally, but the ones I have dealt with I've had good success with our upstream-first model — if it's urgent, I handle the backport myself and queue up a teuthology run before merge; if it isn't then our friendly backport team and their scripts do good work! -Greg On Tue, Mar 24, 2020 at 6:47 AM Lenz Grimmer <lgrimmer@suse.com> wrote:

Hi Sage, all,

On 2020-03-24 02:11, Sage Weil wrote:

...

15.2.0 is pushed to download.ceph.com. I merged the last 2 doc PRs targetted for octopus and merged that back into master. Your PR targets master, so that'll need a cherry-pick/backport to octopus branch after it merges.

Now that the octopus branch is fully merged into master again: would this be a good time to rethink our approach of applying bug fixes to stable versions?

I'm aware that at this point we won't be able to fix the process for older stable releases like nautilus or mimic, but maybe we could consider introducing a different approach starting with octopus, to slowly establish a new practice?

As a recap: if you currently need to fix a bug in a stable release branch, you first need to create a pull request against the master branch, that fixes the issue. Once the PR has been merged, it will eventually be backported into the desired stable branch via a "backport", using "git cherry-pick -x". This work is usually done by members of the "backporting team" (but not necessarily so).

In my view, this approach has several disadvantages:

* Bugs are usually not fixed in the actual branch/release that a bug has been reported against, but rather on top of the current development branch (master). * It can cause long delays for getting critical bugs fixed. The testing and reviewing in master takes time. Once the master PR has been merged, there's delay before the backport is being tackled, which again takes some time before it's queued up for reviewing/testing. From a downstream perspective, this is a very cumbersome process and causes long turnaround times for delivering fixes to customers for vendors that follow an "upstream first" policy. * Performing backports from the development branch into the stable branch comes with a much higher risk of introducing regressions. If a fix is primarily developed and tested on master, it may still cleanly apply using "git cherry-pick". But the further the code bases diverge, it may be missing some related changes from other PRs that are required to make that change fully functional. This has already happened several times and isn't always captured by our tests right away. * If a backport does not apply cleanly, it usually requires attention from the original developer, to help with resolving the merge conflicts or taking over the backport work. * It generates a new changeset in the git history. Sure, by using "-x" flag, we preserve the reference to the original changeset, but we still apply the same change twice, working against git's built-in mechanisms of keeping track of changes and its ability to merge them automatically. Over time, the stable branch diverges more and more.

How can these challenges be addressed?

Starting with Octopus and going forward, I would like to propose the following approach for submitting and merging bug fixes:

* For bugs that affect the octopus branch, the initial PR that fixes it targets the octopus branch first. The developer creates and tests their fix on a local copy of the octopus branch and first submits a PR against octopus when ready. * Once the fix has been reviewed/tested and has been merged into the octopus branch, the developer performs a local merge of that fix into the master branch (or rather a local working branch of it), using the canonical "git merge" process. If a fix needs further modifications or amendments, they could be applied via follow-up commits on top of that merge changeset. Once the fix has been tested locally, a PR of that merge against the master branch is submitted, going through the usual testing and review process. * In case this particular fix would still have to be applied against older stable branches, the usual backporting process would kick in at this state.

Benefits of this approach:

* First and foremost, a fix is developed and tested on the actual branch that is currently used by the community, by the developer who is most familiar with the component/code base. * Turnaround times for bug fixes on stable releases can be drastically improved, making downstream consumers happy. * Propagating fixes is done using git's native merge methods, avoiding the duplication of changesets that perform the same code changes. * The deviation of the code base is limited to changes on the master branch, there are no additional changesets that are only contained in the stable branch.

I know this is completely against our current process and is a drastic change in how things have been done around here for ages. But I'd like to get your thoughts and concerns on this approach. Do you agree to my reasoning? Where do you see obstacles that make this impractical? Would it help if I'd visualize my proposal?

Looking forward to your feedback,

Lenz

-- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

Hi Nathan, thanks for your feedback and comments! Please see my replies inline. On 2020-03-24 17:04, Nathan Cutler wrote:

Based on my experience with how folks comply with the current backporting rules, I would not trust very many people (you are one of the exceptions!) to adhere to this rule that bugfixes should be merged into master after fixing in octopus.

My expectation is that, once the a bug is fixed in octopus the second part (merging into master) will be forgotten in many cases. After all, as you say: that's the "actual branch that is currently used by the community" and once the bug is fixed there, the immediate thorn-in-the-side is gone!

That's indeed a good point. Both ways depend on manual intervention and tracking, but the current way has a built-in motivation factor (as the fix needs to be applied to the affected branches explicitly). One option of doing this would be to mandate a corresponding merge PR into master to be always submitted along with the bug fix PR into the octopus branch, but then one would have to keep multiple PRs updated in parallel. I agree this will be challenging.

Another thing I have learned is that any rule, in order to be effective, must be policed. With the current horribly-imperfect-but-it's-what-we-have system, when someone opens a PR targeting nautilus and we see that it's "original research" (to borrow a term from Wikipedia), we immediately see that. Policing is relatively easy - we see at first glance that it's not a cherry-pick. Thanks to GitHub, we can easily click on the SHA1 and check if it's present in the master branch.

It sounds like your proposed new system is expecting developers will always remember to forward-port to master *after* their bug has been fixed (or feature implemented). But is that a realistic expectation? Would it be prudent to include some reminder/enforcement mechanism? If so, how would it work?

Well, one way of doing this is to have a job that compares the octopus branch with the master branch and reports all currently unmerged changesets and their authors. Actually, once a PR has been merged into octopus, we could set up a trigger job that automatically prepares a merge PR of that changeset into the master branch and opens it for review/approval, assigned to the original author. This way, they don't have to perform the work locally themselves. If the PR is in a mergeable state, propagating it into master is straightforward. If more changes/modifications are necessary, a developer could create a local clone of that branch and perform any required fixes/updates. Would that be feasible? Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)