On Fri, Dec 6, 2019 at 9:52 AM Sage Weil <sweil@redhat.com> wrote:
>
> On Fri, 6 Dec 2019, Alfredo Deza wrote:
> > I would like to raise once again the discussion of improving our
> > release process so that we can minimize these things happening at the
> > very last minute.
> >
> > This isn't the first time we've hit issues that had to be reverted
> > (luckily packages didn't get to download.ceph.com), and since we
> > insist on owning the package and repository creation, there
> > is no way we can retire a "bad" release that is available publicly.
>
> Doesn't the fact that we own the distribution mean that we *can* yank a
> bad release?  If we were relying on downstreams to do this for us we'd
> have no control at all.  Currently the only reason we can't is because
> we're chafing against the tools.
>
> > A few of the items that I believe could improve are:
> >
> > * Start having stricter development and freeze periods
> > * Add a quiet period (or waiting time) of a week for a release to be public
> > * Determine what the SHA to release is, so that testing can be done
> > against that SHA
> > * Avoid depending on back and forth emails asking leads if they are OK
> > with a release or if they need another PR to be merged - let's define
> > criteria to meet this.
>
> I think the main missing piece here is deploying the release candidate
> into production before actually releasing.  Most of these last minute
> issues were caught when we deployed on the lab cluster and my home
> cluster.   I think this is similar to your 'quiet' period: once qa passes,
> we should deploy internally and make sure there aren't things the qa suite
> missed.

Yes exactly, a release candidate would be great.

>
> The other problem was discovered in parallel by David and was totally
> unrelated to the release process/freeze/testing, but was deemed to be a
> release blocker.  I don't think any process can avoid that.

And by no means I want to raise the release issues as something
specific to this last problem, but something
that does need to be addressed.

>
> This release is a bit different, though, because there is a data
> corruption bug we really want to get out.  Perhaps what we should have
> done here is a minimal release that only has that one bug fix so that our
> regression risk is basically 0 and we can expedite.  Instead we grabbed
> everything that had already been tested and merged and went with that.

Yes exactly. We've had similar "merge with what was tested and
release" situations before that have not worked well.

>
> sage
>
>
> >
> >
> > On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha@redhat.com> wrote:
> > >
> > > https://github.com/ceph/ceph/pull/32018 has merged, we should be ready
> > > to build 14.2.5 now.
> > >
> > > On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@redhat.com> wrote:
> > > >
> > > > David Zafman has discovered a buggy patch in nautilus, which we want
> > > > to revert before releasing 14.2.5. More details in
> > > > https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The
> > > > revert PR https://github.com/ceph/ceph/pull/32018 is being tested now.
> > > > We'll need to rebuild 14.2.5 once the revert merges.
> > > >
> > > > Sorry about the inconvenience.
> > > >
> > > > Neha
> > > >
> > > > On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@redhat.com> wrote:
> > > > >
> > > > > On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote:
> > > > > > Yuri Weinstein <yweinste@redhat.com> writes:
> > > > > >
> > > > > > > David, assuming Sage is OK with `ceph-deploy` and overall tests
> > > > > > > results, this is ready for publishing.
> > > > > >
> > > > > > Sage, is this ready to build and start publishing packages for?
> > > > >
> > > > > Yeah, I think we're good to go!
> > > > >
> > > > > Thanks everyone-
> > > > > sage
> > > > >
> > > > > > > Abhishek, Nathan FYI
> > > > > > >
> > > > > > > On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote:
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote:
> > > > > > >>>
> > > > > > >>> Outstanding need approval:
> > > > > > >>>
> > > > > > >>> ceph-deploy - Sage
> > > > > > >>> upgrade/luminous-x (nautilus) - Neha, Josh reviewing
> > > > > > >>
> > > > > > >> approved
> > > > > > >>
> > > > > > >>>
> > > > > > >>> upgrade/mimic-x (nautilus) - Neha, Josh reviewing
> > > > > > >>
> > > > > > >> approved, failure tracked in https://tracker.ceph.com/issues/43048
> > > > > > >>
> > > > > > >>>
> > > > > > >>> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote:
> > > > > > >>> >
> > > > > > >>> > (This is an early update, some tests are still running, as we are
> > > > > > >>> > trying to release this point next week before the US holidays, and
> > > > > > >>> > have more time to review results)
> > > > > > >>> >
> > > > > > >>> > Details of this release summarized here:
> > > > > > >>> > https://tracker.ceph.com/issues/42839#note-3
> > > > > > >>> >
> > > > > > >>> > rados - approved by Neha
> > > > > > >>> > rgw - approved by Casey
> > > > > > >>> > rbd - need approval Jason
> > > > > > >>> > krbd - need approval Jason, Ilya
> > > > > > >>> > fs - need approval Patrick, Ramana
> > > > > > >>> > kcephfs - need approval Patrick, Ramana
> > > > > > >>> > multimds - need approval Patrick, Ramana
> > > > > > >>> > ceph-deploy - FAILED Sage, Alfredo ?
> > > > > > >>> > ceph-disk - N/A
> > > > > > >>> > upgrade/client-upgrade-hammer (nautilus) - N/A
> > > > > > >>> > upgrade/client-upgrade-jewel (nautilus) - PASSED
> > > > > > >>> > upgrade/client-upgrade-mimic (nautilus) - FAILED
> > > > > > >>> > upgrade/luminous-p2p - in progress
> > > > > > >>> > powercycle - in progress
> > > > > > >>> > ceph-ansible - Brad is finxing
> > > > > > >>> > upgrade/luminous-x (nautilus) - in progress
> > > > > > >>> > upgrade/mimic-x (nautilus) - in progress
> > > > > > >>> > ceph-volume - Jan fixing
> > > > > > >>> > (please speak up if something is missing)
> > > > > > >>> >
> > > > > > >>> > Thx
> > > > > > >>> > YuriW
> > > > > > >>>
> > > > > > > _______________________________________________
> > > > > > > Dev mailing list -- dev@ceph.io
> > > > > > > To unsubscribe send an email to dev-leave@ceph.io
> > > > > >
> > > > > > --
> > > > > > Abhishek Lekshmanan
> > > > > > SUSE Software Solutions Germany GmbH
> > > > > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
> > > > > > _______________________________________________
> > > > > > Dev mailing list -- dev@ceph.io
> > > > > > To unsubscribe send an email to dev-leave@ceph.io
> > > > > > _______________________________________________
> > > > > Dev mailing list -- dev@ceph.io
> > > > > To unsubscribe send an email to dev-leave@ceph.io
> > >
> >
> >
_______________________________________________
Dev mailing list -- dev@ceph.io
To unsubscribe send an email to dev-leave@ceph.io