Alfredo Deza <adeza@redhat.com> writes:
I would like to raise once again the discussion of improving our release process so that we can minimize these things happening at the very last minute.
This isn't the first time we've hit issues that had to be reverted (luckily packages didn't get to download.ceph.com), and since we insist on owning the package and repository creation, there is no way we can retire a "bad" release that is available publicly.
In this case, there wasn't a build kicked off yet, so there shouldn't be packages out yet, I agree improvements are needed. Maybe waiting for a day after Sage's approval allows for these things to be caught like in this release
A few of the items that I believe could improve are:
* Start having stricter development and freeze periods * Add a quiet period (or waiting time) of a week for a release to be public * Determine what the SHA to release is, so that testing can be done against that SHA
So far QE is freezed on a SHA, and now repos are also locked down, but finding bugs outside of QE suite is usually one of the triggers that causes a QE validated test also going back for build again. As we improve our coverage hopefully this should die down (very optimistic forward looking statements)
* Avoid depending on back and forth emails asking leads if they are OK with a release or if they need another PR to be merged - let's define criteria to meet this.
I agree this increases the lead time, but here an EAFP approach often leads to this critical fix was needed sort of a thing. Finalizing at the CLT might be easier given the final results aren't out on a thursday.
On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha@redhat.com> wrote:
https://github.com/ceph/ceph/pull/32018 has merged, we should be ready to build 14.2.5 now.
On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@redhat.com> wrote:
David Zafman has discovered a buggy patch in nautilus, which we want to revert before releasing 14.2.5. More details in https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. We'll need to rebuild 14.2.5 once the revert merges.
Sorry about the inconvenience.
Neha
On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@redhat.com> wrote:
On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote:
Yuri Weinstein <yweinste@redhat.com> writes:
David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing.
Sage, is this ready to build and start publishing packages for?
Yeah, I think we're good to go!
Thanks everyone- sage
Abhishek, Nathan FYI
On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote: > > > > On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote: >> >> Outstanding need approval: >> >> ceph-deploy - Sage >> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > > approved > >> >> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > > approved, failure tracked in https://tracker.ceph.com/issues/43048 > >> >> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote: >> > >> > (This is an early update, some tests are still running, as we are >> > trying to release this point next week before the US holidays, and >> > have more time to review results) >> > >> > Details of this release summarized here: >> > https://tracker.ceph.com/issues/42839#note-3 >> > >> > rados - approved by Neha >> > rgw - approved by Casey >> > rbd - need approval Jason >> > krbd - need approval Jason, Ilya >> > fs - need approval Patrick, Ramana >> > kcephfs - need approval Patrick, Ramana >> > multimds - need approval Patrick, Ramana >> > ceph-deploy - FAILED Sage, Alfredo ? >> > ceph-disk - N/A >> > upgrade/client-upgrade-hammer (nautilus) - N/A >> > upgrade/client-upgrade-jewel (nautilus) - PASSED >> > upgrade/client-upgrade-mimic (nautilus) - FAILED >> > upgrade/luminous-p2p - in progress >> > powercycle - in progress >> > ceph-ansible - Brad is finxing >> > upgrade/luminous-x (nautilus) - in progress >> > upgrade/mimic-x (nautilus) - in progress >> > ceph-volume - Jan fixing >> > (please speak up if something is missing) >> > >> > Thx >> > YuriW >> _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)