UPDATE: rados - approved by Neha rgw - approved by Casey rbd - need approval Jason krbd - approved by Ilya fs - approved Patrick kcephfs - approved Patrick multimds - approved Patrick ceph-deploy - FAILED Sage, Alfredo ? ceph-disk - N/A upgrade/client-upgrade-hammer (nautilus) - N/A upgrade/client-upgrade-jewel (nautilus) - PASSED upgrade/client-upgrade-mimic (nautilus) - Jason , https://tracker.ceph.com/issues/41954 do we remove tests ? upgrade/client-upgrade-luminous-nautilus (nautilus) - Jason , https://tracker.ceph.com/issues/41954 do we remove tests ? upgrade/luminous-p2p - PASSED powercycle - PASSED ceph-ansible - Brad is fixing

upgrade/luminous-x (nautilus) - need rerun on fixed https://tracker.ceph.com/issues/43009 upgrade/mimic-x (nautilus) - need rerun on fixed https://tracker.ceph.com/issues/43009 ceph-volume - Jan fixed, PASSED (please speak up if something is missing)

On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: approved approved, failure tracked in https://tracker.ceph.com/issues/43048 > > On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > (This is an early update, some tests are still running, as we are > > trying to release this point next week before the US holidays, and > > have more time to review results) > > > > Details of this release summarized here: > > https://tracker.ceph.com/issues/42839#note-3 > > > > rados - approved by Neha > > rgw - approved by Casey > > rbd - need approval Jason > > krbd - need approval Jason, Ilya > > fs - need approval Patrick, Ramana > > kcephfs - need approval Patrick, Ramana > > multimds - need approval Patrick, Ramana > > ceph-deploy - FAILED Sage, Alfredo ? > > ceph-disk - N/A > > upgrade/client-upgrade-hammer (nautilus) - N/A > > upgrade/client-upgrade-jewel (nautilus) - PASSED > > upgrade/client-upgrade-mimic (nautilus) - FAILED > > upgrade/luminous-p2p - in progress > > powercycle - in progress > > ceph-ansible - Brad is finxing > > upgrade/luminous-x (nautilus) - in progress > > upgrade/mimic-x (nautilus) - in progress > > ceph-volume - Jan fixing > > (please speak up if something is missing) > > > > Thx > > YuriW >

David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing. Abhishek, Nathan FYI On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Outstanding need approval: >> >> ceph-deploy - Sage >> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > > approved > >> >> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > > approved, failure tracked in https://tracker.ceph.com/issues/43048 > >> >> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> > >> > (This is an early update, some tests are still running, as we are >> > trying to release this point next week before the US holidays, and >> > have more time to review results) >> > >> > Details of this release summarized here: >> > https://tracker.ceph.com/issues/42839#note-3 >> > >> > rados - approved by Neha >> > rgw - approved by Casey >> > rbd - need approval Jason >> > krbd - need approval Jason, Ilya >> > fs - need approval Patrick, Ramana >> > kcephfs - need approval Patrick, Ramana >> > multimds - need approval Patrick, Ramana >> > ceph-deploy - FAILED Sage, Alfredo ? >> > ceph-disk - N/A >> > upgrade/client-upgrade-hammer (nautilus) - N/A >> > upgrade/client-upgrade-jewel (nautilus) - PASSED >> > upgrade/client-upgrade-mimic (nautilus) - FAILED >> > upgrade/luminous-p2p - in progress >> > powercycle - in progress >> > ceph-ansible - Brad is finxing >> > upgrade/luminous-x (nautilus) - in progress >> > upgrade/mimic-x (nautilus) - in progress >> > ceph-volume - Jan fixing >> > (please speak up if something is missing) >> > >> > Thx >> > YuriW >>

David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing.

Abhishek, Nathan FYI On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha(a)redhat.com> wrote: _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote: Yeah, I think we're good to go! Thanks everyone- sage Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

https://github.com/ceph/ceph/pull/32018 has merged, we should be ready to build 14.2.5 now. On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha(a)redhat.com> wrote:

On Fri, 6 Dec 2019, Alfredo Deza wrote: Doesn't the fact that we own the distribution mean that we *can* yank a bad release? If we were relying on downstreams to do this for us we'd have no control at all. Currently the only reason we can't is because we're chafing against the tools. I think the main missing piece here is deploying the release candidate into production before actually releasing. Most of these last minute issues were caught when we deployed on the lab cluster and my home cluster. I think this is similar to your 'quiet' period: once qa passes, we should deploy internally and make sure there aren't things the qa suite missed. The other problem was discovered in parallel by David and was totally unrelated to the release process/freeze/testing, but was deemed to be a release blocker. I don't think any process can avoid that. This release is a bit different, though, because there is a data corruption bug we really want to get out. Perhaps what we should have done here is a minimal release that only has that one bug fix so that our regression risk is basically 0 and we can expedite. Instead we grabbed everything that had already been tested and merged and went with that.

sage > > > On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > https://github.com/ceph/ceph/pull/32018 has merged, we should be ready > > to build 14.2.5 now. > > > > On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > > > David Zafman has discovered a buggy patch in nautilus, which we want > > > to revert before releasing 14.2.5. More details in > > > https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The > > > revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. > > > We'll need to rebuild 14.2.5 once the revert merges. > > > > > > Sorry about the inconvenience. > > > > > > Neha > > > > > > On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil(a)redhat.com> wrote: > > > > > > > > On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote: > > > > > Yuri Weinstein <yweinste(a)redhat.com> writes: > > > > > > > > > > > David, assuming Sage is OK with `ceph-deploy` and overall tests > > > > > > results, this is ready for publishing. > > > > > > > > > > Sage, is this ready to build and start publishing packages for? > > > > > > > > Yeah, I think we're good to go! > > > > > > > > Thanks everyone- > > > > sage > > > > > > > > > > Abhishek, Nathan FYI > > > > > > > > > > > > On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > > >>> > > > > > >>> Outstanding need approval: > > > > > >>> > > > > > >>> ceph-deploy - Sage > > > > > >>> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > > > > > >> > > > > > >> approved > > > > > >> > > > > > >>> > > > > > >>> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > > > > > >> > > > > > >> approved, failure tracked in https://tracker.ceph.com/issues/43048 > > > > > >> > > > > > >>> > > > > > >>> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > > >>> > > > > > > >>> > (This is an early update, some tests are still running, as we are > > > > > >>> > trying to release this point next week before the US holidays, and > > > > > >>> > have more time to review results) > > > > > >>> > > > > > > >>> > Details of this release summarized here: > > > > > >>> > https://tracker.ceph.com/issues/42839#note-3 > > > > > >>> > > > > > > >>> > rados - approved by Neha > > > > > >>> > rgw - approved by Casey > > > > > >>> > rbd - need approval Jason > > > > > >>> > krbd - need approval Jason, Ilya > > > > > >>> > fs - need approval Patrick, Ramana > > > > > >>> > kcephfs - need approval Patrick, Ramana > > > > > >>> > multimds - need approval Patrick, Ramana > > > > > >>> > ceph-deploy - FAILED Sage, Alfredo ? > > > > > >>> > ceph-disk - N/A > > > > > >>> > upgrade/client-upgrade-hammer (nautilus) - N/A > > > > > >>> > upgrade/client-upgrade-jewel (nautilus) - PASSED > > > > > >>> > upgrade/client-upgrade-mimic (nautilus) - FAILED > > > > > >>> > upgrade/luminous-p2p - in progress > > > > > >>> > powercycle - in progress > > > > > >>> > ceph-ansible - Brad is finxing > > > > > >>> > upgrade/luminous-x (nautilus) - in progress > > > > > >>> > upgrade/mimic-x (nautilus) - in progress > > > > > >>> > ceph-volume - Jan fixing > > > > > >>> > (please speak up if something is missing) > > > > > >>> > > > > > > >>> > Thx > > > > > >>> > YuriW > > > > > >>> > > > > > > _______________________________________________ > > > > > > Dev mailing list -- dev(a)ceph.io > > > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > > > > -- > > > > > Abhishek Lekshmanan > > > > > SUSE Software Solutions Germany GmbH > > > > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) > > > > > _______________________________________________ > > > > > Dev mailing list -- dev(a)ceph.io > > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > _______________________________________________ > > > > Dev mailing list -- dev(a)ceph.io > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > >

On Fri, 6 Dec 2019, Alfredo Deza wrote: Doesn't the fact that we own the distribution mean that we *can* yank a bad release? If we were relying on downstreams to do this for us we'd have no control at all. Currently the only reason we can't is because we're chafing against the tools. I think the main missing piece here is deploying the release candidate into production before actually releasing. Most of these last minute issues were caught when we deployed on the lab cluster and my home cluster. I think this is similar to your 'quiet' period: once qa passes, we should deploy internally and make sure there aren't things the qa suite missed.

The other problem was discovered in parallel by David and was totally unrelated to the release process/freeze/testing, but was deemed to be a release blocker. I don't think any process can avoid that.

This release is a bit different, though, because there is a data corruption bug we really want to get out. Perhaps what we should have done here is a minimal release that only has that one bug fix so that our regression risk is basically 0 and we can expedite. Instead we grabbed everything that had already been tested and merged and went with that.

sage > > > On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > https://github.com/ceph/ceph/pull/32018 has merged, we should be ready > > to build 14.2.5 now. > > > > On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > > > David Zafman has discovered a buggy patch in nautilus, which we want > > > to revert before releasing 14.2.5. More details in > > > https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The > > > revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. > > > We'll need to rebuild 14.2.5 once the revert merges. > > > > > > Sorry about the inconvenience. > > > > > > Neha > > > > > > On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil(a)redhat.com> wrote: > > > > > > > > On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote: > > > > > Yuri Weinstein <yweinste(a)redhat.com> writes: > > > > > > > > > > > David, assuming Sage is OK with `ceph-deploy` and overall tests > > > > > > results, this is ready for publishing. > > > > > > > > > > Sage, is this ready to build and start publishing packages for? > > > > > > > > Yeah, I think we're good to go! > > > > > > > > Thanks everyone- > > > > sage > > > > > > > > > > Abhishek, Nathan FYI > > > > > > > > > > > > On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > > >>> > > > > > >>> Outstanding need approval: > > > > > >>> > > > > > >>> ceph-deploy - Sage > > > > > >>> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > > > > > >> > > > > > >> approved > > > > > >> > > > > > >>> > > > > > >>> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > > > > > >> > > > > > >> approved, failure tracked in https://tracker.ceph.com/issues/43048 > > > > > >> > > > > > >>> > > > > > >>> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > > >>> > > > > > > >>> > (This is an early update, some tests are still running, as we are > > > > > >>> > trying to release this point next week before the US holidays, and > > > > > >>> > have more time to review results) > > > > > >>> > > > > > > >>> > Details of this release summarized here: > > > > > >>> > https://tracker.ceph.com/issues/42839#note-3 > > > > > >>> > > > > > > >>> > rados - approved by Neha > > > > > >>> > rgw - approved by Casey > > > > > >>> > rbd - need approval Jason > > > > > >>> > krbd - need approval Jason, Ilya > > > > > >>> > fs - need approval Patrick, Ramana > > > > > >>> > kcephfs - need approval Patrick, Ramana > > > > > >>> > multimds - need approval Patrick, Ramana > > > > > >>> > ceph-deploy - FAILED Sage, Alfredo ? > > > > > >>> > ceph-disk - N/A > > > > > >>> > upgrade/client-upgrade-hammer (nautilus) - N/A > > > > > >>> > upgrade/client-upgrade-jewel (nautilus) - PASSED > > > > > >>> > upgrade/client-upgrade-mimic (nautilus) - FAILED > > > > > >>> > upgrade/luminous-p2p - in progress > > > > > >>> > powercycle - in progress > > > > > >>> > ceph-ansible - Brad is finxing > > > > > >>> > upgrade/luminous-x (nautilus) - in progress > > > > > >>> > upgrade/mimic-x (nautilus) - in progress > > > > > >>> > ceph-volume - Jan fixing > > > > > >>> > (please speak up if something is missing) > > > > > >>> > > > > > > >>> > Thx > > > > > >>> > YuriW > > > > > >>> > > > > > > _______________________________________________ > > > > > > Dev mailing list -- dev(a)ceph.io > > > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > > > > -- > > > > > Abhishek Lekshmanan > > > > > SUSE Software Solutions Germany GmbH > > > > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) > > > > > _______________________________________________ > > > > > Dev mailing list -- dev(a)ceph.io > > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > _______________________________________________ > > > > Dev mailing list -- dev(a)ceph.io > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > >

Sorry for nagging, but when will 14.2.5 actually be released? On Fri, Dec 6, 2019, 10:32 AM Alfredo Deza <adeza(a)redhat.com> wrote: > > On Fri, Dec 6, 2019 at 9:52 AM Sage Weil <sweil(a)redhat.com> wrote: > > > > On Fri, 6 Dec 2019, Alfredo Deza wrote: > > > I would like to raise once again the discussion of improving our > > > release process so that we can minimize these things happening at the > > > very last minute. > > > > > > This isn't the first time we've hit issues that had to be reverted > > > (luckily packages didn't get to download.ceph.com), and since we > > > insist on owning the package and repository creation, there > > > is no way we can retire a "bad" release that is available publicly. > > > > Doesn't the fact that we own the distribution mean that we *can* yank a > > bad release? If we were relying on downstreams to do this for us we'd > > have no control at all. Currently the only reason we can't is because > > we're chafing against the tools. > > > > > A few of the items that I believe could improve are: > > > > > > * Start having stricter development and freeze periods > > > * Add a quiet period (or waiting time) of a week for a release to be public > > > * Determine what the SHA to release is, so that testing can be done > > > against that SHA > > > * Avoid depending on back and forth emails asking leads if they are OK > > > with a release or if they need another PR to be merged - let's define > > > criteria to meet this. > > > > I think the main missing piece here is deploying the release candidate > > into production before actually releasing. Most of these last minute > > issues were caught when we deployed on the lab cluster and my home > > cluster. I think this is similar to your 'quiet' period: once qa passes, > > we should deploy internally and make sure there aren't things the qa suite > > missed. > > Yes exactly, a release candidate would be great. > > > > > The other problem was discovered in parallel by David and was totally > > unrelated to the release process/freeze/testing, but was deemed to be a > > release blocker. I don't think any process can avoid that. > > And by no means I want to raise the release issues as something > specific to this last problem, but something > that does need to be addressed. > > > > > This release is a bit different, though, because there is a data > > corruption bug we really want to get out. Perhaps what we should have > > done here is a minimal release that only has that one bug fix so that our > > regression risk is basically 0 and we can expedite. Instead we grabbed > > everything that had already been tested and merged and went with that. > > Yes exactly. We've had similar "merge with what was tested and > release" situations before that have not worked well. > > > > > sage > > > > > > > > > > > > > On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > > > > > https://github.com/ceph/ceph/pull/32018 has merged, we should be ready > > > > to build 14.2.5 now. > > > > > > > > On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > > > > > > > David Zafman has discovered a buggy patch in nautilus, which we want > > > > > to revert before releasing 14.2.5. More details in > > > > > https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The > > > > > revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. > > > > > We'll need to rebuild 14.2.5 once the revert merges. > > > > > > > > > > Sorry about the inconvenience. > > > > > > > > > > Neha > > > > > > > > > > On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil(a)redhat.com> wrote: > > > > > > > > > > > > On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote: > > > > > > > Yuri Weinstein <yweinste(a)redhat.com> writes: > > > > > > > > > > > > > > > David, assuming Sage is OK with `ceph-deploy` and overall tests > > > > > > > > results, this is ready for publishing. > > > > > > > > > > > > > > Sage, is this ready to build and start publishing packages for? > > > > > > > > > > > > Yeah, I think we're good to go! > > > > > > > > > > > > Thanks everyone- > > > > > > sage > > > > > > > > > > > > > > Abhishek, Nathan FYI > > > > > > > > > > > > > > > > On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha(a)redhat.com> wrote: > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > > > > >>> > > > > > > > >>> Outstanding need approval: > > > > > > > >>> > > > > > > > >>> ceph-deploy - Sage > > > > > > > >>> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > > > > > > > >> > > > > > > > >> approved > > > > > > > >> > > > > > > > >>> > > > > > > > >>> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > > > > > > > >> > > > > > > > >> approved, failure tracked in https://tracker.ceph.com/issues/43048 > > > > > > > >> > > > > > > > >>> > > > > > > > >>> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > > > > >>> > > > > > > > > >>> > (This is an early update, some tests are still running, as we are > > > > > > > >>> > trying to release this point next week before the US holidays, and > > > > > > > >>> > have more time to review results) > > > > > > > >>> > > > > > > > > >>> > Details of this release summarized here: > > > > > > > >>> > https://tracker.ceph.com/issues/42839#note-3 > > > > > > > >>> > > > > > > > > >>> > rados - approved by Neha > > > > > > > >>> > rgw - approved by Casey > > > > > > > >>> > rbd - need approval Jason > > > > > > > >>> > krbd - need approval Jason, Ilya > > > > > > > >>> > fs - need approval Patrick, Ramana > > > > > > > >>> > kcephfs - need approval Patrick, Ramana > > > > > > > >>> > multimds - need approval Patrick, Ramana > > > > > > > >>> > ceph-deploy - FAILED Sage, Alfredo ? > > > > > > > >>> > ceph-disk - N/A > > > > > > > >>> > upgrade/client-upgrade-hammer (nautilus) - N/A > > > > > > > >>> > upgrade/client-upgrade-jewel (nautilus) - PASSED > > > > > > > >>> > upgrade/client-upgrade-mimic (nautilus) - FAILED > > > > > > > >>> > upgrade/luminous-p2p - in progress > > > > > > > >>> > powercycle - in progress > > > > > > > >>> > ceph-ansible - Brad is finxing > > > > > > > >>> > upgrade/luminous-x (nautilus) - in progress > > > > > > > >>> > upgrade/mimic-x (nautilus) - in progress > > > > > > > >>> > ceph-volume - Jan fixing > > > > > > > >>> > (please speak up if something is missing) > > > > > > > >>> > > > > > > > > >>> > Thx > > > > > > > >>> > YuriW > > > > > > > >>> > > > > > > > > _______________________________________________ > > > > > > > > Dev mailing list -- dev(a)ceph.io > > > > > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > > > > > > > > -- > > > > > > > Abhishek Lekshmanan > > > > > > > SUSE Software Solutions Germany GmbH > > > > > > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) > > > > > > > _______________________________________________ > > > > > > > Dev mailing list -- dev(a)ceph.io > > > > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > _______________________________________________ > > > > > > Dev mailing list -- dev(a)ceph.io > > > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > > > > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io