UPDATE: rados - approved by Neha rgw - approved by Casey rbd - need approval Jason krbd - approved by Ilya fs - approved Patrick kcephfs - approved Patrick multimds - approved Patrick ceph-deploy - FAILED Sage, Alfredo ? ceph-disk - N/A upgrade/client-upgrade-hammer (nautilus) - N/A upgrade/client-upgrade-jewel (nautilus) - PASSED upgrade/client-upgrade-mimic (nautilus) - Jason , https://tracker.ceph.com/issues/41954 do we remove tests ? upgrade/client-upgrade-luminous-nautilus (nautilus) - Jason , https://tracker.ceph.com/issues/41954 do we remove tests ? upgrade/luminous-p2p - PASSED powercycle - PASSED ceph-ansible - Brad is fixing upgrade/luminous-x (nautilus) - need rerun on fixed https://tracker.ceph.com/issues/43009 upgrade/mimic-x (nautilus) - need rerun on fixed https://tracker.ceph.com/issues/43009 ceph-volume - Jan fixed, PASSED (please speak up if something is missing)

On Mon, Nov 25, 2019 at 11:30 AM Yuri Weinstein <yweinste@redhat.com> wrote:

UPDATE:

rados - approved by Neha rgw - approved by Casey

rbd - need approval Jason

rbd approved. Thx!

krbd - approved by Ilya fs - approved Patrick kcephfs - approved Patrick multimds - approved Patrick

ceph-deploy - FAILED Sage, Alfredo ?

ceph-disk - N/A upgrade/client-upgrade-hammer (nautilus) - N/A upgrade/client-upgrade-jewel (nautilus) - PASSED

upgrade/client-upgrade-mimic (nautilus) - Jason , https://tracker.ceph.com/issues/41954 do we remove tests ? upgrade/client-upgrade-luminous-nautilus (nautilus) - Jason , https://tracker.ceph.com/issues/41954 do we remove tests ?

upgrade/luminous-p2p - PASSED powercycle - PASSED

ceph-ansible - Brad is fixing

upgrade/luminous-x (nautilus) - need rerun on fixed https://tracker.ceph.com/issues/43009 upgrade/mimic-x (nautilus) - need rerun on fixed https://tracker.ceph.com/issues/43009

ceph-volume - Jan fixed, PASSED (please speak up if something is missing)

-- Jason

On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote:

Outstanding need approval:

ceph-deploy - Sage upgrade/luminous-x (nautilus) - Neha, Josh reviewing

approved

upgrade/mimic-x (nautilus) - Neha, Josh reviewing

approved, failure tracked in https://tracker.ceph.com/issues/43048

On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote:

...

(This is an early update, some tests are still running, as we are trying to release this point next week before the US holidays, and have more time to review results)

Details of this release summarized here: https://tracker.ceph.com/issues/42839#note-3

rados - approved by Neha rgw - approved by Casey rbd - need approval Jason krbd - need approval Jason, Ilya fs - need approval Patrick, Ramana kcephfs - need approval Patrick, Ramana multimds - need approval Patrick, Ramana ceph-deploy - FAILED Sage, Alfredo ? ceph-disk - N/A upgrade/client-upgrade-hammer (nautilus) - N/A upgrade/client-upgrade-jewel (nautilus) - PASSED upgrade/client-upgrade-mimic (nautilus) - FAILED upgrade/luminous-p2p - in progress powercycle - in progress ceph-ansible - Brad is finxing upgrade/luminous-x (nautilus) - in progress upgrade/mimic-x (nautilus) - in progress ceph-volume - Jan fixing (please speak up if something is missing)

Thx YuriW

On 11/27/19 9:43 PM, Yuri Weinstein wrote:

David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing.

Thank you very much for getting this one out the door! Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

On Fri, 29 Nov 2019, Yuri Weinstein wrote:

Sage, pls see the latest cleaner nightly `ceph-deploy` run: http://pulpito.ceph.com/teuthology-2019-11-29_05:55:03-ceph-deploy-nautilus-...

The good news is the deployment works fine... clean run on ubuntu and selinux denials on centos.

and added: https://tracker.ceph.com/issues/43064

the selinux denials look like they're related (mostly) to our use of libudev?

https://tracker.ceph.com/issues/43065

mira109 has a bad hdd; moved ot the sepia project and updated the ticket

On Wed, Nov 27, 2019 at 12:43 PM Yuri Weinstein <yweinste@redhat.com> wrote:

...

David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing.

Abhishek, Nathan FYI

On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote:

...

On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote:

...

Outstanding need approval:

ceph-deploy - Sage upgrade/luminous-x (nautilus) - Neha, Josh reviewing

approved

...

upgrade/mimic-x (nautilus) - Neha, Josh reviewing

approved, failure tracked in https://tracker.ceph.com/issues/43048

...

On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote:

...

(This is an early update, some tests are still running, as we are trying to release this point next week before the US holidays, and have more time to review results)

Details of this release summarized here: https://tracker.ceph.com/issues/42839#note-3

rados - approved by Neha rgw - approved by Casey rbd - need approval Jason krbd - need approval Jason, Ilya fs - need approval Patrick, Ramana kcephfs - need approval Patrick, Ramana multimds - need approval Patrick, Ramana ceph-deploy - FAILED Sage, Alfredo ? ceph-disk - N/A upgrade/client-upgrade-hammer (nautilus) - N/A upgrade/client-upgrade-jewel (nautilus) - PASSED upgrade/client-upgrade-mimic (nautilus) - FAILED upgrade/luminous-p2p - in progress powercycle - in progress ceph-ansible - Brad is finxing upgrade/luminous-x (nautilus) - in progress upgrade/mimic-x (nautilus) - in progress ceph-volume - Jan fixing (please speak up if something is missing)

Thx YuriW

On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote:

Yuri Weinstein <yweinste@redhat.com> writes:

...

David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing.

Sage, is this ready to build and start publishing packages for?

Yeah, I think we're good to go! Thanks everyone- sage

...

Abhishek, Nathan FYI

On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote:

...

On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote:

...

Outstanding need approval:

ceph-deploy - Sage upgrade/luminous-x (nautilus) - Neha, Josh reviewing

approved

...

upgrade/mimic-x (nautilus) - Neha, Josh reviewing

approved, failure tracked in https://tracker.ceph.com/issues/43048

...

On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote:

...

(This is an early update, some tests are still running, as we are trying to release this point next week before the US holidays, and have more time to review results)

Details of this release summarized here: https://tracker.ceph.com/issues/42839#note-3

rados - approved by Neha rgw - approved by Casey rbd - need approval Jason krbd - need approval Jason, Ilya fs - need approval Patrick, Ramana kcephfs - need approval Patrick, Ramana multimds - need approval Patrick, Ramana ceph-deploy - FAILED Sage, Alfredo ? ceph-disk - N/A upgrade/client-upgrade-hammer (nautilus) - N/A upgrade/client-upgrade-jewel (nautilus) - PASSED upgrade/client-upgrade-mimic (nautilus) - FAILED upgrade/luminous-p2p - in progress powercycle - in progress ceph-ansible - Brad is finxing upgrade/luminous-x (nautilus) - in progress upgrade/mimic-x (nautilus) - in progress ceph-volume - Jan fixing (please speak up if something is missing)

Thx YuriW

_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

https://github.com/ceph/ceph/pull/32018 has merged, we should be ready to build 14.2.5 now. On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@redhat.com> wrote:

David Zafman has discovered a buggy patch in nautilus, which we want to revert before releasing 14.2.5. More details in https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. We'll need to rebuild 14.2.5 once the revert merges.

Sorry about the inconvenience.

Neha

On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@redhat.com> wrote:

...

On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote:

...

Yuri Weinstein <yweinste@redhat.com> writes:

...

David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing.

Sage, is this ready to build and start publishing packages for?

Yeah, I think we're good to go!

Thanks everyone- sage

...

...

Abhishek, Nathan FYI

On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote:

...

On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote:

...

Outstanding need approval:

ceph-deploy - Sage upgrade/luminous-x (nautilus) - Neha, Josh reviewing

approved

...

upgrade/mimic-x (nautilus) - Neha, Josh reviewing

approved, failure tracked in https://tracker.ceph.com/issues/43048

...

On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote: > > (This is an early update, some tests are still running, as we are > trying to release this point next week before the US holidays, and > have more time to review results) > > Details of this release summarized here: > https://tracker.ceph.com/issues/42839#note-3 > > rados - approved by Neha > rgw - approved by Casey > rbd - need approval Jason > krbd - need approval Jason, Ilya > fs - need approval Patrick, Ramana > kcephfs - need approval Patrick, Ramana > multimds - need approval Patrick, Ramana > ceph-deploy - FAILED Sage, Alfredo ? > ceph-disk - N/A > upgrade/client-upgrade-hammer (nautilus) - N/A > upgrade/client-upgrade-jewel (nautilus) - PASSED > upgrade/client-upgrade-mimic (nautilus) - FAILED > upgrade/luminous-p2p - in progress > powercycle - in progress > ceph-ansible - Brad is finxing > upgrade/luminous-x (nautilus) - in progress > upgrade/mimic-x (nautilus) - in progress > ceph-volume - Jan fixing > (please speak up if something is missing) > > Thx > YuriW

_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

On Fri, 6 Dec 2019, Alfredo Deza wrote:

I would like to raise once again the discussion of improving our release process so that we can minimize these things happening at the very last minute.

This isn't the first time we've hit issues that had to be reverted (luckily packages didn't get to download.ceph.com), and since we insist on owning the package and repository creation, there is no way we can retire a "bad" release that is available publicly.

Doesn't the fact that we own the distribution mean that we *can* yank a bad release? If we were relying on downstreams to do this for us we'd have no control at all. Currently the only reason we can't is because we're chafing against the tools.

A few of the items that I believe could improve are:

* Start having stricter development and freeze periods * Add a quiet period (or waiting time) of a week for a release to be public * Determine what the SHA to release is, so that testing can be done against that SHA * Avoid depending on back and forth emails asking leads if they are OK with a release or if they need another PR to be merged - let's define criteria to meet this.

I think the main missing piece here is deploying the release candidate into production before actually releasing. Most of these last minute issues were caught when we deployed on the lab cluster and my home cluster. I think this is similar to your 'quiet' period: once qa passes, we should deploy internally and make sure there aren't things the qa suite missed. The other problem was discovered in parallel by David and was totally unrelated to the release process/freeze/testing, but was deemed to be a release blocker. I don't think any process can avoid that. This release is a bit different, though, because there is a data corruption bug we really want to get out. Perhaps what we should have done here is a minimal release that only has that one bug fix so that our regression risk is basically 0 and we can expedite. Instead we grabbed everything that had already been tested and merged and went with that. sage

On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha@redhat.com> wrote:

...

https://github.com/ceph/ceph/pull/32018 has merged, we should be ready to build 14.2.5 now.

On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@redhat.com> wrote:

...

David Zafman has discovered a buggy patch in nautilus, which we want to revert before releasing 14.2.5. More details in https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. We'll need to rebuild 14.2.5 once the revert merges.

Sorry about the inconvenience.

Neha

On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@redhat.com> wrote:

...

On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote:

...

Yuri Weinstein <yweinste@redhat.com> writes:

...

David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing.

Sage, is this ready to build and start publishing packages for?

Yeah, I think we're good to go!

Thanks everyone- sage

...

...

Abhishek, Nathan FYI

On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote: > > > > On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote: >> >> Outstanding need approval: >> >> ceph-deploy - Sage >> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > > approved > >> >> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > > approved, failure tracked in https://tracker.ceph.com/issues/43048 > >> >> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote: >> > >> > (This is an early update, some tests are still running, as we are >> > trying to release this point next week before the US holidays, and >> > have more time to review results) >> > >> > Details of this release summarized here: >> > https://tracker.ceph.com/issues/42839#note-3 >> > >> > rados - approved by Neha >> > rgw - approved by Casey >> > rbd - need approval Jason >> > krbd - need approval Jason, Ilya >> > fs - need approval Patrick, Ramana >> > kcephfs - need approval Patrick, Ramana >> > multimds - need approval Patrick, Ramana >> > ceph-deploy - FAILED Sage, Alfredo ? >> > ceph-disk - N/A >> > upgrade/client-upgrade-hammer (nautilus) - N/A >> > upgrade/client-upgrade-jewel (nautilus) - PASSED >> > upgrade/client-upgrade-mimic (nautilus) - FAILED >> > upgrade/luminous-p2p - in progress >> > powercycle - in progress >> > ceph-ansible - Brad is finxing >> > upgrade/luminous-x (nautilus) - in progress >> > upgrade/mimic-x (nautilus) - in progress >> > ceph-volume - Jan fixing >> > (please speak up if something is missing) >> > >> > Thx >> > YuriW >> _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

On Fri, Dec 6, 2019 at 6:58 AM Abhishek Lekshmanan <abhishek@suse.com> wrote:

Sage Weil <sweil@redhat.com> writes:

...

On Fri, 6 Dec 2019, Alfredo Deza wrote:

...

I would like to raise once again the discussion of improving our release process so that we can minimize these things happening at the very last minute.

This isn't the first time we've hit issues that had to be reverted (luckily packages didn't get to download.ceph.com), and since we insist on owning the package and repository creation, there is no way we can retire a "bad" release that is available publicly.

Doesn't the fact that we own the distribution mean that we *can* yank a bad release? If we were relying on downstreams to do this for us we'd have no control at all. Currently the only reason we can't is because we're chafing against the tools.

...

A few of the items that I believe could improve are:

* Start having stricter development and freeze periods * Add a quiet period (or waiting time) of a week for a release to be public * Determine what the SHA to release is, so that testing can be done against that SHA * Avoid depending on back and forth emails asking leads if they are OK with a release or if they need another PR to be merged - let's define criteria to meet this.

I think the main missing piece here is deploying the release candidate into production before actually releasing. Most of these last minute issues were caught when we deployed on the lab cluster and my home cluster. I think this is similar to your 'quiet' period: once qa passes, we should deploy internally and make sure there aren't things the qa suite missed.

The other problem was discovered in parallel by David and was totally unrelated to the release process/freeze/testing, but was deemed to be a release blocker. I don't think any process can avoid that.

This release is a bit different, though, because there is a data corruption bug we really want to get out. Perhaps what we should have done here is a minimal release that only has that one bug fix so that our regression risk is basically 0 and we can expedite. Instead we grabbed everything that had already been tested and merged and went with that.

I agree that this should have been a minimal release with just the bluestore bug fix. That's what we were trying to push for when we wanted to expedite 14.2.5 couple of weeks ago. However, the fact that 14.2.5 already had a bunch of fixes(14.2.4 released in early September) led to more follow-on bug fixes that we wanted to deliver in 14.2.5. One could argue that those "follow-on bug fixes" could land in 14.2.6, if we did a timely 14.2.5 release. However, since this time we were able to catch a few annoying bugs ourselves that would have been reported by users anyway at some point, I think we are better off pushing the release by a few more days to avoid bad impression and having to do an immediate 14.2.6 to deal with those. In no way does the above mean that we should not have stricter deadline for freezes. We've been using email, github and IRC to inform everyone when we caught such blocker blogs. Is there a better medium(or common forum) that we can use to bring everyone on the same page when such things happen? Neha Neha

We currently merge to the release branch regularly after QE validation, we could go to merges to {release}-next branches and only merge to {release} after final QE validation and approval, this makes expedited releases easier since we could merge a different branch containing only fixes and merge that to the release branch, but workflow tools would need adoption

...

sage

...

On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha@redhat.com> wrote:

...

https://github.com/ceph/ceph/pull/32018 has merged, we should be ready to build 14.2.5 now.

On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@redhat.com> wrote:

...

David Zafman has discovered a buggy patch in nautilus, which we want to revert before releasing 14.2.5. More details in https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. We'll need to rebuild 14.2.5 once the revert merges.

Sorry about the inconvenience.

Neha

On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@redhat.com> wrote:

...

On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote: > Yuri Weinstein <yweinste@redhat.com> writes: > > > David, assuming Sage is OK with `ceph-deploy` and overall tests > > results, this is ready for publishing. > > Sage, is this ready to build and start publishing packages for?

Yeah, I think we're good to go!

Thanks everyone- sage

> > Abhishek, Nathan FYI > > > > On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote: > >> > >> > >> > >> On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote: > >>> > >>> Outstanding need approval: > >>> > >>> ceph-deploy - Sage > >>> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > >> > >> approved > >> > >>> > >>> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > >> > >> approved, failure tracked in https://tracker.ceph.com/issues/43048 > >> > >>> > >>> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote: > >>> > > >>> > (This is an early update, some tests are still running, as we are > >>> > trying to release this point next week before the US holidays, and > >>> > have more time to review results) > >>> > > >>> > Details of this release summarized here: > >>> > https://tracker.ceph.com/issues/42839#note-3 > >>> > > >>> > rados - approved by Neha > >>> > rgw - approved by Casey > >>> > rbd - need approval Jason > >>> > krbd - need approval Jason, Ilya > >>> > fs - need approval Patrick, Ramana > >>> > kcephfs - need approval Patrick, Ramana > >>> > multimds - need approval Patrick, Ramana > >>> > ceph-deploy - FAILED Sage, Alfredo ? > >>> > ceph-disk - N/A > >>> > upgrade/client-upgrade-hammer (nautilus) - N/A > >>> > upgrade/client-upgrade-jewel (nautilus) - PASSED > >>> > upgrade/client-upgrade-mimic (nautilus) - FAILED > >>> > upgrade/luminous-p2p - in progress > >>> > powercycle - in progress > >>> > ceph-ansible - Brad is finxing > >>> > upgrade/luminous-x (nautilus) - in progress > >>> > upgrade/mimic-x (nautilus) - in progress > >>> > ceph-volume - Jan fixing > >>> > (please speak up if something is missing) > >>> > > >>> > Thx > >>> > YuriW > >>> > > _______________________________________________ > > Dev mailing list -- dev@ceph.io > > To unsubscribe send an email to dev-leave@ceph.io > > -- > Abhishek Lekshmanan > SUSE Software Solutions Germany GmbH > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) > _______________________________________________ > Dev mailing list -- dev@ceph.io > To unsubscribe send an email to dev-leave@ceph.io > _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

Sorry for nagging, but when will 14.2.5 actually be released? On Fri, Dec 6, 2019, 10:32 AM Alfredo Deza <adeza@redhat.com> wrote:

On Fri, Dec 6, 2019 at 9:52 AM Sage Weil <sweil@redhat.com> wrote:

...

On Fri, 6 Dec 2019, Alfredo Deza wrote:

...

I would like to raise once again the discussion of improving our release process so that we can minimize these things happening at the very last minute.

This isn't the first time we've hit issues that had to be reverted (luckily packages didn't get to download.ceph.com), and since we insist on owning the package and repository creation, there is no way we can retire a "bad" release that is available publicly.

Doesn't the fact that we own the distribution mean that we *can* yank a bad release? If we were relying on downstreams to do this for us we'd have no control at all. Currently the only reason we can't is because we're chafing against the tools.

...

A few of the items that I believe could improve are:

* Start having stricter development and freeze periods * Add a quiet period (or waiting time) of a week for a release to be

...

...

* Determine what the SHA to release is, so that testing can be done against that SHA * Avoid depending on back and forth emails asking leads if they are OK with a release or if they need another PR to be merged - let's define criteria to meet this.

I think the main missing piece here is deploying the release candidate into production before actually releasing. Most of these last minute issues were caught when we deployed on the lab cluster and my home cluster. I think this is similar to your 'quiet' period: once qa

public passes,

...

we should deploy internally and make sure there aren't things the qa suite missed.

Yes exactly, a release candidate would be great.

...

The other problem was discovered in parallel by David and was totally unrelated to the release process/freeze/testing, but was deemed to be a release blocker. I don't think any process can avoid that.

And by no means I want to raise the release issues as something specific to this last problem, but something that does need to be addressed.

...

This release is a bit different, though, because there is a data corruption bug we really want to get out. Perhaps what we should have done here is a minimal release that only has that one bug fix so that our regression risk is basically 0 and we can expedite. Instead we grabbed everything that had already been tested and merged and went with that.

Yes exactly. We've had similar "merge with what was tested and release" situations before that have not worked well.

...

sage

...

On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha@redhat.com> wrote:

...

https://github.com/ceph/ceph/pull/32018 has merged, we should be

ready

...

...

...

to build 14.2.5 now.

On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@redhat.com> wrote:

...

David Zafman has discovered a buggy patch in nautilus, which we

want

...

to revert before releasing 14.2.5. More details in https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. We'll need to rebuild 14.2.5 once the revert merges.

Sorry about the inconvenience.

Neha

On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@redhat.com> wrote:

...

On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote: > Yuri Weinstein <yweinste@redhat.com> writes: > > > David, assuming Sage is OK with `ceph-deploy` and overall

tests

...

> > results, this is ready for publishing. > > Sage, is this ready to build and start publishing packages for?

Yeah, I think we're good to go!

Thanks everyone- sage

> > Abhishek, Nathan FYI > > > > On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote: > >> > >> > >> > >> On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein < yweinste@redhat.com> wrote: > >>> > >>> Outstanding need approval: > >>> > >>> ceph-deploy - Sage > >>> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > >> > >> approved > >> > >>> > >>> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > >> > >> approved, failure tracked in https://tracker.ceph.com/issues/43048 > >> > >>> > >>> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein < yweinste@redhat.com> wrote: > >>> > > >>> > (This is an early update, some tests are still running, as we are > >>> > trying to release this point next week before the US holidays, and > >>> > have more time to review results) > >>> > > >>> > Details of this release summarized here: > >>> > https://tracker.ceph.com/issues/42839#note-3 > >>> > > >>> > rados - approved by Neha > >>> > rgw - approved by Casey > >>> > rbd - need approval Jason > >>> > krbd - need approval Jason, Ilya > >>> > fs - need approval Patrick, Ramana > >>> > kcephfs - need approval Patrick, Ramana > >>> > multimds - need approval Patrick, Ramana > >>> > ceph-deploy - FAILED Sage, Alfredo ? > >>> > ceph-disk - N/A > >>> > upgrade/client-upgrade-hammer (nautilus) - N/A > >>> > upgrade/client-upgrade-jewel (nautilus) - PASSED > >>> > upgrade/client-upgrade-mimic (nautilus) - FAILED > >>> > upgrade/luminous-p2p - in progress > >>> > powercycle - in progress > >>> > ceph-ansible - Brad is finxing > >>> > upgrade/luminous-x (nautilus) - in progress > >>> > upgrade/mimic-x (nautilus) - in progress > >>> > ceph-volume - Jan fixing > >>> > (please speak up if something is missing) > >>> > > >>> > Thx > >>> > YuriW > >>> > > _______________________________________________ > > Dev mailing list -- dev@ceph.io > > To unsubscribe send an email to dev-leave@ceph.io > > -- > Abhishek Lekshmanan > SUSE Software Solutions Germany GmbH > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) > _______________________________________________ > Dev mailing list -- dev@ceph.io > To unsubscribe send an email to dev-leave@ceph.io > _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

---

Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

Alfredo Deza <adeza@redhat.com> writes:

I would like to raise once again the discussion of improving our release process so that we can minimize these things happening at the very last minute.

This isn't the first time we've hit issues that had to be reverted (luckily packages didn't get to download.ceph.com), and since we insist on owning the package and repository creation, there is no way we can retire a "bad" release that is available publicly.

In this case, there wasn't a build kicked off yet, so there shouldn't be packages out yet, I agree improvements are needed. Maybe waiting for a day after Sage's approval allows for these things to be caught like in this release

A few of the items that I believe could improve are:

* Start having stricter development and freeze periods * Add a quiet period (or waiting time) of a week for a release to be public * Determine what the SHA to release is, so that testing can be done against that SHA

So far QE is freezed on a SHA, and now repos are also locked down, but finding bugs outside of QE suite is usually one of the triggers that causes a QE validated test also going back for build again. As we improve our coverage hopefully this should die down (very optimistic forward looking statements)

* Avoid depending on back and forth emails asking leads if they are OK with a release or if they need another PR to be merged - let's define criteria to meet this.

I agree this increases the lead time, but here an EAFP approach often leads to this critical fix was needed sort of a thing. Finalizing at the CLT might be easier given the final results aren't out on a thursday.

On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha@redhat.com> wrote:

...

https://github.com/ceph/ceph/pull/32018 has merged, we should be ready to build 14.2.5 now.

On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@redhat.com> wrote:

...

David Zafman has discovered a buggy patch in nautilus, which we want to revert before releasing 14.2.5. More details in https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. We'll need to rebuild 14.2.5 once the revert merges.

Sorry about the inconvenience.

Neha

On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@redhat.com> wrote:

...

On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote:

...

Yuri Weinstein <yweinste@redhat.com> writes:

...

David, assuming Sage is OK with `ceph-deploy` and overall tests results, this is ready for publishing.

Sage, is this ready to build and start publishing packages for?

Yeah, I think we're good to go!

Thanks everyone- sage

...

...

Abhishek, Nathan FYI

On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@redhat.com> wrote: > > > > On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@redhat.com> wrote: >> >> Outstanding need approval: >> >> ceph-deploy - Sage >> upgrade/luminous-x (nautilus) - Neha, Josh reviewing > > approved > >> >> upgrade/mimic-x (nautilus) - Neha, Josh reviewing > > approved, failure tracked in https://tracker.ceph.com/issues/43048 > >> >> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@redhat.com> wrote: >> > >> > (This is an early update, some tests are still running, as we are >> > trying to release this point next week before the US holidays, and >> > have more time to review results) >> > >> > Details of this release summarized here: >> > https://tracker.ceph.com/issues/42839#note-3 >> > >> > rados - approved by Neha >> > rgw - approved by Casey >> > rbd - need approval Jason >> > krbd - need approval Jason, Ilya >> > fs - need approval Patrick, Ramana >> > kcephfs - need approval Patrick, Ramana >> > multimds - need approval Patrick, Ramana >> > ceph-deploy - FAILED Sage, Alfredo ? >> > ceph-disk - N/A >> > upgrade/client-upgrade-hammer (nautilus) - N/A >> > upgrade/client-upgrade-jewel (nautilus) - PASSED >> > upgrade/client-upgrade-mimic (nautilus) - FAILED >> > upgrade/luminous-p2p - in progress >> > powercycle - in progress >> > ceph-ansible - Brad is finxing >> > upgrade/luminous-x (nautilus) - in progress >> > upgrade/mimic-x (nautilus) - in progress >> > ceph-volume - Jan fixing >> > (please speak up if something is missing) >> > >> > Thx >> > YuriW >> _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)