Re: [ceph-users] Re: reef 18.2.3 QE validation status
Please see the update from Laura below for the status of this release. Dev Leads help is appreciated to expedite fixes necessary to publish it soon. "Hi all, we have hit another blocker with this release. Due to centos 8 stream going end of life, we only have the option of releasing centos 9 stream containers. However, we did not test the efficacy of centos 9 stream containers against the orch and upgrade suites during the initial 18.2.3 release cycle. This problem is tracked here: https://tracker.ceph.com/issues/66334 What needs to happen now is: 1. The orch team needs to fix all references to centos 8 stream in the orch suite 2. fs, rados, etc. need to fix their relative jobs the same way in the upgrade suite The easiest way to tackle that is to raise a PR against main and backport to stable releases since this problem actually affects main and all other releases. Then, we will: 1. Rerun orch and upgrade with these fixes 2. Re-approve orch and upgrade 3. Re-upgrade gibba and LRC Then the release will be unblocked." On Tue, Jun 4, 2024 at 3:26 PM Laura Flores <lflores@redhat.com> wrote:
Rados results were approved, and we successfully upgraded the gibba cluster.
Now waiting on @Dan Mick <dmick@redhat.com> to upgrade the LRC.
On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein <yweinste@redhat.com> wrote:
I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits and seeking approvals from Radek and Laure
https://tracker.ceph.com/issues/65393#note-1
On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein <yweinste@redhat.com> wrote:
We have discovered some issues (#1 and #2) during the final stages of testing that require considering a delay in this point release until all options and risks are assessed and resolved.
We will keep you all updated on the progress.
Thank you for your patience!
#1 https://tracker.ceph.com/issues/66260 #2 https://tracker.ceph.com/issues/61948#note-21
On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein <yweinste@redhat.com>
We've run into a problem during the last verification steps before publishing this release after upgrading the LRC to it => https://tracker.ceph.com/issues/65733
After this issue is resolved, we will continue testing and publishing this point release.
Thanks for your patience!
On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann <christian.rohmann@inovex.de> wrote:
On 18.04.24 8:13 PM, Laura Flores wrote:
Thanks for bringing this to our attention. The leads have decided
wrote: that
since this PR hasn't been merged to main yet and isn't approved, it will not go in v18.2.3, but it will be prioritized for v18.2.4. I've already added the PR to the v18.2.4 milestone so it's sure to be picked up.
Thanks a bunch. If you miss the train, you miss the train - fair enough. Nice to know there is another one going soon and that bug is going to be on it !
Regards
Christian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
After fixing the issues identified below we cherry-picked all PRs from this list for 18.2.3 https://pad.ceph.com/p/release-cherry-pick-coordination. The question to the dev leads: do you think we can proceed with the release without rerunning suites, as they were already approved? Please reply with your recommendations. On Thu, Jun 6, 2024 at 4:59 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Please see the update from Laura below for the status of this release.
Dev Leads help is appreciated to expedite fixes necessary to publish it soon.
"Hi all, we have hit another blocker with this release.
Due to centos 8 stream going end of life, we only have the option of releasing centos 9 stream containers.
However, we did not test the efficacy of centos 9 stream containers against the orch and upgrade suites during the initial 18.2.3 release cycle.
This problem is tracked here: https://tracker.ceph.com/issues/66334
What needs to happen now is:
1. The orch team needs to fix all references to centos 8 stream in the orch suite 2. fs, rados, etc. need to fix their relative jobs the same way in the upgrade suite
The easiest way to tackle that is to raise a PR against main and backport to stable releases since this problem actually affects main and all other releases.
Then, we will:
1. Rerun orch and upgrade with these fixes 2. Re-approve orch and upgrade 3. Re-upgrade gibba and LRC
Then the release will be unblocked."
On Tue, Jun 4, 2024 at 3:26 PM Laura Flores <lflores@redhat.com> wrote:
Rados results were approved, and we successfully upgraded the gibba cluster.
Now waiting on @Dan Mick <dmick@redhat.com> to upgrade the LRC.
On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein <yweinste@redhat.com> wrote:
I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits and seeking approvals from Radek and Laure
https://tracker.ceph.com/issues/65393#note-1
On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein <yweinste@redhat.com> wrote:
We have discovered some issues (#1 and #2) during the final stages of testing that require considering a delay in this point release until all options and risks are assessed and resolved.
We will keep you all updated on the progress.
Thank you for your patience!
#1 https://tracker.ceph.com/issues/66260 #2 https://tracker.ceph.com/issues/61948#note-21
On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein <yweinste@redhat.com>
wrote:
We've run into a problem during the last verification steps before publishing this release after upgrading the LRC to it => https://tracker.ceph.com/issues/65733
After this issue is resolved, we will continue testing and publishing this point release.
Thanks for your patience!
On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann <christian.rohmann@inovex.de> wrote:
On 18.04.24 8:13 PM, Laura Flores wrote: > Thanks for bringing this to our attention. The leads have
decided that
> since this PR hasn't been merged to main yet and isn't approved, it > will not go in v18.2.3, but it will be prioritized for v18.2.4. > I've already added the PR to the v18.2.4 milestone so it's sure to be > picked up.
Thanks a bunch. If you miss the train, you miss the train - fair enough. Nice to know there is another one going soon and that bug is going to be on it !
Regards
Christian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
(cc Thomas Goirand) in April, an 18.2.3 tarball was uploaded to https://download.ceph.com/tarballs/ceph_18.2.3.orig.tar.gz. that's been picked up and packaged by the Debian project under the assumption that it was a supported release when we do finally release 18.2.3, we will presumably overwrite that existing tarball with the latest contents. Thomas has requested that we re-number this release to 18.2.4 to prevent further confusion. Thomas, could you explain the consequences of overwriting that tarball in place? On Tue, Jul 2, 2024 at 10:06 AM Yuri Weinstein <yweinste@redhat.com> wrote:
After fixing the issues identified below we cherry-picked all PRs from this list for 18.2.3 https://pad.ceph.com/p/release-cherry-pick-coordination.
The question to the dev leads: do you think we can proceed with the release without rerunning suites, as they were already approved?
Please reply with your recommendations.
On Thu, Jun 6, 2024 at 4:59 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Please see the update from Laura below for the status of this release.
Dev Leads help is appreciated to expedite fixes necessary to publish it soon.
"Hi all, we have hit another blocker with this release.
Due to centos 8 stream going end of life, we only have the option of releasing centos 9 stream containers.
However, we did not test the efficacy of centos 9 stream containers against the orch and upgrade suites during the initial 18.2.3 release cycle.
This problem is tracked here: https://tracker.ceph.com/issues/66334
What needs to happen now is:
1. The orch team needs to fix all references to centos 8 stream in the orch suite 2. fs, rados, etc. need to fix their relative jobs the same way in the upgrade suite
The easiest way to tackle that is to raise a PR against main and backport to stable releases since this problem actually affects main and all other releases.
Then, we will:
1. Rerun orch and upgrade with these fixes 2. Re-approve orch and upgrade 3. Re-upgrade gibba and LRC
Then the release will be unblocked."
On Tue, Jun 4, 2024 at 3:26 PM Laura Flores <lflores@redhat.com> wrote:
Rados results were approved, and we successfully upgraded the gibba cluster.
Now waiting on @Dan Mick <dmick@redhat.com> to upgrade the LRC.
On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein <yweinste@redhat.com> wrote:
I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits and seeking approvals from Radek and Laure
https://tracker.ceph.com/issues/65393#note-1
On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein <yweinste@redhat.com> wrote:
We have discovered some issues (#1 and #2) during the final stages of testing that require considering a delay in this point release until all options and risks are assessed and resolved.
We will keep you all updated on the progress.
Thank you for your patience!
#1 https://tracker.ceph.com/issues/66260 #2 https://tracker.ceph.com/issues/61948#note-21
On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein <yweinste@redhat.com>
We've run into a problem during the last verification steps before publishing this release after upgrading the LRC to it => https://tracker.ceph.com/issues/65733
After this issue is resolved, we will continue testing and
wrote: publishing
this point release.
Thanks for your patience!
On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann <christian.rohmann@inovex.de> wrote: > > On 18.04.24 8:13 PM, Laura Flores wrote: > > Thanks for bringing this to our attention. The leads have decided that > > since this PR hasn't been merged to main yet and isn't approved, it > > will not go in v18.2.3, but it will be prioritized for v18.2.4. > > I've already added the PR to the v18.2.4 milestone so it's sure to be > > picked up. > > Thanks a bunch. If you miss the train, you miss the train - fair enough. > Nice to know there is another one going soon and that bug is going to be > on it ! > > > Regards > > Christian > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Wed, Jul 3, 2024 at 11:17 AM Casey Bodley <cbodley@redhat.com> wrote:
(cc Thomas Goirand)
in April, an 18.2.3 tarball was uploaded to https://download.ceph.com/tarballs/ceph_18.2.3.orig.tar.gz. that's been picked up and packaged by the Debian project under the assumption that it was a supported release
when we do finally release 18.2.3, we will presumably overwrite that existing tarball with the latest contents. Thomas has requested that we re-number this release to 18.2.4 to prevent further confusion. Thomas, could you explain the consequences of overwriting that tarball in place?
Debian is not alone in this regard. :-/ Putting tar files on download.ceph.com/tarballs ahead of the *real* GA is misleading at best. Replacing the tar file is problematic too, if only because it's a potential source of confusion for people who aren't paying attention. AIUI, after the tar file lands, if the testing is successful, a corresponding src.rpm and a set of debian packaging files land in download.ceph.com/rpm-$ver/... and download.ceph.com/debian-$ver/... respectively. I gather that most of the time testing goes well, and the sources download.ceph.com/tarballs/... match whats in download.ceph.com/{rpm,debian}-$ver and match what's tagged in github. But IIRC the 18.2.2 testing did not go well what was ultimately tagged as 18.2.2 does not match the tar file in .../tarballs/ceph-18.2.2.tar.gz. (Or did the tar file get replaced?) And the same thing is about to happen again with 18.2.3. I'm not sure I believe that making this next release 18.2.4 really solves anything, especially if we're not solving the underlying problem. Can we at least fix the mistake of uploading a tar file to .../tarballs if it's not the final, official tar file that matches what is tagged in github. I'm sort of okay with replacing the 18.2.3 tar file. I'll just respin Fedora and CentOS Storage SIG packages, either with the new tarball, or with the sources extracted from the GA .../rpm-$ver or .../debian-$ver. IOW with a higher Release than the previous spins) (Fedora and CentOS track the source through their look-aside cache and the sources file in distgit, so there's actually a trail of cookie crumbs.) Otherwise I suppose I will be forced to wait for files to land in .../rpm-$ver and .../debian-$ver if the tar file in .../tarballs can't be relied on.
On Tue, Jul 2, 2024 at 10:06 AM Yuri Weinstein <yweinste@redhat.com> wrote:
After fixing the issues identified below we cherry-picked all PRs from this list for 18.2.3 https://pad.ceph.com/p/release-cherry-pick-coordination.
The question to the dev leads: do you think we can proceed with the release without rerunning suites, as they were already approved?
Please reply with your recommendations.
On Thu, Jun 6, 2024 at 4:59 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Please see the update from Laura below for the status of this release.
Dev Leads help is appreciated to expedite fixes necessary to publish it soon.
"Hi all, we have hit another blocker with this release.
Due to centos 8 stream going end of life, we only have the option of releasing centos 9 stream containers.
However, we did not test the efficacy of centos 9 stream containers against the orch and upgrade suites during the initial 18.2.3 release cycle.
This problem is tracked here: https://tracker.ceph.com/issues/66334
What needs to happen now is:
1. The orch team needs to fix all references to centos 8 stream in the orch suite 2. fs, rados, etc. need to fix their relative jobs the same way in the upgrade suite
The easiest way to tackle that is to raise a PR against main and backport to stable releases since this problem actually affects main and all other releases.
Then, we will:
1. Rerun orch and upgrade with these fixes 2. Re-approve orch and upgrade 3. Re-upgrade gibba and LRC
Then the release will be unblocked."
On Tue, Jun 4, 2024 at 3:26 PM Laura Flores <lflores@redhat.com> wrote:
Rados results were approved, and we successfully upgraded the gibba cluster.
Now waiting on @Dan Mick <dmick@redhat.com> to upgrade the LRC.
On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein <yweinste@redhat.com> wrote:
I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits and seeking approvals from Radek and Laure
https://tracker.ceph.com/issues/65393#note-1
On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein <yweinste@redhat.com> wrote:
We have discovered some issues (#1 and #2) during the final stages of testing that require considering a delay in this point release until all options and risks are assessed and resolved.
We will keep you all updated on the progress.
Thank you for your patience!
#1 https://tracker.ceph.com/issues/66260 #2 https://tracker.ceph.com/issues/61948#note-21
On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein <yweinste@redhat.com>
> > We've run into a problem during the last verification steps before > publishing this release after upgrading the LRC to it => > https://tracker.ceph.com/issues/65733 > > After this issue is resolved, we will continue testing and
wrote: publishing
> this point release. > > Thanks for your patience! > > On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann > <christian.rohmann@inovex.de> wrote: > > > > On 18.04.24 8:13 PM, Laura Flores wrote: > > > Thanks for bringing this to our attention. The leads have decided that > > > since this PR hasn't been merged to main yet and isn't approved, it > > > will not go in v18.2.3, but it will be prioritized for v18.2.4. > > > I've already added the PR to the v18.2.4 milestone so it's sure to be > > > picked up. > > > > Thanks a bunch. If you miss the train, you miss the train - fair enough. > > Nice to know there is another one going soon and that bug is going to be > > on it ! > > > > > > Regards > > > > Christian > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > >
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Kaleb
Hi Yuri, On Tue, Jul 2, 2024 at 7:36 PM Yuri Weinstein <yweinste@redhat.com> wrote:
After fixing the issues identified below we cherry-picked all PRs from this list for 18.2.3 https://pad.ceph.com/p/release-cherry-pick-coordination.
The question to the dev leads: do you think we can proceed with the release without rerunning suites, as they were already approved?
Yes, I think so it's safe to proceed (at least for CephFS). If time permits, an fs:upgrade run would not hurt.
Please reply with your recommendations.
On Thu, Jun 6, 2024 at 4:59 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Please see the update from Laura below for the status of this release.
Dev Leads help is appreciated to expedite fixes necessary to publish it soon.
"Hi all, we have hit another blocker with this release.
Due to centos 8 stream going end of life, we only have the option of releasing centos 9 stream containers.
However, we did not test the efficacy of centos 9 stream containers against the orch and upgrade suites during the initial 18.2.3 release cycle.
This problem is tracked here: https://tracker.ceph.com/issues/66334
What needs to happen now is:
1. The orch team needs to fix all references to centos 8 stream in the orch suite 2. fs, rados, etc. need to fix their relative jobs the same way in the upgrade suite
The easiest way to tackle that is to raise a PR against main and backport to stable releases since this problem actually affects main and all other releases.
Then, we will:
1. Rerun orch and upgrade with these fixes 2. Re-approve orch and upgrade 3. Re-upgrade gibba and LRC
Then the release will be unblocked."
On Tue, Jun 4, 2024 at 3:26 PM Laura Flores <lflores@redhat.com> wrote:
Rados results were approved, and we successfully upgraded the gibba cluster.
Now waiting on @Dan Mick <dmick@redhat.com> to upgrade the LRC.
On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein <yweinste@redhat.com> wrote:
I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits and seeking approvals from Radek and Laure
https://tracker.ceph.com/issues/65393#note-1
On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein <yweinste@redhat.com> wrote:
We have discovered some issues (#1 and #2) during the final stages of testing that require considering a delay in this point release until all options and risks are assessed and resolved.
We will keep you all updated on the progress.
Thank you for your patience!
#1 https://tracker.ceph.com/issues/66260 #2 https://tracker.ceph.com/issues/61948#note-21
On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein <yweinste@redhat.com>
We've run into a problem during the last verification steps before publishing this release after upgrading the LRC to it => https://tracker.ceph.com/issues/65733
After this issue is resolved, we will continue testing and
wrote: publishing
this point release.
Thanks for your patience!
On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann <christian.rohmann@inovex.de> wrote: > > On 18.04.24 8:13 PM, Laura Flores wrote: > > Thanks for bringing this to our attention. The leads have decided that > > since this PR hasn't been merged to main yet and isn't approved, it > > will not go in v18.2.3, but it will be prioritized for v18.2.4. > > I've already added the PR to the v18.2.4 milestone so it's sure to be > > picked up. > > Thanks a bunch. If you miss the train, you miss the train - fair enough. > Nice to know there is another one going soon and that bug is going to be > on it ! > > > Regards > > Christian > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Cheers, Venky
participants (4)
-
Casey Bodley
-
Kaleb Keithley
-
Venky Shankar
-
Yuri Weinstein