squid 19.1.1 RC QE validation status
Details of this release are summarized here: https://tracker.ceph.com/issues/67340#note-1 Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD Seeking approvals/reviews for: rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready) rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume Pls let me know if any tests were missed from this list.
On Mon, Aug 5, 2024 at 10:32 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
rbd and krbd approved. (For krbd, https://tracker.ceph.com/issues/67353 popped up, but it's not blocker.) Thanks, Ilya
The PR from the core team https://github.com/ceph/ceph/pull/59020 was cherry-picked and rados tests are running. Still seeking reviews and approvals for other suites. On Mon, Aug 5, 2024 at 1:53 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Mon, Aug 5, 2024 at 10:32 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
rbd and krbd approved.
(For krbd, https://tracker.ceph.com/issues/67353 popped up, but it's not blocker.)
Thanks,
Ilya
orch approved On Mon, Aug 5, 2024 at 4:33 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey @Adam King <adking@redhat.com>, can you take a look at this tracker? https://tracker.ceph.com/issues/66883#note-26 I summarized the full issue in the last note. I believe it is an orch problem blocking the upgrade tests, and I would like to hear your thoughts. On Fri, Aug 9, 2024 at 9:14 AM Adam King <adking@redhat.com> wrote:
orch approved
On Mon, Aug 5, 2024 at 4:33 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Hi Yuri, On Tue, Aug 6, 2024 at 2:03 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky
There are a couple of new bugs seen in this run, but those are not blockers for this RC. Failures are noted here https://tracker.ceph.com/projects/cephfs/wiki/Squid#2024-Aug-13-RC-1911 Consider this run as "fs approved".
orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
Still waiting to hear back: rgw - Eric, Adam E quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume On Tue, Aug 13, 2024 at 9:27 PM Venky Shankar <vshankar@redhat.com> wrote:
Hi Yuri,
On Tue, Aug 6, 2024 at 2:03 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky
There are a couple of new bugs seen in this run, but those are not blockers for this RC.
Failures are noted here
https://tracker.ceph.com/projects/cephfs/wiki/Squid#2024-Aug-13-RC-1911
Consider this run as "fs approved".
orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Crimson approved (Failures are known). On Wed, Aug 14, 2024 at 6:05 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Still waiting to hear back:
rgw - Eric, Adam E quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume
On Tue, Aug 13, 2024 at 9:27 PM Venky Shankar <vshankar@redhat.com> wrote:
Hi Yuri,
On Tue, Aug 6, 2024 at 2:03 AM Yuri Weinstein <yweinste@redhat.com>
wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky
There are a couple of new bugs seen in this run, but those are not blockers for this RC.
Failures are noted here
https://tracker.ceph.com/projects/cephfs/wiki/Squid#2024-Aug-13-RC-1911
Consider this run as "fs approved".
orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey @Yuri Weinstein <yweinste@redhat.com>, We've fixed a couple of issues and now need a few things rerun. 1. *Can you please rerun upgrade/reef-x and upgrade/quincy-x? * - Reasoning: Many jobs in those suites died due to https://tracker.ceph.com/issues/66883, which we deduced was a recent merge to teuthology. Now that the affecting commit was reverted, we are ready to have those rerun. 2. *Can you please cherry-pick https://github.com/ceph/ceph/pull/58607 <https://github.com/ceph/ceph/pull/58607> to squid-release and reschedule rados:thrash-old-clients?* - Reasoning: Since we stopped building focal for squid, we can no longer test squid against pacific clients. - For this second RC, we had to make the decision to drop pacific from the *rados:thrash-old-clients* tests, which will now use centos 9 stream packages to test against only reef and quincy clients ( https://github.com/ceph/ceph/pull/58607). - We have raised https://tracker.ceph.com/issues/67469 to track the implementation of a containerized solution for older clients that don't have centos 9 stream packages, so that we can reincorporate pacific in the future. After these two things are rescheduled, we can proceed with a rados suite approval and an upgrade suite approval. Thanks, Laura On Wed, Aug 14, 2024 at 12:49 PM Adam Emerson <aemerson@redhat.com> wrote:
On 14/08/2024, Yuri Weinstein wrote:
Still waiting to hear back:
rgw - Eric, Adam E
Approved.
(Sorry, I thought we were supposed to reply on the tracker.) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Laura, The PR was cherry-picked and the `squid-release` branch was built. Please review the run results in the tracker. On Wed, Aug 14, 2024 at 2:18 PM Laura Flores <lflores@redhat.com> wrote:
Hey @Yuri Weinstein <yweinste@redhat.com>,
We've fixed a couple of issues and now need a few things rerun.
1. *Can you please rerun upgrade/reef-x and upgrade/quincy-x? * - Reasoning: Many jobs in those suites died due to https://tracker.ceph.com/issues/66883, which we deduced was a recent merge to teuthology. Now that the affecting commit was reverted, we are ready to have those rerun. 2. *Can you please cherry-pick https://github.com/ceph/ceph/pull/58607 <https://github.com/ceph/ceph/pull/58607> to squid-release and reschedule rados:thrash-old-clients?* - Reasoning: Since we stopped building focal for squid, we can no longer test squid against pacific clients. - For this second RC, we had to make the decision to drop pacific from the *rados:thrash-old-clients* tests, which will now use centos 9 stream packages to test against only reef and quincy clients ( https://github.com/ceph/ceph/pull/58607). - We have raised https://tracker.ceph.com/issues/67469 to track the implementation of a containerized solution for older clients that don't have centos 9 stream packages, so that we can reincorporate pacific in the future.
After these two things are rescheduled, we can proceed with a rados suite approval and an upgrade suite approval.
Thanks, Laura
On Wed, Aug 14, 2024 at 12:49 PM Adam Emerson <aemerson@redhat.com> wrote:
On 14/08/2024, Yuri Weinstein wrote:
Still waiting to hear back:
rgw - Eric, Adam E
Approved.
(Sorry, I thought we were supposed to reply on the tracker.) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Rados approved. Failures tracked here: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1911-httpstrackercephcom... On Thu, Aug 15, 2024 at 11:30 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Laura,
The PR was cherry-picked and the `squid-release` branch was built. Please review the run results in the tracker.
On Wed, Aug 14, 2024 at 2:18 PM Laura Flores <lflores@redhat.com> wrote:
Hey @Yuri Weinstein <yweinste@redhat.com>,
We've fixed a couple of issues and now need a few things rerun.
1. *Can you please rerun upgrade/reef-x and upgrade/quincy-x? * - Reasoning: Many jobs in those suites died due to https://tracker.ceph.com/issues/66883, which we deduced was a
recent
merge to teuthology. Now that the affecting commit was reverted,
we are
ready to have those rerun. 2. *Can you please cherry-pick
<https://github.com/ceph/ceph/pull/58607> to squid-release and reschedule rados:thrash-old-clients?* - Reasoning: Since we stopped building focal for squid, we can no longer test squid against pacific clients. - For this second RC, we had to make the decision to drop pacific from the *rados:thrash-old-clients* tests, which will now use centos 9 stream packages to test against only reef and quincy clients ( https://github.com/ceph/ceph/pull/58607). - We have raised https://tracker.ceph.com/issues/67469 to track
https://github.com/ceph/ceph/pull/58607 the
implementation of a containerized solution for older clients that
don't
have centos 9 stream packages, so that we can reincorporate pacific in the future.
After these two things are rescheduled, we can proceed with a rados suite approval and an upgrade suite approval.
Thanks, Laura
On Wed, Aug 14, 2024 at 12:49 PM Adam Emerson <aemerson@redhat.com>
wrote:
On 14/08/2024, Yuri Weinstein wrote:
Still waiting to hear back:
rgw - Eric, Adam E
Approved.
(Sorry, I thought we were supposed to reply on the tracker.) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
The upgrade suites look mostly good to me, except for one tracker I think would be in @Adam King <adking@redhat.com>'s realm to look at. If the new grafana issue below is deemed okay, then we can proceed with approving the upgrade suite. *This issue stood out to me, where the cluster had trouble pulling the grafana image locally to redeploy it. *@Adam King <adking@redhat.com>* can you take a look?* - *https://tracker.ceph.com/issues/67583 <https://tracker.ceph.com/issues/67583> - upgrade:quincy-x/stress-split: Cluster fails to redeploy grafana daemon after image is unable to be found locally* Otherwise, tests failed from cluster log warnings that are expected during upgrade tests. Many of these warnings have already been fixed and are in the stages of getting backported. I checked for each test that the cluster had upgraded all daemons to 19.1.1, and that was the case. - https://tracker.ceph.com/issues/66602 - rados/upgrade: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - https://tracker.ceph.com/issues/65422 - upgrade/quincy-x: "1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/67584 - upgrade:quincy-x: cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log - https://tracker.ceph.com/issues/64460 - rados/upgrade: "[WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log - https://tracker.ceph.com/issues/66809 - upgrade/quincy-x; upgrade/reef-x: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log On Thu, Aug 15, 2024 at 11:55 AM Laura Flores <lflores@redhat.com> wrote:
Rados approved. Failures tracked here: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1911-httpstrackercephcom...
On Thu, Aug 15, 2024 at 11:30 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Laura,
The PR was cherry-picked and the `squid-release` branch was built. Please review the run results in the tracker.
On Wed, Aug 14, 2024 at 2:18 PM Laura Flores <lflores@redhat.com> wrote:
Hey @Yuri Weinstein <yweinste@redhat.com>,
We've fixed a couple of issues and now need a few things rerun.
1. *Can you please rerun upgrade/reef-x and upgrade/quincy-x? * - Reasoning: Many jobs in those suites died due to https://tracker.ceph.com/issues/66883, which we deduced was a
recent
merge to teuthology. Now that the affecting commit was reverted,
we are
ready to have those rerun. 2. *Can you please cherry-pick
<https://github.com/ceph/ceph/pull/58607> to squid-release and reschedule rados:thrash-old-clients?* - Reasoning: Since we stopped building focal for squid, we can no longer test squid against pacific clients. - For this second RC, we had to make the decision to drop pacific from the *rados:thrash-old-clients* tests, which will now use centos 9 stream packages to test against only reef and quincy clients ( https://github.com/ceph/ceph/pull/58607). - We have raised https://tracker.ceph.com/issues/67469 to track
https://github.com/ceph/ceph/pull/58607 the
implementation of a containerized solution for older clients that
don't
have centos 9 stream packages, so that we can reincorporate pacific in the future.
After these two things are rescheduled, we can proceed with a rados
suite
approval and an upgrade suite approval.
Thanks, Laura
On Wed, Aug 14, 2024 at 12:49 PM Adam Emerson <aemerson@redhat.com> wrote:
On 14/08/2024, Yuri Weinstein wrote:
Still waiting to hear back:
rgw - Eric, Adam E
Approved.
(Sorry, I thought we were supposed to reply on the tracker.) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
https://tracker.ceph.com/issues/67583 didn't reproduce across 10 reruns https://pulpito.ceph.com/lflores-2024-08-16_00:04:51-upgrade:quincy-x-squid-.... Given the original failure was just "Unable to find image ' quay.io/ceph/grafana:9.4.12' locally" which doesn't look very serious anyway, I don't think there's any reason for the failure to hold up the release On Thu, Aug 15, 2024 at 6:53 PM Laura Flores <lflores@redhat.com> wrote:
The upgrade suites look mostly good to me, except for one tracker I think would be in @Adam King <adking@redhat.com>'s realm to look at. If the new grafana issue below is deemed okay, then we can proceed with approving the upgrade suite.
*This issue stood out to me, where the cluster had trouble pulling the grafana image locally to redeploy it. *@Adam King <adking@redhat.com>* can you take a look?*
- *https://tracker.ceph.com/issues/67583 <https://tracker.ceph.com/issues/67583> - upgrade:quincy-x/stress-split: Cluster fails to redeploy grafana daemon after image is unable to be found locally*
Otherwise, tests failed from cluster log warnings that are expected during upgrade tests. Many of these warnings have already been fixed and are in the stages of getting backported. I checked for each test that the cluster had upgraded all daemons to 19.1.1, and that was the case.
- https://tracker.ceph.com/issues/66602 - rados/upgrade: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - https://tracker.ceph.com/issues/65422 - upgrade/quincy-x: "1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/67584 - upgrade:quincy-x: cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log - https://tracker.ceph.com/issues/64460 - rados/upgrade: "[WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log - https://tracker.ceph.com/issues/66809 - upgrade/quincy-x; upgrade/reef-x: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
On Thu, Aug 15, 2024 at 11:55 AM Laura Flores <lflores@redhat.com> wrote:
Rados approved. Failures tracked here: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1911-httpstrackercephcom...
On Thu, Aug 15, 2024 at 11:30 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Laura,
The PR was cherry-picked and the `squid-release` branch was built. Please review the run results in the tracker.
On Wed, Aug 14, 2024 at 2:18 PM Laura Flores <lflores@redhat.com> wrote:
Hey @Yuri Weinstein <yweinste@redhat.com>,
We've fixed a couple of issues and now need a few things rerun.
1. *Can you please rerun upgrade/reef-x and upgrade/quincy-x? * - Reasoning: Many jobs in those suites died due to https://tracker.ceph.com/issues/66883, which we deduced was a
recent
merge to teuthology. Now that the affecting commit was reverted,
we are
ready to have those rerun. 2. *Can you please cherry-pick
<https://github.com/ceph/ceph/pull/58607> to squid-release and reschedule rados:thrash-old-clients?* - Reasoning: Since we stopped building focal for squid, we can no longer test squid against pacific clients. - For this second RC, we had to make the decision to drop pacific from the *rados:thrash-old-clients* tests, which will now use centos 9 stream packages to test against only reef and quincy clients ( https://github.com/ceph/ceph/pull/58607). - We have raised https://tracker.ceph.com/issues/67469 to track
https://github.com/ceph/ceph/pull/58607 the
implementation of a containerized solution for older clients
that don't
have centos 9 stream packages, so that we can reincorporate pacific in the future.
After these two things are rescheduled, we can proceed with a rados
suite
approval and an upgrade suite approval.
Thanks, Laura
On Wed, Aug 14, 2024 at 12:49 PM Adam Emerson <aemerson@redhat.com> wrote:
On 14/08/2024, Yuri Weinstein wrote:
Still waiting to hear back:
rgw - Eric, Adam E
Approved.
(Sorry, I thought we were supposed to reply on the tracker.) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Thanks @Adam King <adking@redhat.com>! @Yuri Weinstein <yweinste@redhat.com> the upgrade suites are approved. On Mon, Aug 19, 2024 at 9:28 AM Adam King <adking@redhat.com> wrote:
https://tracker.ceph.com/issues/67583 didn't reproduce across 10 reruns https://pulpito.ceph.com/lflores-2024-08-16_00:04:51-upgrade:quincy-x-squid-.... Given the original failure was just "Unable to find image ' quay.io/ceph/grafana:9.4.12' locally" which doesn't look very serious anyway, I don't think there's any reason for the failure to hold up the release
On Thu, Aug 15, 2024 at 6:53 PM Laura Flores <lflores@redhat.com> wrote:
The upgrade suites look mostly good to me, except for one tracker I think would be in @Adam King <adking@redhat.com>'s realm to look at. If the new grafana issue below is deemed okay, then we can proceed with approving the upgrade suite.
*This issue stood out to me, where the cluster had trouble pulling the grafana image locally to redeploy it. *@Adam King <adking@redhat.com>* can you take a look?*
- *https://tracker.ceph.com/issues/67583 <https://tracker.ceph.com/issues/67583> - upgrade:quincy-x/stress-split: Cluster fails to redeploy grafana daemon after image is unable to be found locally*
Otherwise, tests failed from cluster log warnings that are expected during upgrade tests. Many of these warnings have already been fixed and are in the stages of getting backported. I checked for each test that the cluster had upgraded all daemons to 19.1.1, and that was the case.
- https://tracker.ceph.com/issues/66602 - rados/upgrade: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - https://tracker.ceph.com/issues/65422 - upgrade/quincy-x: "1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/67584 - upgrade:quincy-x: cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log - https://tracker.ceph.com/issues/64460 - rados/upgrade: "[WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log - https://tracker.ceph.com/issues/66809 - upgrade/quincy-x; upgrade/reef-x: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
On Thu, Aug 15, 2024 at 11:55 AM Laura Flores <lflores@redhat.com> wrote:
Rados approved. Failures tracked here: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1911-httpstrackercephcom...
On Thu, Aug 15, 2024 at 11:30 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Laura,
The PR was cherry-picked and the `squid-release` branch was built. Please review the run results in the tracker.
On Wed, Aug 14, 2024 at 2:18 PM Laura Flores <lflores@redhat.com> wrote:
Hey @Yuri Weinstein <yweinste@redhat.com>,
We've fixed a couple of issues and now need a few things rerun.
1. *Can you please rerun upgrade/reef-x and upgrade/quincy-x? * - Reasoning: Many jobs in those suites died due to https://tracker.ceph.com/issues/66883, which we deduced was a
recent
merge to teuthology. Now that the affecting commit was
reverted, we are
ready to have those rerun. 2. *Can you please cherry-pick
<https://github.com/ceph/ceph/pull/58607> to squid-release and reschedule rados:thrash-old-clients?* - Reasoning: Since we stopped building focal for squid, we can no longer test squid against pacific clients. - For this second RC, we had to make the decision to drop
https://github.com/ceph/ceph/pull/58607 pacific
from the *rados:thrash-old-clients* tests, which will now use
centos
9 stream packages to test against only reef and quincy clients ( https://github.com/ceph/ceph/pull/58607). - We have raised https://tracker.ceph.com/issues/67469 to
track the
implementation of a containerized solution for older clients
that don't
have centos 9 stream packages, so that we can reincorporate pacific in the future.
After these two things are rescheduled, we can proceed with a rados
suite
approval and an upgrade suite approval.
Thanks, Laura
On Wed, Aug 14, 2024 at 12:49 PM Adam Emerson <aemerson@redhat.com> wrote:
On 14/08/2024, Yuri Weinstein wrote: > Still waiting to hear back: > > rgw - Eric, Adam E
Approved.
(Sorry, I thought we were supposed to reply on the tracker.) _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
We need approval from Guillaume Laura, and gibba upgrade. On Mon, Aug 19, 2024 at 7:31 AM Laura Flores <lflores@redhat.com> wrote:
Thanks @Adam King <adking@redhat.com>!
@Yuri Weinstein <yweinste@redhat.com> the upgrade suites are approved.
On Mon, Aug 19, 2024 at 9:28 AM Adam King <adking@redhat.com> wrote:
https://tracker.ceph.com/issues/67583 didn't reproduce across 10 reruns https://pulpito.ceph.com/lflores-2024-08-16_00:04:51-upgrade:quincy-x-squid-.... Given the original failure was just "Unable to find image ' quay.io/ceph/grafana:9.4.12' locally" which doesn't look very serious anyway, I don't think there's any reason for the failure to hold up the release
On Thu, Aug 15, 2024 at 6:53 PM Laura Flores <lflores@redhat.com> wrote:
The upgrade suites look mostly good to me, except for one tracker I think would be in @Adam King <adking@redhat.com>'s realm to look at. If the new grafana issue below is deemed okay, then we can proceed with approving the upgrade suite.
*This issue stood out to me, where the cluster had trouble pulling the grafana image locally to redeploy it. *@Adam King <adking@redhat.com>* can you take a look?*
- *https://tracker.ceph.com/issues/67583 <https://tracker.ceph.com/issues/67583> - upgrade:quincy-x/stress-split: Cluster fails to redeploy grafana daemon after image is unable to be found locally*
Otherwise, tests failed from cluster log warnings that are expected during upgrade tests. Many of these warnings have already been fixed and are in the stages of getting backported. I checked for each test that the cluster had upgraded all daemons to 19.1.1, and that was the case.
- https://tracker.ceph.com/issues/66602 - rados/upgrade: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - https://tracker.ceph.com/issues/65422 - upgrade/quincy-x: "1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/67584 - upgrade:quincy-x: cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log - https://tracker.ceph.com/issues/64460 - rados/upgrade: "[WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log - https://tracker.ceph.com/issues/66809 - upgrade/quincy-x; upgrade/reef-x: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
On Thu, Aug 15, 2024 at 11:55 AM Laura Flores <lflores@redhat.com> wrote:
Rados approved. Failures tracked here: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1911-httpstrackercephcom...
On Thu, Aug 15, 2024 at 11:30 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Laura,
The PR was cherry-picked and the `squid-release` branch was built. Please review the run results in the tracker.
On Wed, Aug 14, 2024 at 2:18 PM Laura Flores <lflores@redhat.com> wrote:
Hey @Yuri Weinstein <yweinste@redhat.com>,
We've fixed a couple of issues and now need a few things rerun.
1. *Can you please rerun upgrade/reef-x and upgrade/quincy-x? * - Reasoning: Many jobs in those suites died due to https://tracker.ceph.com/issues/66883, which we deduced was a
recent
merge to teuthology. Now that the affecting commit was
reverted, we are
ready to have those rerun. 2. *Can you please cherry-pick
<https://github.com/ceph/ceph/pull/58607> to squid-release and reschedule rados:thrash-old-clients?* - Reasoning: Since we stopped building focal for squid, we can no longer test squid against pacific clients. - For this second RC, we had to make the decision to drop
https://github.com/ceph/ceph/pull/58607 pacific
from the *rados:thrash-old-clients* tests, which will now use
centos
9 stream packages to test against only reef and quincy clients
(
https://github.com/ceph/ceph/pull/58607). - We have raised https://tracker.ceph.com/issues/67469 to
track the
implementation of a containerized solution for older clients
that don't
have centos 9 stream packages, so that we can reincorporate pacific in the future.
After these two things are rescheduled, we can proceed with a rados
suite
approval and an upgrade suite approval.
Thanks, Laura
On Wed, Aug 14, 2024 at 12:49 PM Adam Emerson <aemerson@redhat.com> wrote:
> On 14/08/2024, Yuri Weinstein wrote: > > Still waiting to hear back: > > > > rgw - Eric, Adam E > > Approved. > > (Sorry, I thought we were supposed to reply on the tracker.) > _______________________________________________ > Dev mailing list -- dev@ceph.io > To unsubscribe send an email to dev-leave@ceph.io > >
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
I can do the gibba upgrade after everything's approved. On Mon, Aug 19, 2024 at 9:47 AM Yuri Weinstein <yweinste@redhat.com> wrote:
We need approval from Guillaume
Laura, and gibba upgrade.
On Mon, Aug 19, 2024 at 7:31 AM Laura Flores <lflores@redhat.com> wrote:
Thanks @Adam King <adking@redhat.com>!
@Yuri Weinstein <yweinste@redhat.com> the upgrade suites are approved.
On Mon, Aug 19, 2024 at 9:28 AM Adam King <adking@redhat.com> wrote:
https://tracker.ceph.com/issues/67583 didn't reproduce across 10 reruns https://pulpito.ceph.com/lflores-2024-08-16_00:04:51-upgrade:quincy-x-squid-.... Given the original failure was just "Unable to find image ' quay.io/ceph/grafana:9.4.12' locally" which doesn't look very serious anyway, I don't think there's any reason for the failure to hold up the release
On Thu, Aug 15, 2024 at 6:53 PM Laura Flores <lflores@redhat.com> wrote:
The upgrade suites look mostly good to me, except for one tracker I think would be in @Adam King <adking@redhat.com>'s realm to look at. If the new grafana issue below is deemed okay, then we can proceed with approving the upgrade suite.
*This issue stood out to me, where the cluster had trouble pulling the grafana image locally to redeploy it. *@Adam King <adking@redhat.com>* can you take a look?*
- *https://tracker.ceph.com/issues/67583 <https://tracker.ceph.com/issues/67583> - upgrade:quincy-x/stress-split: Cluster fails to redeploy grafana daemon after image is unable to be found locally*
Otherwise, tests failed from cluster log warnings that are expected during upgrade tests. Many of these warnings have already been fixed and are in the stages of getting backported. I checked for each test that the cluster had upgraded all daemons to 19.1.1, and that was the case.
- https://tracker.ceph.com/issues/66602 - rados/upgrade: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) - https://tracker.ceph.com/issues/65422 - upgrade/quincy-x: "1 pg degraded (PG_DEGRADED)" in cluster log - https://tracker.ceph.com/issues/67584 - upgrade:quincy-x: cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log - https://tracker.ceph.com/issues/64460 - rados/upgrade: "[WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log - https://tracker.ceph.com/issues/66809 - upgrade/quincy-x; upgrade/reef-x: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
On Thu, Aug 15, 2024 at 11:55 AM Laura Flores <lflores@redhat.com> wrote:
Rados approved. Failures tracked here: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1911-httpstrackercephcom...
On Thu, Aug 15, 2024 at 11:30 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Laura,
The PR was cherry-picked and the `squid-release` branch was built. Please review the run results in the tracker.
On Wed, Aug 14, 2024 at 2:18 PM Laura Flores <lflores@redhat.com> wrote: > > Hey @Yuri Weinstein <yweinste@redhat.com>, > > We've fixed a couple of issues and now need a few things rerun. > > > 1. *Can you please rerun upgrade/reef-x and upgrade/quincy-x? * > - Reasoning: Many jobs in those suites died due to > https://tracker.ceph.com/issues/66883, which we deduced was a recent > merge to teuthology. Now that the affecting commit was reverted, we are > ready to have those rerun. > 2. *Can you please cherry-pick https://github.com/ceph/ceph/pull/58607 > <https://github.com/ceph/ceph/pull/58607> to squid-release and reschedule > rados:thrash-old-clients?* > - Reasoning: Since we stopped building focal for squid, we can no > longer test squid against pacific clients. > - For this second RC, we had to make the decision to drop pacific > from the *rados:thrash-old-clients* tests, which will now use centos > 9 stream packages to test against only reef and quincy clients ( > https://github.com/ceph/ceph/pull/58607). > - We have raised https://tracker.ceph.com/issues/67469 to track the > implementation of a containerized solution for older clients that don't > have centos 9 stream packages, so that we can reincorporate > pacific in the > future. > > After these two things are rescheduled, we can proceed with a rados suite > approval and an upgrade suite approval. > > Thanks, > Laura > > On Wed, Aug 14, 2024 at 12:49 PM Adam Emerson <aemerson@redhat.com> wrote: > > > On 14/08/2024, Yuri Weinstein wrote: > > > Still waiting to hear back: > > > > > > rgw - Eric, Adam E > > > > Approved. > > > > (Sorry, I thought we were supposed to reply on the tracker.) > > _______________________________________________ > > Dev mailing list -- dev@ceph.io > > To unsubscribe send an email to dev-leave@ceph.io > > > > > > -- > > Laura Flores > > She/Her/Hers > > Software Engineer, Ceph Storage <https://ceph.io> > > Chicago, IL > > lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
On Tue, Aug 6, 2024 at 6:33 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-... is a problem with the cfuse_workunit_kernel_untar_build task where it's failing to build the kernel, so a problem with the task itself I believe at this point. https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-... is a problem with the cfuse_workunit_suites_ffsb task where it's reporting 2024-08-03T06:51:35.402 INFO:tasks.workunit.client.0.smithi089.stdout:Probably out of disk space I'll chase these down, but I don't think they are powercycle issues per se at this stage. I will prioritise identifying the specific root cause however. APPROVED.
crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Brad
On Thu, Aug 15, 2024 at 11:50 AM Brad Hubbard <bhubbard@redhat.com> wrote:
On Tue, Aug 6, 2024 at 6:33 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-... is a problem with the cfuse_workunit_kernel_untar_build task where it's failing to build the kernel, so a problem with the task itself I believe at this point.
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-... is a problem with the cfuse_workunit_suites_ffsb task where it's reporting 2024-08-03T06:51:35.402 INFO:tasks.workunit.client.0.smithi089.stdout:Probably out of disk space
I'm pretty sure the first of these, and possibly the second as well, are ceph-fuse issues and I've created https://tracker.ceph.com/issues/67565 and asked for input from the FS team.
I'll chase these down, but I don't think they are powercycle issues per se at this stage. I will prioritise identifying the specific root cause however.
APPROVED.
crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Brad
-- Cheers, Brad
Hi Brad, On Fri, Aug 16, 2024 at 8:59 AM Brad Hubbard <bhubbard@redhat.com> wrote:
On Thu, Aug 15, 2024 at 11:50 AM Brad Hubbard <bhubbard@redhat.com> wrote:
On Tue, Aug 6, 2024 at 6:33 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-... is a problem with the cfuse_workunit_kernel_untar_build task where it's failing to build the kernel, so a problem with the task itself I believe at this point.
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-... is a problem with the cfuse_workunit_suites_ffsb task where it's reporting 2024-08-03T06:51:35.402 INFO:tasks.workunit.client.0.smithi089.stdout:Probably out of disk space
I'm pretty sure the first of these, and possibly the second as well, are ceph-fuse issues and I've created https://tracker.ceph.com/issues/67565 and asked for input from the FS team.
I have pushed a fix for the crash. The bug exists in 19.1.0 RC too -- do we consider it as a blocker for this RC (Neha/Patrick)?
I'll chase these down, but I don't think they are powercycle issues per se at this stage. I will prioritise identifying the specific root cause however.
APPROVED.
crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Brad
-- Cheers, Brad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
On Mon, Aug 19, 2024 at 4:46 PM Venky Shankar <vshankar@redhat.com> wrote:
Hi Brad,
On Fri, Aug 16, 2024 at 8:59 AM Brad Hubbard <bhubbard@redhat.com> wrote:
On Thu, Aug 15, 2024 at 11:50 AM Brad Hubbard <bhubbard@redhat.com> wrote:
On Tue, Aug 6, 2024 at 6:33 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-... is a problem with the cfuse_workunit_kernel_untar_build task where it's failing to build the kernel, so a problem with the task itself I believe at this point.
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-... is a problem with the cfuse_workunit_suites_ffsb task where it's reporting 2024-08-03T06:51:35.402 INFO:tasks.workunit.client.0.smithi089.stdout:Probably out of disk space
I'm pretty sure the first of these, and possibly the second as well, are ceph-fuse issues and I've created https://tracker.ceph.com/issues/67565 and asked for input from the FS team.
I have pushed a fix for the crash. The bug exists in 19.1.0 RC too -- do we consider it as a blocker for this RC (Neha/Patrick)?
It was decided in a conversation b/w me, Greg and Laura that this should not block the RC.
I'll chase these down, but I don't think they are powercycle issues per se at this stage. I will prioritise identifying the specific root cause however.
APPROVED.
crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Brad
-- Cheers, Brad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
-- Cheers, Venky
The gibba upgrade is complete! All daemons upgraded successfully. On Tue, Aug 20, 2024 at 10:44 AM Venky Shankar <vshankar@redhat.com> wrote:
On Mon, Aug 19, 2024 at 4:46 PM Venky Shankar <vshankar@redhat.com> wrote:
Hi Brad,
On Fri, Aug 16, 2024 at 8:59 AM Brad Hubbard <bhubbard@redhat.com>
wrote:
On Thu, Aug 15, 2024 at 11:50 AM Brad Hubbard <bhubbard@redhat.com>
wrote:
On Tue, Aug 6, 2024 at 6:33 AM Yuri Weinstein <yweinste@redhat.com>
wrote:
Details of this release are summarized here:
https://tracker.ceph.com/issues/67340#note-1
Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD
Seeking approvals/reviews for:
rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is
being
tested and will be cherry-picked when ready)
rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya
quincy-x, reef-x - Laura, Neha
powercycle - Brad
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-...
is a problem with the cfuse_workunit_kernel_untar_build task where it's failing to build the kernel, so a problem with the task itself I believe at this point.
https://pulpito.ceph.com/yuriw-2024-08-02_15:42:13-powercycle-squid-release-...
is a problem with the cfuse_workunit_suites_ffsb task where it's reporting 2024-08-03T06:51:35.402 INFO:tasks.workunit.client.0.smithi089.stdout:Probably out of disk space
I'm pretty sure the first of these, and possibly the second as well, are ceph-fuse issues and I've created https://tracker.ceph.com/issues/67565 and asked for input from the FS team.
I have pushed a fix for the crash. The bug exists in 19.1.0 RC too -- do we consider it as a blocker for this RC (Neha/Patrick)?
It was decided in a conversation b/w me, Greg and Laura that this should not block the RC.
I'll chase these down, but I don't think they are powercycle issues per se at this stage. I will prioritise identifying the specific root cause however.
APPROVED.
crimson-rados - Matan, Samuel
ceph-volume - Guillaume
Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Brad
-- Cheers, Brad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
-- Cheers, Venky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Hi Yuri, ceph-volume approved https://jenkins.ceph.com/job/ceph-volume-test/601/ Regards, -- Guillaume Abrioux Software Engineer From: Yuri Weinstein <yweinste@redhat.com> Date: Monday, 5 August 2024 at 22:33 To: dev <dev@ceph.io>, ceph-users <ceph-users@ceph.io> Subject: [EXTERNAL] [ceph-users] squid 19.1.1 RC QE validation status Details of this release are summarized here: https://tracker.ceph.com/issues/67340#note-1 Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD Seeking approvals/reviews for: rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready) rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume Pls let me know if any tests were missed from this list. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z
participants (9)
-
Adam Emerson
-
Adam King
-
Brad Hubbard
-
Guillaume ABRIOUX
-
Ilya Dryomov
-
Laura Flores
-
Matan Breizman
-
Venky Shankar
-
Yuri Weinstein