29 Jun
2020
29 Jun
'20
7:34 p.m.
Details of this release summarized here:
https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED
rados - approved Neha?
rgw - approved Casey?
rbd - approved Jason?
krbd - approved Jason, Ilya?
fs - approved Patrick?
kcephfs - approved Patrick?
multimds - approved Patrick?
ceph-deploy - N/A
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it?
upgrade/nautilus-x (octopus) - PASSED
upgrade/octopus-p2p - FAILED, Josh, Jason approved?
upgrade/client-upgrade-luminous-octopus - PASSED
upgrade/client-upgrade-mimic-octopus - PASSED
upgrade/client-upgrade-nautilus-octopus - PASSED
powercycle - approved Neha?
ceph-ansible - approved Brad?
ceph-volume - approved Jan? (ceph-ansible bug? number?)
(please speak up if something is missing)
David, Josh - is it to upgrade time sepia ?.
Thx
YuriW
29 Jun
29 Jun
8:38 p.m.
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste(a)redhat.com> wrote:
fs - approved Patrick?
There's a new upgrade test failure:
https://pulpito.ceph.com/yuriw-2020-06-25_22:31:00-fs-octopus-distro-basic-…
2020-06-26T07:13:13.444 INFO:teuthology.orchestra.run.smithi087:> sudo
adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
timeout 120 ceph --cluster ceph --log-early osd dump --format=json
...
2020-06-26T07:13:13.709
INFO:teuthology.orchestra.run.smithi087.stderr:Error EINVAL: invalid
command
I vaguely recall seeing this before but can't find the tracker ticket.
The last successful run was on the 24th (the day before you scheduled
tests!!):
https://pulpito.ceph.com/teuthology-2020-06-24_04:10:03-fs-octopus-distro-b…
The difference in commits
5c9eee5a7ad1bb8b161f33b4560d5dfc4a000b6d..upstream/heads/octopus does
not shine light on why this would happen.
Any ideas?
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
8:58 p.m.
On Mon, Jun 29, 2020 at 11:38 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste(a)redhat.com> wrote:
Looks like this is caused by an older version of Mimic, v13.2.8. The
successful run from June 24th:
2020-06-25T06:10:56.756 DEBUG:teuthology.packaging:Querying
https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor…
2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Pulling from
https://4.chacra.ceph.com/r/ceph/mimic/8e9675a830b7e7a7164659a2a8f8ad08b8f6…
2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Package
version is 13.2.10-15-g8e9675a830-1bionic
vs. the failure on June 25th:
2020-06-26T07:11:31.659 DEBUG:teuthology.packaging:Querying
https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor…
2020-06-26T07:11:31.820 INFO:teuthology.task.install.deb:Pulling from
https://chacra.ceph.com/r/ceph/mimic/5579a94fafbc1f9cc913a0f5d362953a5d9c3a…
2020-06-26T07:11:31.821 INFO:teuthology.task.install.deb:Package
version is 13.2.8-1bionic
This is probably caused by the LRC data loss. Looks like we need to
get new builds of Mimic.
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
fs - approved Patrick?
There's a new upgrade test failure:
https://pulpito.ceph.com/yuriw-2020-06-25_22:31:00-fs-octopus-distro-basic-…
2020-06-26T07:13:13.444 INFO:teuthology.orchestra.run.smithi087:> sudo
adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
timeout 120 ceph --cluster ceph --log-early osd dump --format=json
...
2020-06-26T07:13:13.709
INFO:teuthology.orchestra.run.smithi087.stderr:Error EINVAL: invalid
command
I vaguely recall seeing this before but can't find the tracker ticket.
The last successful run was on the 24th (the day before you scheduled
tests!!):
https://pulpito.ceph.com/teuthology-2020-06-24_04:10:03-fs-octopus-distro-b…
The difference in commits
5c9eee5a7ad1bb8b161f33b4560d5dfc4a000b6d..upstream/heads/octopus does
not shine light on why this would happen.
Any ideas? 
9:01 p.m.
On Mon, Jun 29, 2020 at 11:58 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
On Mon, Jun 29, 2020 at 11:38 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
yes, the bug we are hitting was fixed by
https://github.com/ceph/ceph/pull/33130.
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste(a)redhat.com> wrote:
Looks like this is caused by an older version of Mimic, v13.2.8. The
successful run from June 24th:
2020-06-25T06:10:56.756 DEBUG:teuthology.packaging:Querying
https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor…
2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Pulling from
https://4.chacra.ceph.com/r/ceph/mimic/8e9675a830b7e7a7164659a2a8f8ad08b8f6…
2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Package
version is 13.2.10-15-g8e9675a830-1bionic
vs. the failure on June 25th:
2020-06-26T07:11:31.659 DEBUG:teuthology.packaging:Querying
https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor…
2020-06-26T07:11:31.820 INFO:teuthology.task.install.deb:Pulling from
https://chacra.ceph.com/r/ceph/mimic/5579a94fafbc1f9cc913a0f5d362953a5d9c3a…
2020-06-26T07:11:31.821 INFO:teuthology.task.install.deb:Package
version is 13.2.8-1bionic
This is probably caused by the LRC data loss. Looks like we need to
get new builds of Mimic. fs - approved Patrick?
There's a new upgrade test failure:
https://pulpito.ceph.com/yuriw-2020-06-25_22:31:00-fs-octopus-distro-basic-…
2020-06-26T07:13:13.444 INFO:teuthology.orchestra.run.smithi087:> sudo
adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
timeout 120 ceph --cluster ceph --log-early osd dump --format=json
...
2020-06-26T07:13:13.709
INFO:teuthology.orchestra.run.smithi087.stderr:Error EINVAL: invalid
command
I vaguely recall seeing this before but can't find the tracker ticket.
The last successful run was on the 24th (the day before you scheduled
tests!!):
https://pulpito.ceph.com/teuthology-2020-06-24_04:10:03-fs-octopus-distro-b…
The difference in commits
5c9eee5a7ad1bb8b161f33b4560d5dfc4a000b6d..upstream/heads/octopus does
not shine light on why this would happen.
Any ideas?
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
9:03 p.m.
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste(a)redhat.com> wrote:
fs - approved Patrick?
kcephfs - approved Patrick?
multimds - approved Patrick?
Approved!
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
30 Jun
30 Jun
12:41 a.m.
New subject: [Ceph-qa] 15.2.4 QE Octopus validation status (corrected title)
On 6/29/20 10:34 AM, Yuri Weinstein wrote:
Details of this release summarized here:
https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it?
Still needed, your re-run with the rebuild of mimic is looking good so far.
upgrade/octopus-p2p - FAILED, Josh, Jason approved?
Looks good to me.
Josh
4:30 a.m.
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste(a)redhat.com> wrote:
Details of this release summarized here:
https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED
rados - approved Neha?
approved
rgw - approved Casey?
rbd - approved Jason?
krbd - approved Jason, Ilya?
fs - approved Patrick?
kcephfs - approved Patrick?
multimds - approved Patrick?
ceph-deploy - N/A
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it?
upgrade/nautilus-x (octopus) - PASSED
upgrade/octopus-p2p - FAILED, Josh, Jason approved?
upgrade/client-upgrade-luminous-octopus - PASSED
upgrade/client-upgrade-mimic-octopus - PASSED
upgrade/client-upgrade-nautilus-octopus - PASSED
powercycle - approved Neha?
approved
Thanks,
Neha
ceph-ansible - approved Brad?
ceph-volume - approved Jan? (ceph-ansible bug? number?)
(please speak up if something is missing)
David, Josh - is it to upgrade time sepia ?.
Thx
YuriW
9:12 a.m.
On Mon, Jun 29, 2020 at 10:34:43AM -0700, Yuri Weinstein wrote:
Details of this release summarized here:
https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED
rados - approved Neha?
rgw - approved Casey?
rbd - approved Jason?
krbd - approved Jason, Ilya?
fs - approved Patrick?
kcephfs - approved Patrick?
multimds - approved Patrick?
ceph-deploy - N/A
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it?
upgrade/nautilus-x (octopus) - PASSED
upgrade/octopus-p2p - FAILED, Josh, Jason approved?
upgrade/client-upgrade-luminous-octopus - PASSED
upgrade/client-upgrade-mimic-octopus - PASSED
upgrade/client-upgrade-nautilus-octopus - PASSED
powercycle - approved Neha?
ceph-ansible - approved Brad?
ceph-volume - approved Jan? (ceph-ansible bug? number?)
Approved! Test fix and
accompanying ansible fix are being backported.
https://tracker.ceph.com/issues/46131
https://github.com/ceph/ceph-ansible/pull/5413
(please speak up if something is missing)
David, Josh - is it to upgrade time sepia ?.
Thx
YuriW
--
Jan Fajerski
Senior Software Engineer Enterprise Storage
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
10:24 a.m.
New subject: [Ceph-qa] Re: 15.2.4 QE Octopus validation status (corrected title)
Am 30.06.20 um 04:30 schrieb Neha Ojha:
rados/cephadm
--
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein
<yweinste(a)redhat.com> wrote:
rados/cephadm looks good and the cephadm failures in trash-old-clients
are pending backport. approved.
Details of this release summarized here:
https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED
rados - approved Neha?
approved rgw - approved Casey?
rbd - approved Jason?
krbd - approved Jason, Ilya?
fs - approved Patrick?
kcephfs - approved Patrick?
multimds - approved Patrick?
ceph-deploy - N/A
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it?
upgrade/nautilus-x (octopus) - PASSED
upgrade/octopus-p2p - FAILED, Josh, Jason approved?
upgrade/client-upgrade-luminous-octopus - PASSED
upgrade/client-upgrade-mimic-octopus - PASSED
upgrade/client-upgrade-nautilus-octopus - PASSED
powercycle - approved Neha?
approved
Thanks,
Neha
> ceph-ansible - approved Brad?
> ceph-volume - approved Jan? (ceph-ansible bug? number?)
>
> (please speak up if something is missing) >
> David, Josh - is it to upgrade time sepia ?.
+1
Thx
YuriW
_______________________________________________
Ceph-qa mailing list -- ceph-qa(a)ceph.io
To unsubscribe send an email to ceph-qa-leave(a)ceph.io
10:57 a.m.
On Mon, Jun 29, 2020 at 7:34 PM Yuri Weinstein <yweinste(a)redhat.com> wrote:
Details of this release summarized here:
https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED
rados - approved Neha?
rgw - approved Casey?
rbd - approved Jason?
krbd - approved Jason, Ilya?
Approved.
Thanks,
Ilya
5:05 p.m.
Josh, 15.2.4 is ready for final approval
mimic-x - seems to be one dead job passed in the other run today, suspect infra
ceph-ansible ?
On Tue, Jun 30, 2020 at 1:57 AM Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Mon, Jun 29, 2020 at 7:34 PM Yuri Weinstein <yweinste(a)redhat.com> wrote:
Details of this release summarized here:
https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED
rados - approved Neha?
rgw - approved Casey?
rbd - approved Jason?
krbd - approved Jason, Ilya?
Approved.
Thanks,
Ilya
5:17 p.m.
New subject: [Ceph-qa] Re: 15.2.4 QE Octopus validation status (corrected title)
On 6/30/20 8:05 AM, Yuri Weinstein wrote:
Josh, 15.2.4 is ready for final approval
Approved!
mimic-x - seems to be one dead job passed in the other
run today, suspect infra
ceph-ansible ?
On Tue, Jun 30, 2020 at 1:57 AM Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Mon, Jun 29, 2020 at 7:34 PM Yuri Weinstein <yweinste(a)redhat.com> wrote:
_______________________________________________
Ceph-qa mailing list -- ceph-qa(a)ceph.io
To unsubscribe send an email to ceph-qa-leave(a)ceph.io
Details of this release summarized here:
https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED
rados - approved Neha?
rgw - approved Casey?
rbd - approved Jason?
krbd - approved Jason, Ilya?
Approved.
Thanks,
Ilya
1390
days inactive
1391
days old
11 comments
7 participants
participants (7)
-
Ilya Dryomov -
Jan Fajerski -
Josh Durgin -
Neha Ojha -
Patrick Donnelly -
Sebastian Wagner -
Yuri Weinstein