15.2.4 QE Octopus validation status (corrected title)
Details of this release summarized here: https://tracker.ceph.com/issues/46159#note-2 All failed suites need to have failed jobs reran. smoke - PASSED rados - approved Neha? rgw - approved Casey? rbd - approved Jason? krbd - approved Jason, Ilya? fs - approved Patrick? kcephfs - approved Patrick? multimds - approved Patrick? ceph-deploy - N/A upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it? upgrade/nautilus-x (octopus) - PASSED upgrade/octopus-p2p - FAILED, Josh, Jason approved? upgrade/client-upgrade-luminous-octopus - PASSED upgrade/client-upgrade-mimic-octopus - PASSED upgrade/client-upgrade-nautilus-octopus - PASSED powercycle - approved Neha? ceph-ansible - approved Brad? ceph-volume - approved Jan? (ceph-ansible bug? number?) (please speak up if something is missing) David, Josh - is it to upgrade time sepia ?. Thx YuriW
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste@redhat.com> wrote:
fs - approved Patrick?
There's a new upgrade test failure: https://pulpito.ceph.com/yuriw-2020-06-25_22:31:00-fs-octopus-distro-basic-s... 2020-06-26T07:13:13.444 INFO:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json ... 2020-06-26T07:13:13.709 INFO:teuthology.orchestra.run.smithi087.stderr:Error EINVAL: invalid command I vaguely recall seeing this before but can't find the tracker ticket. The last successful run was on the 24th (the day before you scheduled tests!!): https://pulpito.ceph.com/teuthology-2020-06-24_04:10:03-fs-octopus-distro-ba... The difference in commits 5c9eee5a7ad1bb8b161f33b4560d5dfc4a000b6d..upstream/heads/octopus does not shine light on why this would happen. Any ideas? -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Mon, Jun 29, 2020 at 11:38 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste@redhat.com> wrote:
fs - approved Patrick?
There's a new upgrade test failure: https://pulpito.ceph.com/yuriw-2020-06-25_22:31:00-fs-octopus-distro-basic-s...
2020-06-26T07:13:13.444 INFO:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json ... 2020-06-26T07:13:13.709 INFO:teuthology.orchestra.run.smithi087.stderr:Error EINVAL: invalid command
I vaguely recall seeing this before but can't find the tracker ticket. The last successful run was on the 24th (the day before you scheduled tests!!):
https://pulpito.ceph.com/teuthology-2020-06-24_04:10:03-fs-octopus-distro-ba...
The difference in commits 5c9eee5a7ad1bb8b161f33b4560d5dfc4a000b6d..upstream/heads/octopus does not shine light on why this would happen.
Any ideas?
Looks like this is caused by an older version of Mimic, v13.2.8. The successful run from June 24th: 2020-06-25T06:10:56.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=mimic 2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/mimic/8e9675a830b7e7a7164659a2a8f8ad08b8f61... 2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Package version is 13.2.10-15-g8e9675a830-1bionic vs. the failure on June 25th: 2020-06-26T07:11:31.659 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=mimic 2020-06-26T07:11:31.820 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/mimic/5579a94fafbc1f9cc913a0f5d362953a5d9c3ae... 2020-06-26T07:11:31.821 INFO:teuthology.task.install.deb:Package version is 13.2.8-1bionic This is probably caused by the LRC data loss. Looks like we need to get new builds of Mimic. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Mon, Jun 29, 2020 at 11:58 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Mon, Jun 29, 2020 at 11:38 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste@redhat.com> wrote:
fs - approved Patrick?
There's a new upgrade test failure: https://pulpito.ceph.com/yuriw-2020-06-25_22:31:00-fs-octopus-distro-basic-s...
2020-06-26T07:13:13.444 INFO:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json ... 2020-06-26T07:13:13.709 INFO:teuthology.orchestra.run.smithi087.stderr:Error EINVAL: invalid command
I vaguely recall seeing this before but can't find the tracker ticket. The last successful run was on the 24th (the day before you scheduled tests!!):
https://pulpito.ceph.com/teuthology-2020-06-24_04:10:03-fs-octopus-distro-ba...
The difference in commits 5c9eee5a7ad1bb8b161f33b4560d5dfc4a000b6d..upstream/heads/octopus does not shine light on why this would happen.
Any ideas?
Looks like this is caused by an older version of Mimic, v13.2.8. The successful run from June 24th:
2020-06-25T06:10:56.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=mimic 2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/mimic/8e9675a830b7e7a7164659a2a8f8ad08b8f61... 2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Package version is 13.2.10-15-g8e9675a830-1bionic
vs. the failure on June 25th:
2020-06-26T07:11:31.659 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=mimic 2020-06-26T07:11:31.820 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/mimic/5579a94fafbc1f9cc913a0f5d362953a5d9c3ae... 2020-06-26T07:11:31.821 INFO:teuthology.task.install.deb:Package version is 13.2.8-1bionic
This is probably caused by the LRC data loss. Looks like we need to get new builds of Mimic.
yes, the bug we are hitting was fixed by https://github.com/ceph/ceph/pull/33130.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste@redhat.com> wrote:
fs - approved Patrick? kcephfs - approved Patrick? multimds - approved Patrick?
Approved! -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 6/29/20 10:34 AM, Yuri Weinstein wrote:
Details of this release summarized here: https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it?
Still needed, your re-run with the rebuild of mimic is looking good so far.
upgrade/octopus-p2p - FAILED, Josh, Jason approved?
Looks good to me. Josh
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release summarized here: https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED rados - approved Neha?
approved
rgw - approved Casey? rbd - approved Jason? krbd - approved Jason, Ilya? fs - approved Patrick? kcephfs - approved Patrick? multimds - approved Patrick? ceph-deploy - N/A
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it? upgrade/nautilus-x (octopus) - PASSED upgrade/octopus-p2p - FAILED, Josh, Jason approved? upgrade/client-upgrade-luminous-octopus - PASSED upgrade/client-upgrade-mimic-octopus - PASSED upgrade/client-upgrade-nautilus-octopus - PASSED powercycle - approved Neha?
approved Thanks, Neha
ceph-ansible - approved Brad? ceph-volume - approved Jan? (ceph-ansible bug? number?)
(please speak up if something is missing)
David, Josh - is it to upgrade time sepia ?.
Thx YuriW
Am 30.06.20 um 04:30 schrieb Neha Ojha:
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release summarized here: https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED rados - approved Neha?
approved
rados/cephadm looks good and the cephadm failures in trash-old-clients are pending backport. approved.
rgw - approved Casey? rbd - approved Jason? krbd - approved Jason, Ilya? fs - approved Patrick? kcephfs - approved Patrick? multimds - approved Patrick? ceph-deploy - N/A
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it? upgrade/nautilus-x (octopus) - PASSED upgrade/octopus-p2p - FAILED, Josh, Jason approved? upgrade/client-upgrade-luminous-octopus - PASSED upgrade/client-upgrade-mimic-octopus - PASSED upgrade/client-upgrade-nautilus-octopus - PASSED powercycle - approved Neha?
approved
Thanks, Neha
ceph-ansible - approved Brad? ceph-volume - approved Jan? (ceph-ansible bug? number?)
(please speak up if something is missing)
rados/cephadm
David, Josh - is it to upgrade time sepia ?.
+1
Thx YuriW
_______________________________________________ Ceph-qa mailing list -- ceph-qa@ceph.io To unsubscribe send an email to ceph-qa-leave@ceph.io
-- SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer
On Mon, Jun 29, 2020 at 10:34:43AM -0700, Yuri Weinstein wrote:
Details of this release summarized here: https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED rados - approved Neha? rgw - approved Casey? rbd - approved Jason? krbd - approved Jason, Ilya? fs - approved Patrick? kcephfs - approved Patrick? multimds - approved Patrick? ceph-deploy - N/A
upgrade/mimic-x (octopus) - FAILED, Josh can we deprecate it? upgrade/nautilus-x (octopus) - PASSED upgrade/octopus-p2p - FAILED, Josh, Jason approved? upgrade/client-upgrade-luminous-octopus - PASSED upgrade/client-upgrade-mimic-octopus - PASSED upgrade/client-upgrade-nautilus-octopus - PASSED powercycle - approved Neha? ceph-ansible - approved Brad? ceph-volume - approved Jan? (ceph-ansible bug? number?) Approved! Test fix and accompanying ansible fix are being backported. https://tracker.ceph.com/issues/46131 https://github.com/ceph/ceph-ansible/pull/5413
(please speak up if something is missing)
David, Josh - is it to upgrade time sepia ?.
Thx YuriW
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
On Mon, Jun 29, 2020 at 7:34 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release summarized here: https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED rados - approved Neha? rgw - approved Casey? rbd - approved Jason? krbd - approved Jason, Ilya?
Approved. Thanks, Ilya
Josh, 15.2.4 is ready for final approval mimic-x - seems to be one dead job passed in the other run today, suspect infra ceph-ansible ? On Tue, Jun 30, 2020 at 1:57 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Mon, Jun 29, 2020 at 7:34 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release summarized here: https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED rados - approved Neha? rgw - approved Casey? rbd - approved Jason? krbd - approved Jason, Ilya?
Approved.
Thanks,
Ilya
On 6/30/20 8:05 AM, Yuri Weinstein wrote:
Josh, 15.2.4 is ready for final approval
Approved!
mimic-x - seems to be one dead job passed in the other run today, suspect infra ceph-ansible ?
On Tue, Jun 30, 2020 at 1:57 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Mon, Jun 29, 2020 at 7:34 PM Yuri Weinstein <yweinste@redhat.com> wrote:
Details of this release summarized here: https://tracker.ceph.com/issues/46159#note-2
All failed suites need to have failed jobs reran.
smoke - PASSED rados - approved Neha? rgw - approved Casey? rbd - approved Jason? krbd - approved Jason, Ilya?
Approved.
Thanks,
Ilya
_______________________________________________ Ceph-qa mailing list -- ceph-qa@ceph.io To unsubscribe send an email to ceph-qa-leave@ceph.io
participants (7)
-
Ilya Dryomov
-
Jan Fajerski
-
Josh Durgin
-
Neha Ojha
-
Patrick Donnelly
-
Sebastian Wagner
-
Yuri Weinstein