On Mon, Jun 29, 2020 at 11:58 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Mon, Jun 29, 2020 at 11:38 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste@redhat.com> wrote:
fs - approved Patrick?
There's a new upgrade test failure: https://pulpito.ceph.com/yuriw-2020-06-25_22:31:00-fs-octopus-distro-basic-s...
2020-06-26T07:13:13.444 INFO:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json ... 2020-06-26T07:13:13.709 INFO:teuthology.orchestra.run.smithi087.stderr:Error EINVAL: invalid command
I vaguely recall seeing this before but can't find the tracker ticket. The last successful run was on the 24th (the day before you scheduled tests!!):
https://pulpito.ceph.com/teuthology-2020-06-24_04:10:03-fs-octopus-distro-ba...
The difference in commits 5c9eee5a7ad1bb8b161f33b4560d5dfc4a000b6d..upstream/heads/octopus does not shine light on why this would happen.
Any ideas?
Looks like this is caused by an older version of Mimic, v13.2.8. The successful run from June 24th:
2020-06-25T06:10:56.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=mimic 2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/mimic/8e9675a830b7e7a7164659a2a8f8ad08b8f61... 2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Package version is 13.2.10-15-g8e9675a830-1bionic
vs. the failure on June 25th:
2020-06-26T07:11:31.659 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=mimic 2020-06-26T07:11:31.820 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/mimic/5579a94fafbc1f9cc913a0f5d362953a5d9c3ae... 2020-06-26T07:11:31.821 INFO:teuthology.task.install.deb:Package version is 13.2.8-1bionic
This is probably caused by the LRC data loss. Looks like we need to get new builds of Mimic.
yes, the bug we are hitting was fixed by https://github.com/ceph/ceph/pull/33130.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D