Patrick,
This PR
https://github.com/ceph/ceph/pull/30285 introduced the slow ops fix
recently.
I haven't been able to reproduce the issue that you hit so far. Can you
please let me
know the "--subset" you used for your run
(pdonnell-2019-09-14_22:39:31-fs-master-distro-basic-smithi)
on the fs suite?
Just to clarify, before the fix, even if there were legitimate slow ops on
1 or more osds,
they would not get reported as part of the "ceph -s" output. The above fix
just addressed the issue
of the slow ops not getting shown as part of "ceph -s" command if were any
reported.
Since the tests actually parse the cluster logs for slow ops, it doesn't
seem that the above fix
introduced this issue. The thing that needs investigation is the slow ops
being reported with a
timestamp of 0.000000. Therefore, it would help if you can share the
"--subset" option that
you used for your run.
Thanks,
-Sridhar
On Thu, Sep 26, 2019 at 3:08 AM Gregory Farnum <gfarnum(a)redhat.com> wrote:
On Tue, Sep 24, 2019 at 10:32 AM Casey Bodley
<cbodley(a)redhat.com> wrote:
The tests are detecting these failures by grepping the cluster log. It
looks like ceph-mon is responsible for writing these 'Health check
failed' warnings there, so I'm not sure that ceph-mgr bug is involved
here - but maybe someone more familiar with ceph-mgr could say for sure?
I believe the slow op warnings are one of the things that the monitor
sources from the manager as one of the collating services it performs.
-Greg
On 9/23/19 3:21 AM, Dan van der Ster wrote:
> Since mimic, OSD slow ops have not been displayed by the cluster
> health [1] -- this was fixed recently:
>
>
https://github.com/ceph/ceph/pull/30285/commits/02cc60f6935a5005aa461da183c…
>
> -- Dan
>
> [1] original report:
https://tracker.ceph.com/issues/40993
>
> On Mon, Sep 23, 2019 at 8:48 AM Patrick Donnelly <pdonnell(a)redhat.com>
wrote:
https://tracker.ceph.com/issues/41834
This is broadly affecting Ceph QA. Hoping this mail will get the
notice of the person whose changes maybe broke it.
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
--
Sridhar Seshasayee
Principal Software Engineer
Red Hat <https://www.redhat.com>
<https://www.redhat.com>