slow op warnings from OSDs due to recent master changes
https://tracker.ceph.com/issues/41834 This is broadly affecting Ceph QA. Hoping this mail will get the notice of the person whose changes maybe broke it. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Since mimic, OSD slow ops have not been displayed by the cluster health [1] -- this was fixed recently: https://github.com/ceph/ceph/pull/30285/commits/02cc60f6935a5005aa461da183c6... -- Dan [1] original report: https://tracker.ceph.com/issues/40993 On Mon, Sep 23, 2019 at 8:48 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
https://tracker.ceph.com/issues/41834
This is broadly affecting Ceph QA. Hoping this mail will get the notice of the person whose changes maybe broke it.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
The tests are detecting these failures by grepping the cluster log. It looks like ceph-mon is responsible for writing these 'Health check failed' warnings there, so I'm not sure that ceph-mgr bug is involved here - but maybe someone more familiar with ceph-mgr could say for sure? On 9/23/19 3:21 AM, Dan van der Ster wrote:
Since mimic, OSD slow ops have not been displayed by the cluster health [1] -- this was fixed recently:
https://github.com/ceph/ceph/pull/30285/commits/02cc60f6935a5005aa461da183c6...
-- Dan
[1] original report: https://tracker.ceph.com/issues/40993
On Mon, Sep 23, 2019 at 8:48 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
https://tracker.ceph.com/issues/41834
This is broadly affecting Ceph QA. Hoping this mail will get the notice of the person whose changes maybe broke it.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Tue, Sep 24, 2019 at 10:32 AM Casey Bodley <cbodley@redhat.com> wrote:
The tests are detecting these failures by grepping the cluster log. It looks like ceph-mon is responsible for writing these 'Health check failed' warnings there, so I'm not sure that ceph-mgr bug is involved here - but maybe someone more familiar with ceph-mgr could say for sure?
I believe the slow op warnings are one of the things that the monitor sources from the manager as one of the collating services it performs. -Greg
On 9/23/19 3:21 AM, Dan van der Ster wrote:
Since mimic, OSD slow ops have not been displayed by the cluster health [1] -- this was fixed recently:
https://github.com/ceph/ceph/pull/30285/commits/02cc60f6935a5005aa461da183c6...
-- Dan
[1] original report: https://tracker.ceph.com/issues/40993
On Mon, Sep 23, 2019 at 8:48 AM Patrick Donnelly <pdonnell@redhat.com> wrote:
https://tracker.ceph.com/issues/41834
This is broadly affecting Ceph QA. Hoping this mail will get the notice of the person whose changes maybe broke it.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Patrick, This PR https://github.com/ceph/ceph/pull/30285 introduced the slow ops fix recently. I haven't been able to reproduce the issue that you hit so far. Can you please let me know the "--subset" you used for your run (pdonnell-2019-09-14_22:39:31-fs-master-distro-basic-smithi) on the fs suite? Just to clarify, before the fix, even if there were legitimate slow ops on 1 or more osds, they would not get reported as part of the "ceph -s" output. The above fix just addressed the issue of the slow ops not getting shown as part of "ceph -s" command if were any reported. Since the tests actually parse the cluster logs for slow ops, it doesn't seem that the above fix introduced this issue. The thing that needs investigation is the slow ops being reported with a timestamp of 0.000000. Therefore, it would help if you can share the "--subset" option that you used for your run. Thanks, -Sridhar On Thu, Sep 26, 2019 at 3:08 AM Gregory Farnum <gfarnum@redhat.com> wrote:
On Tue, Sep 24, 2019 at 10:32 AM Casey Bodley <cbodley@redhat.com> wrote:
The tests are detecting these failures by grepping the cluster log. It looks like ceph-mon is responsible for writing these 'Health check failed' warnings there, so I'm not sure that ceph-mgr bug is involved here - but maybe someone more familiar with ceph-mgr could say for sure?
I believe the slow op warnings are one of the things that the monitor sources from the manager as one of the collating services it performs. -Greg
On 9/23/19 3:21 AM, Dan van der Ster wrote:
Since mimic, OSD slow ops have not been displayed by the cluster health [1] -- this was fixed recently:
https://github.com/ceph/ceph/pull/30285/commits/02cc60f6935a5005aa461da183c6...
-- Dan
[1] original report: https://tracker.ceph.com/issues/40993
On Mon, Sep 23, 2019 at 8:48 AM Patrick Donnelly <pdonnell@redhat.com>
wrote:
https://tracker.ceph.com/issues/41834
This is broadly affecting Ceph QA. Hoping this mail will get the notice of the person whose changes maybe broke it.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Sridhar Seshasayee Principal Software Engineer Red Hat <https://www.redhat.com> <https://www.redhat.com>
On Fri, Sep 27, 2019 at 8:50 AM Sridhar Seshasayee <sseshasa@redhat.com> wrote:
This PR https://github.com/ceph/ceph/pull/30285 introduced the slow ops fix recently.
I haven't been able to reproduce the issue that you hit so far. Can you please let me know the "--subset" you used for your run (pdonnell-2019-09-14_22:39:31-fs-master-distro-basic-smithi) on the fs suite?
teuthology-suite -p 49 -m smithi --ceph master --ceph-repo https://github.com/ceph/ceph --suite-repo https://github.com/ceph/ceph -S 79847358cf39f1d68c1efdde2eaa215ffb0496f2 --filter '{begin.yaml ceph-thrash/mon.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{centos_7.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}' --suite fs --num 10 http://pulpito.ceph.com/pdonnell-2019-10-01_03:28:40-fs-master-distro-basic-... That was with 10 runs of the same job so it's not trivially reproduced.
Just to clarify, before the fix, even if there were legitimate slow ops on 1 or more osds, they would not get reported as part of the "ceph -s" output. The above fix just addressed the issue of the slow ops not getting shown as part of "ceph -s" command if were any reported.
Might not be related then. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
a friendly ping about this https://tracker.ceph.com/issues/41834. we're considering disabling tests on ec pools in the rgw suite if this isn't resolved On 10/1/19 12:17 AM, Patrick Donnelly wrote:
On Fri, Sep 27, 2019 at 8:50 AM Sridhar Seshasayee <sseshasa@redhat.com> wrote:
This PR https://github.com/ceph/ceph/pull/30285 introduced the slow ops fix recently.
I haven't been able to reproduce the issue that you hit so far. Can you please let me know the "--subset" you used for your run (pdonnell-2019-09-14_22:39:31-fs-master-distro-basic-smithi) on the fs suite? teuthology-suite -p 49 -m smithi --ceph master --ceph-repo https://github.com/ceph/ceph --suite-repo https://github.com/ceph/ceph -S 79847358cf39f1d68c1efdde2eaa215ffb0496f2 --filter '{begin.yaml ceph-thrash/mon.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{centos_7.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}' --suite fs --num 10
http://pulpito.ceph.com/pdonnell-2019-10-01_03:28:40-fs-master-distro-basic-...
That was with 10 runs of the same job so it's not trivially reproduced.
Just to clarify, before the fix, even if there were legitimate slow ops on 1 or more osds, they would not get reported as part of the "ceph -s" output. The above fix just addressed the issue of the slow ops not getting shown as part of "ceph -s" command if were any reported. Might not be related then.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
rather than disable testing entirely, we merged https://github.com/ceph/ceph/pull/30944 to whitelist the SLOW_OPS warnings On 10/15/19 8:20 AM, Casey Bodley wrote:
a friendly ping about this https://tracker.ceph.com/issues/41834. we're considering disabling tests on ec pools in the rgw suite if this isn't resolved
On 10/1/19 12:17 AM, Patrick Donnelly wrote:
On Fri, Sep 27, 2019 at 8:50 AM Sridhar Seshasayee <sseshasa@redhat.com> wrote:
This PR https://github.com/ceph/ceph/pull/30285 introduced the slow ops fix recently.
I haven't been able to reproduce the issue that you hit so far. Can you please let me know the "--subset" you used for your run (pdonnell-2019-09-14_22:39:31-fs-master-distro-basic-smithi) on the fs suite? teuthology-suite -p 49 -m smithi --ceph master --ceph-repo https://github.com/ceph/ceph --suite-repo https://github.com/ceph/ceph -S 79847358cf39f1d68c1efdde2eaa215ffb0496f2 --filter '{begin.yaml ceph-thrash/mon.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{centos_7.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}' --suite fs --num 10
http://pulpito.ceph.com/pdonnell-2019-10-01_03:28:40-fs-master-distro-basic-...
That was with 10 runs of the same job so it's not trivially reproduced.
Just to clarify, before the fix, even if there were legitimate slow ops on 1 or more osds, they would not get reported as part of the "ceph -s" output. The above fix just addressed the issue of the slow ops not getting shown as part of "ceph -s" command if were any reported. Might not be related then.
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
participants (5)
-
Casey Bodley
-
Dan van der Ster
-
Gregory Farnum
-
Patrick Donnelly
-
Sridhar Seshasayee