On Tue, Sep 24, 2019 at 10:32 AM Casey Bodley <cbodley(a)redhat.com> wrote: I believe the slow op warnings are one of the things that the monitor sources from the manager as one of the collating services it performs. -Greg > > On 9/23/19 3:21 AM, Dan van der Ster wrote: > > Since mimic, OSD slow ops have not been displayed by the cluster > > health [1] -- this was fixed recently: > > > > https://github.com/ceph/ceph/pull/30285/commits/02cc60f6935a5005aa461da183c… > > > > -- Dan > > > > [1] original report: https://tracker.ceph.com/issues/40993 > > > > On Mon, Sep 23, 2019 at 8:48 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote: > >> https://tracker.ceph.com/issues/41834 > >> > >> This is broadly affecting Ceph QA. Hoping this mail will get the > >> notice of the person whose changes maybe broke it. > >> > >> -- > >> Patrick Donnelly, Ph.D. > >> He / Him / His > >> Senior Software Engineer > >> Red Hat Sunnyvale, CA > >> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > >> _______________________________________________ > >> Dev mailing list -- dev(a)ceph.io > >> To unsubscribe send an email to dev-leave(a)ceph.io > > _______________________________________________ > > Dev mailing list -- dev(a)ceph.io > > To unsubscribe send an email to dev-leave(a)ceph.io > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

On Fri, Sep 27, 2019 at 8:50 AM Sridhar Seshasayee <sseshasa(a)redhat.com> wrote: teuthology-suite -p 49 -m smithi --ceph master --ceph-repo https://github.com/ceph/ceph --suite-repo https://github.com/ceph/ceph -S 79847358cf39f1d68c1efdde2eaa215ffb0496f2 --filter '{begin.yaml ceph-thrash/mon.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{centos_7.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}' --suite fs --num 10 http://pulpito.ceph.com/pdonnell-2019-10-01_03:28:40-fs-master-distro-basic… That was with 10 runs of the same job so it's not trivially reproduced. Might not be related then. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

rather than disable testing entirely, we merged https://github.com/ceph/ceph/pull/30944 to whitelist the SLOW_OPS warnings On 10/15/19 8:20 AM, Casey Bodley wrote: > a friendly ping about this https://tracker.ceph.com/issues/41834. > we're considering disabling tests on ec pools in the rgw suite if this > isn't resolved > > On 10/1/19 12:17 AM, Patrick Donnelly wrote: >> On Fri, Sep 27, 2019 at 8:50 AM Sridhar Seshasayee >> <sseshasa(a)redhat.com> wrote: >>> This PR https://github.com/ceph/ceph/pull/30285 introduced the slow >>> ops fix recently. >>> >>> I haven't been able to reproduce the issue that you hit so far. Can >>> you please let me >>> know the "--subset" you used for your run >>> (pdonnell-2019-09-14_22:39:31-fs-master-distro-basic-smithi) >>> on the fs suite? >> teuthology-suite -p 49 -m smithi --ceph master --ceph-repo >> https://github.com/ceph/ceph --suite-repo https://github.com/ceph/ceph >> -S 79847358cf39f1d68c1efdde2eaa215ffb0496f2 --filter '{begin.yaml >> ceph-thrash/mon.yaml clusters/1-mds-1-client-coloc.yaml >> conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml >> msgr-failures/osd-mds-delay.yaml >> objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml >> whitelist_health.yaml whitelist_wrongly_marked_down.yaml} >> supported-random-distros$/{centos_7.yaml} >> tasks/cfuse_workunit_suites_fsstress.yaml}' --suite fs --num 10 >> >> http://pulpito.ceph.com/pdonnell-2019-10-01_03:28:40-fs-master-distro-basic… >> >> >> That was with 10 runs of the same job so it's not trivially reproduced. >> >>> Just to clarify, before the fix, even if there were legitimate slow >>> ops on 1 or more osds, >>> they would not get reported as part of the "ceph -s" output. The >>> above fix just addressed the issue >>> of the slow ops not getting shown as part of "ceph -s" command if >>> were any reported. >> Might not be related then. >> >> >> -- >> Patrick Donnelly, Ph.D. >> He / Him / His >> Senior Software Engineer >> Red Hat Sunnyvale, CA >> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D