just issue the commands scrub pg deep-scrub 17.1cs this will deep scrub this pg ceph pg repair 17.7ff repairs the pg Hi Everyone, I also have seen this inconsistent with empty when you do list-inconsistent-obj $ sudo ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1 pgs not deep-scrubbed in time OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21] PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811 $ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty { "epoch": 183807, "inconsistents": [] } Usually these are caused by read errors on the disks, but I've checked all osd hosts that are part of this osd and there's no smart or dmesg errors. Rich ------------------------------ <CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com> output! <patrakov(a)gmail.com> inconsistent --format=json-pretty` Ceph _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Paste your `ceph versions` please k Sent from my iPhone > On 27 Jan 2021, at 03:07, Richard Bade <hitrich(a)gmail.com> wrote: > > Ceph v14.2.13 by the way.

Thanks Dan and Anthony your suggestions have pointed me in the right direction. Looking back through the logs at when the first error was detected I found this: ceph-osd: 2021-01-24 01:04:55.905 7f0c17821700 -1 log_channel(cluster) log [ERR] : 17.7ffs0 scrub : stat mismatch, got 112867/112868 objects, 0/0 clones, 112867/112868 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 473372381184/473376575488 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. As Anthony suggested the error is not in the rados objects but actually the stats. I assume that a repair will fix this up? Thanks again everyone. Rich On Thu, 28 Jan 2021 at 03:59, Dan van der Ster <dan(a)vanderster.com> wrote: > > Usually the ceph.log prints the reason for the inconsistency when it > is first detected by scrubbing. > > -- dan > > On Wed, Jan 27, 2021 at 12:41 AM Richard Bade <hitrich(a)gmail.com> wrote: > > > > Hi Everyone, > > I also have seen this inconsistent with empty when you do list-inconsistent-obj > > > > $ sudo ceph health detail > > HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1 > > pgs not deep-scrubbed in time > > OSD_SCRUB_ERRORS 1 scrub errors > > PG_DAMAGED Possible data damage: 1 pg inconsistent > > pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21] > > PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time > > pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811 > > > > $ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty > > { > > "epoch": 183807, > > "inconsistents": [] > > } > > > > Usually these are caused by read errors on the disks, but I've checked > > all osd hosts that are part of this osd and there's no smart or dmesg > > errors. > > > > Rich > > > > ------------------------------ > > > > > > Date: Sun, 17 Jan 2021 14:00:01 +0330 > > > From: Seena Fallah <seenafallah(a)gmail.com> > > > Subject: [ceph-users] Re: PG inconsistent with empty inconsistent > > > objects > > > To: "Alexander E. Patrakov" <patrakov(a)gmail.com> > > > Cc: ceph-users <ceph-users(a)ceph.io> > > > Message-ID: > > > <CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com> > > > Content-Type: text/plain; charset="UTF-8" > > > > > > It's for a long time ago and I don't have the `ceph health detail` output! > > > > > > On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov <patrakov(a)gmail.com> > > > wrote: > > > > > > > For a start, please post the "ceph health detail" output. > > > > > > > > сб, 19 дек. 2020 г. в 23:48, Seena Fallah <seenafallah(a)gmail.com>om>: > > > > > > > > > > Hi, > > > > > > > > > > I'm facing something strange! One of the PGs in my pool got inconsistent > > > > > and when I run `rados list-inconsistent-obj $PG_ID --format=json-pretty` > > > > > the `inconsistents` key was empty! What is this? Is it a bug in Ceph > > > > or..? > > > > > > > > > > Thanks. > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > -- > > > > Alexander E. Patrakov > > > > CV: http://u.pc.cd/wT8otalK > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io