27 Jan
2021
27 Jan
'21
5:10 a.m.
Hi Everyone,
I also have seen this inconsistent with empty when you do list-inconsistent-obj
$ sudo ceph health detail
HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1
pgs not deep-scrubbed in time
OSD_SCRUB_ERRORS 1 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21]
PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time
pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811
$ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty
{
"epoch": 183807,
"inconsistents": []
}
Usually these are caused by read errors on the disks, but I've checked
all osd hosts that are part of this osd and there's no smart or dmesg
errors.
Rich
------------------------------
>
> Date: Sun, 17 Jan 2021 14:00:01 +0330
> From: Seena Fallah <seenafallah(a)gmail.com>
> Subject: [ceph-users] Re: PG inconsistent with empty inconsistent
> objects
> To: "Alexander E. Patrakov" <patrakov(a)gmail.com>
> Cc: ceph-users <ceph-users(a)ceph.io>
> Message-ID:
> <CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
>
> It's for a long time ago and I don't have the `ceph health detail` output!
>
> On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov <patrakov(a)gmail.com>
> wrote:
>
> > For a start, please post the "ceph health detail" output.
> >
> > сб, 19 дек. 2020 г. в 23:48, Seena Fallah <seenafallah(a)gmail.com>om>:
> > >
> > > Hi,
> > >
> > > I'm facing something strange! One of the PGs in my pool got
inconsistent
> > > and when I run `rados list-inconsistent-obj $PG_ID --format=json-pretty`
> > > the `inconsistents` key was empty! What is this? Is it a bug in Ceph
> > or..?
> > >
> > > Thanks.
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> >
> >
> > --
> > Alexander E. Patrakov
> > CV: http://u.pc.cd/wT8otalK
> >
>
27 Jan
27 Jan
5:27 a.m.
New subject: PG inconsistent with empty inconsistent objects
just issue the commands
scrub pg deep-scrub 17.1cs
this will deep scrub this pg
ceph pg repair 17.7ff
repairs the pg
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> Richard Bade <hitrich(a)gmail.com>
1/26/2021 3:40 PM >>>
Hi Everyone,
I also have seen this inconsistent with empty when you do
list-inconsistent-obj
$ sudo ceph health detail
HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1
pgs not deep-scrubbed in time
OSD_SCRUB_ERRORS 1 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 17.7ff is active+clean+inconsistent, acting
[232,242,34,280,266,21]
PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time
pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811
$ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty
{
"epoch": 183807,
"inconsistents": []
}
Usually these are caused by read errors on the disks, but I've checked
all osd hosts that are part of this osd and there's no smart or dmesg
errors.
Rich
------------------------------
Date: Sun, 17 Jan 2021 14:00:01 +0330
From: Seena Fallah <seenafallah(a)gmail.com>
Subject: [ceph-users] Re: PG inconsistent with empty inconsistent
objects
To: "Alexander E. Patrakov" <patrakov(a)gmail.com>
Cc: ceph-users <ceph-users(a)ceph.io>
Message-ID:
<CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
It's for a long time ago and I don't have the `ceph health detail`
output!
On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov
<patrakov(a)gmail.com>
wrote:
> For a start, please post the "ceph health detail" output.
>
> сб, 19 дек. 2020 г. в 23:48, Seena Fallah <seenafallah(a)gmail.com>om>:
> >
> > Hi,
> >
> > I'm facing something strange! One of the PGs in my pool got
inconsistent
> > and when I run `rados list-inconsistent-obj
$PG_ID
--format=json-pretty`
> > the `inconsistents` key was empty! What is
this? Is it a bug in
Ceph
or..?
Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Alexander E. Patrakov
CV: http://u.pc.cd/wT8otalK
5:35 a.m.
New subject: PG inconsistent with empty inconsistent objects
Thanks Joe for your reply.
Yes I realise I can scrub the one that's behind, that's not my issue this
time. I'm interested in the inconsistent pg.
Usually the list inconsistent obj command gives which copy is wrong and
what the issue is. In this case it reports nothing.
I don't really want to blindly repair as in the past the repair has copied
the primary over the other copies (this may have been fixed).
Usually in this situation I match up the read errors with the failing disk,
set the disk out and run a deep scrub and all is well again.
Ceph v14.2.13 by the way.
Rich
On Wed, 27 Jan 2021, 12:57 Joe Comeau, <Joe.Comeau(a)hli.ubc.ca> wrote:
just issue the commands
scrub pg deep-scrub 17.1cs
this will deep scrub this pg
ceph pg repair 17.7ff
repairs the pg
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> Richard Bade <hitrich(a)gmail.com>
1/26/2021 3:40 PM >>>
Hi Everyone,
I also have seen this inconsistent with empty when you do
list-inconsistent-obj
$ sudo ceph health detail
HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1
pgs not deep-scrubbed in time
OSD_SCRUB_ERRORS 1 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21]
PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time
pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811
$ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty
{
"epoch": 183807,
"inconsistents": []
}
Usually these are caused by read errors on the disks, but I've checked
all osd hosts that are part of this osd and there's no smart or dmesg
errors.
Rich
------------------------------
Date: Sun, 17 Jan 2021 14:00:01 +0330
From: Seena Fallah <seenafallah(a)gmail.com>
Subject: [ceph-users] Re: PG inconsistent with empty inconsistent
objects
To: "Alexander E. Patrakov" <patrakov(a)gmail.com>
Cc: ceph-users <ceph-users(a)ceph.io>
Message-ID:
<
CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com>
Content-Type: text/plain;
charset="UTF-8"
It's for a long time ago and I don't have the `ceph health detail`
output!
On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov <
patrakov(a)gmail.com>
wrote:
> For a start, please post the "ceph health detail" output.
>
> сб, 19 дек. 2020 г. в 23:48, Seena Fallah <seenafallah(a)gmail.com>om>:
> >
> > Hi,
> >
> > I'm facing something strange! One of the PGs in my pool got
inconsistent
> > and when I run `rados
list-inconsistent-obj $PG_ID
--format=json-pretty`
the `inconsistents` key was empty! What is this? Is it
a bug in Ceph
or..?
Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Alexander E. Patrakov
CV: http://u.pc.cd/wT8otalK
8:19 p.m.
New subject: PG inconsistent with empty inconsistent objects
Paste your `ceph versions` please
k
Sent from my iPhone
> On 27 Jan 2021, at 03:07, Richard Bade <hitrich(a)gmail.com> wrote:
>
> Ceph v14.2.13 by the way.
8:28 p.m.
New subject: PG inconsistent with empty inconsistent objects
Usually the ceph.log prints the reason for the inconsistency when it
is first detected by scrubbing.
-- dan
On Wed, Jan 27, 2021 at 12:41 AM Richard Bade <hitrich(a)gmail.com> wrote:
>
> Hi Everyone,
> I also have seen this inconsistent with empty when you do list-inconsistent-obj
>
> $ sudo ceph health detail
> HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1
> pgs not deep-scrubbed in time
> OSD_SCRUB_ERRORS 1 scrub errors
> PG_DAMAGED Possible data damage: 1 pg inconsistent
> pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21]
> PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time
> pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811
>
> $ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty
> {
> "epoch": 183807,
> "inconsistents": []
> }
>
> Usually these are caused by read errors on the disks, but I've checked
> all osd hosts that are part of this osd and there's no smart or dmesg
> errors.
>
> Rich
>
> ------------------------------
> >
> > Date: Sun, 17 Jan 2021 14:00:01 +0330
> > From: Seena Fallah <seenafallah(a)gmail.com>
> > Subject: [ceph-users] Re: PG inconsistent with empty inconsistent
> > objects
> > To: "Alexander E. Patrakov" <patrakov(a)gmail.com>
> > Cc: ceph-users <ceph-users(a)ceph.io>
> > Message-ID:
> >
<CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > It's for a long time ago and I don't have the `ceph health detail`
output!
> >
> > On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov
<patrakov(a)gmail.com>
> > wrote:
> >
> > > For a start, please post the "ceph health detail" output.
> > >
> > > сб, 19 дек. 2020 г. в 23:48, Seena Fallah <seenafallah(a)gmail.com>om>:
> > > >
> > > > Hi,
> > > >
> > > > I'm facing something strange! One of the PGs in my pool got
inconsistent
> > > > and when I run `rados list-inconsistent-obj $PG_ID
--format=json-pretty`
> > > > the `inconsistents` key was empty! What is this? Is it a bug in Ceph
> > > or..?
> > > >
> > > > Thanks.
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > >
> > >
> > >
> > > --
> > > Alexander E. Patrakov
> > > CV: http://u.pc.cd/wT8otalK
> > >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
28 Jan
28 Jan
4:11 a.m.
New subject: PG inconsistent with empty inconsistent objects
Thanks Dan and Anthony your suggestions have pointed me in the right
direction. Looking back through the logs at when the first error was
detected I found this:
ceph-osd: 2021-01-24 01:04:55.905 7f0c17821700 -1 log_channel(cluster)
log [ERR] : 17.7ffs0 scrub : stat mismatch, got 112867/112868 objects,
0/0 clones, 112867/112868 dirty, 0/0 omap, 0/0 pinned, 0/0
hit_set_archive, 0/0 whiteouts, 473372381184/473376575488 bytes, 0/0
manifest objects, 0/0 hit_set_archive bytes.
As Anthony suggested the error is not in the rados objects but
actually the stats.
I assume that a repair will fix this up?
Thanks again everyone.
Rich
On Thu, 28 Jan 2021 at 03:59, Dan van der Ster <dan(a)vanderster.com> wrote:
>
> Usually the ceph.log prints the reason for the inconsistency when it
> is first detected by scrubbing.
>
> -- dan
>
> On Wed, Jan 27, 2021 at 12:41 AM Richard Bade <hitrich(a)gmail.com> wrote:
> >
> > Hi Everyone,
> > I also have seen this inconsistent with empty when you do list-inconsistent-obj
> >
> > $ sudo ceph health detail
> > HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1
> > pgs not deep-scrubbed in time
> > OSD_SCRUB_ERRORS 1 scrub errors
> > PG_DAMAGED Possible data damage: 1 pg inconsistent
> > pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21]
> > PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time
> > pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811
> >
> > $ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty
> > {
> > "epoch": 183807,
> > "inconsistents": []
> > }
> >
> > Usually these are caused by read errors on the disks, but I've checked
> > all osd hosts that are part of this osd and there's no smart or dmesg
> > errors.
> >
> > Rich
> >
> > ------------------------------
> > >
> > > Date: Sun, 17 Jan 2021 14:00:01 +0330
> > > From: Seena Fallah <seenafallah(a)gmail.com>
> > > Subject: [ceph-users] Re: PG inconsistent with empty inconsistent
> > > objects
> > > To: "Alexander E. Patrakov" <patrakov(a)gmail.com>
> > > Cc: ceph-users <ceph-users(a)ceph.io>
> > > Message-ID:
> > >
<CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > It's for a long time ago and I don't have the `ceph health detail`
output!
> > >
> > > On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov
<patrakov(a)gmail.com>
> > > wrote:
> > >
> > > > For a start, please post the "ceph health detail" output.
> > > >
> > > > сб, 19 дек. 2020 г. в 23:48, Seena Fallah
<seenafallah(a)gmail.com>om>:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm facing something strange! One of the PGs in my pool got
inconsistent
> > > > > and when I run `rados list-inconsistent-obj $PG_ID
--format=json-pretty`
> > > > > the `inconsistents` key was empty! What is this? Is it a bug in
Ceph
> > > > or..?
> > > > >
> > > > > Thanks.
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > >
> > > >
> > > >
> > > > --
> > > > Alexander E. Patrakov
> > > > CV: http://u.pc.cd/wT8otalK
> > > >
> > >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
1193
days inactive
1194
days old
5 comments
4 participants
participants (4)
-
Dan van der Ster -
Joe Comeau -
Konstantin Shalygin -
Richard Bade