1 Apr
2021
1 Apr
'21
10:07 a.m.
Hi,
I’m continuously getting scrub errors in my index pool and log pool that I need to repair
always.
HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent
[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 20.19 is active+clean+inconsistent, acting [39,41,37]
Why is this?
I have no cue at all, no log entry no anything ☹
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
1 Apr
1 Apr
10:09 a.m.
Forgot the very minimum entries after the scrub done:
2021-04-01T11:37:43.559539+0700 osd.39 (osd.39) 50 : cluster [DBG] 20.19 repair starts
2021-04-01T11:37:43.889909+0700 osd.39 (osd.39) 51 : cluster [ERR] 20.19 soid
20:990258ea:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.17263260.1.237:head : omap_digest
0x775cd866 != omap_digest 0xda11ecd0 from shard 39
2021-04-01T11:37:43.950318+0700 osd.39 (osd.39) 52 : cluster [ERR] 20.19 soid
20:994159a0:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.17263260.1.35:head : omap_digest
0xb61affda != omap_digest 0xb3467a38 from shard 39
2021-04-01T11:37:45.397338+0700 mgr.sg-cephmon-6s01 (mgr.25028786) 81795 : cluster [DBG]
pgmap v81983: 225 pgs: 1 active+clean+scrubbing+deep+inconsistent+repair, 224
active+clean; 4.8 TiB data, 25 TiB used, 506 TiB / 531 TiB avail; 14 MiB/s rd, 5.0 MiB/s
wr, 14.28k op/s
2021-04-01T11:37:45.690930+0700 osd.39 (osd.39) 53 : cluster [ERR] 20.19 repair 0 missing,
2 inconsistent objects
2021-04-01T11:37:45.690951+0700 osd.39 (osd.39) 54 : cluster [ERR] 20.19 repair 2 errors,
0 fixed
2021-04-01T11:37:45.762565+0700 osd.39 (osd.39) 55 : cluster [DBG] 20.19 deep-scrub
starts
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com
---------------------------------------------------
-----Original Message-----
From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Sent: Thursday, April 1, 2021 11:38 AM
To: ceph-users <ceph-users(a)ceph.io>
Subject: [ceph-users] [ERR] OSD_SCRUB_ERRORS: 2 scrub errors
Hi,
I’m continuously getting scrub errors in my index pool and log pool that I need to repair
always.
HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS:
2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 20.19 is active+clean+inconsistent, acting [39,41,37]
Why is this?
I have no cue at all, no log entry no anything ☹
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
9 Jan
9 Jan
3:07 p.m.
Hey all,
I'd like to pick up on this topic, since we also see regular scrub
errors recently.
Roughly one per week for around six weeks now.
It's always a different PG and the repair command always helps after a
while.
But the regular re-occurrence seems it bit unsettling.
How to best troubleshoot this.
We are currently on ceph version 17.2.1
(ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)
Best Wishes,
Mathias
10 Jan
10 Jan
3:35 a.m.
"dmesg" on all the linux hosts and look for signs of failing drives. Look at
smart data, your HBAs/disk controllers, OOB management logs, and so forth. If you're
seeing scrub errors, it's probably a bad disk backing an OSD or OSDs.
Is there a common OSD in the PGs you've run the repairs on?
On Mon, Jan 9, 2023, at 03:37, Kuhring, Mathias wrote:
Hey all,
I'd like to pick up on this topic, since we also see regular scrub
errors recently.
Roughly one per week for around six weeks now.
It's always a different PG and the repair command always helps after a
while.
But the regular re-occurrence seems it bit unsettling.
How to best troubleshoot this.
We are currently on ceph version 17.2.1
(ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)
Best Wishes,
Mathias
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:08 a.m.
Hi,
We too kept seeing this until a few months ago in a cluster with ~400 HDDs, while all the
drive SMART statistics was always A-OK. Since we use erasure coding each PG involves up to
10 HDDs.
It took us a while to realize we shouldn't expect scrub errors on healthy drives, but
eventually we decided to track it down, and found documentation suggesting to use
rados list-inconsistent-obj <PG> --format=json-pretty
... before you repair the PG. If you look into that (long) output, you are likely going to
find a "read_error" for a specific OSD. Then we started to make a note of the
HDD that saw the error.
This helped us identify two HDDs that had multiple read errors within a few weeks, even
though their SMART data was still perfectly fine. Now that *might* just be bad luck, but
we have enough drives that we don't care, so we just replaced them, and since then
I've only had a single drive report an error.
One conclusion (in our case) is that it could be a drive that likely would have failed
sooner or later, even though it hadn't yet reached a threshold for SMART to worry, or
the alternative might be that it's a drive that just has more frequent read errors,
but it's technically within the allowed variation. Assuming you have configured your
cluster with reasonable redundancy you shouldn't run any risk of data losses, but for
us we figured it's worth replacing a few outlier drives to sleep better.
Cheers,
Erik
--
Erik Lindahl <erik.lindahl(a)gmail.com>
On 9 Jan 2023 at 23:06 +0100, David Orman <ormandj(a)corenode.com>om>, wrote:
"dmesg" on all the linux hosts and look for
signs of failing drives. Look at smart data, your HBAs/disk controllers, OOB management
logs, and so forth. If you're seeing scrub errors, it's probably a bad disk
backing an OSD or OSDs.
Is there a common OSD in the PGs you've run the repairs on?
On Mon, Jan 9, 2023, at 03:37, Kuhring, Mathias wrote:
Hey all,
I'd like to pick up on this topic, since we also see regular scrub
errors recently.
Roughly one per week for around six weeks now.
It's always a different PG and the repair command always helps after a
while.
But the regular re-occurrence seems it bit unsettling.
How to best troubleshoot this.
We are currently on ceph version 17.2.1
(ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)
Best Wishes,
Mathias
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:16 a.m.
It's important to note we do not suggest using the SMART "OK" indicator as
the drive being valid. We monitor correctable/uncorrectable error counts, as you can see a
dramatic rise when the drives start to fail. 'OK' will be reported for SMART
health long after the drive is throwing many uncorrectable errors and needs replacement.
You have to look at the actual counters, themselves.
That said, you will generally see these uncorrectable errors in the kernel output from
dmesg, as well.
On Mon, Jan 9, 2023, at 16:38, Erik Lindahl wrote:
Hi,
We too kept seeing this until a few months ago in a cluster with ~400 HDDs, while all the
drive SMART statistics was always A-OK. Since we use erasure coding each PG involves up to
10 HDDs.
It took us a while to realize we shouldn't expect scrub errors on healthy drives, but
eventually we decided to track it down, and found documentation suggesting to use
rados list-inconsistent-obj <PG> --format=json-pretty
... before you repair the PG. If you look into that (long) output, you are likely going
to find a "read_error" for a specific OSD. Then we started to make a note of the
HDD that saw the error.
This helped us identify two HDDs that had multiple read errors within a few weeks, even
though their SMART data was still perfectly fine. Now that *might* just be bad luck, but
we have enough drives that we don't care, so we just replaced them, and since then
I've only had a single drive report an error.
One conclusion (in our case) is that it could be a drive that likely would have failed
sooner or later, even though it hadn't yet reached a threshold for SMART to worry, or
the alternative might be that it's a drive that just has more frequent read errors,
but it's technically within the allowed variation. Assuming you have configured your
cluster with reasonable redundancy you shouldn't run any risk of data losses, but for
us we figured it's worth replacing a few outlier drives to sleep better.
Cheers,
Erik
--
Erik Lindahl <erik.lindahl(a)gmail.com>
On 9 Jan 2023 at 23:06 +0100, David Orman <ormandj(a)corenode.com>om>, wrote:
"dmesg" on all the linux hosts and look
for signs of failing drives. Look at smart data, your HBAs/disk controllers, OOB
management logs, and so forth. If you're seeing scrub errors, it's probably a bad
disk backing an OSD or OSDs.
Is there a common OSD in the PGs you've run the repairs on?
On Mon, Jan 9, 2023, at 03:37, Kuhring, Mathias wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hey all,
I'd like to pick up on this topic, since we also see regular scrub
errors recently.
Roughly one per week for around six weeks now.
It's always a different PG and the repair command always helps after a
while.
But the regular re-occurrence seems it bit unsettling.
How to best troubleshoot this.
We are currently on ceph version 17.2.1
(ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)
Best Wishes,
Mathias
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
4:38 a.m.
On Jan 9, 2023, at 17:46, David Orman
<ormandj(a)corenode.com> wrote:
It's important to note we do not suggest using the SMART "OK" indicator as
the drive being valid. We monitor correctable/uncorrectable error counts, as you can see a
dramatic rise when the drives start to fail. 'OK' will be reported for SMART
health long after the drive is throwing many uncorrectable errors and needs replacement.
You have to look at the actual counters, themselves.
I strongly agree, especially given personal experience with SSD firmware design flaws.
Also, examining UDMA / CRC error rates led to the discovery that certain aftermarket drive
carriers had lower tolerances than those from the chassis vendor, resulting in drives that
were silently slow. Reseating in most cases restored performance.
— aad
5:16 a.m.
Hi,
Good points; however, given that ceph already collects all this statistics,
isn't there any way to set (?) reasonable thresholds and actually have ceph detect
the amount of read errors and suggest that a given drive should be replaced?
It seems a bit strange that we all should have to wait for a PG read error, then log into
each node to check the number of read errors for each device and keep track of this? Of
course it's possible to write scripts for everything, but there must be numerous Ceph
sites with hundreds of OSD nodes, so I'm a bit surprised this isn't more
automated...
Cheers,
Erik
--
Erik Lindahl <erik.lindahl(a)gmail.com>
On 10 Jan 2023 at 00:09 +0100, Anthony D'Atri <aad(a)dreamsnake.net>et>, wrote:
On Jan 9, 2023, at 17:46, David Orman
<ormandj(a)corenode.com> wrote:
It's important to note we do not suggest using the SMART "OK" indicator as
the drive being valid. We monitor correctable/uncorrectable error counts, as you can see a
dramatic rise when the drives start to fail. 'OK' will be reported for SMART
health long after the drive is throwing many uncorrectable errors and needs replacement.
You have to look at the actual counters, themselves.
I strongly agree, especially given personal experience with SSD firmware design flaws.
Also, examining UDMA / CRC error rates led to the discovery that certain aftermarket
drive carriers had lower tolerances than those from the chassis vendor, resulting in
drives that were silently slow. Reseating in most cases restored performance.
— aad
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:40 a.m.
We ship all of this to our centralized monitoring system (and a lot more) and have
dashboards/proactive monitoring/alerting with 100PiB+ of Ceph. If you're running Ceph
in production, I believe host-level monitoring is critical, above and beyond Ceph level.
Things like inlet/outlet temperature, hardware state of various components, and various
other details are probably best served by monitoring external to Ceph itself.
I did a quick glance and didn't see this data (OSD errors re: reads/writes) exposed in
the Pacific version of Ceph's Prometheus-style exporter, but I may have overlooked it.
This would be nice to have, as well, if it does not exist.
We collect drive counters at the host level, and alert at levels prior to general impact.
Even a failing drive can cause latency spikes which are frustrating, before it starts
returning errors (correctable errors) - the OSD will not see these other than longer
latency on operations. Seeing a change in the smart counters either at a high rate or
above thresholds you define is most certainly something I would suggest ensuring is
covered in whatever host-level monitoring you're already performing for production
usage.
David
On Mon, Jan 9, 2023, at 17:46, Erik Lindahl wrote:
Hi,
Good points; however, given that ceph already collects all this statistics, isn't
there any way to set (?) reasonable thresholds and actually have ceph detect the amount of
read errors and suggest that a given drive should be replaced?
It seems a bit strange that we all should have to wait for a PG read error, then log into
each node to check the number of read errors for each device and keep track of this? Of
course it's possible to write scripts for everything, but there must be numerous Ceph
sites with hundreds of OSD nodes, so I'm a bit surprised this isn't more
automated...
Cheers,
Erik
--
Erik Lindahl <erik.lindahl(a)gmail.com>
On 10 Jan 2023 at 00:09 +0100, Anthony D'Atri <aad(a)dreamsnake.net>et>, wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
On Jan 9, 2023, at 17:46, David Orman
<ormandj(a)corenode.com> wrote:
It's important to note we do not suggest using the SMART "OK" indicator as
the drive being valid. We monitor correctable/uncorrectable error counts, as you can see a
dramatic rise when the drives start to fail. 'OK' will be reported for SMART
health long after the drive is throwing many uncorrectable errors and needs replacement.
You have to look at the actual counters, themselves.
I strongly agree, especially given personal experience with SSD firmware design flaws.
Also, examining UDMA / CRC error rates led to the discovery that certain aftermarket
drive carriers had lower tolerances than those from the chassis vendor, resulting in
drives that were silently slow. Reseating in most cases restored performance.
— aad
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
11 Jan
11 Jan
3:56 p.m.
Hi,
On 10 Jan 2023, at 07:10, David Orman
<ormandj(a)corenode.com> wrote:
We ship all of this to our centralized monitoring system (and a lot more) and have
dashboards/proactive monitoring/alerting with 100PiB+ of Ceph. If you're running Ceph
in production, I believe host-level monitoring is critical, above and beyond Ceph level.
Things like inlet/outlet temperature, hardware state of various components, and various
other details are probably best served by monitoring external to Ceph itself.
I agree with David's suggestions
I did a quick glance and didn't see this data (OSD errors re: reads/writes) exposed
in the Pacific version of Ceph's Prometheus-style exporter, but I may have overlooked
it. This would be nice to have, as well, if it does not exist.
We collect drive counters at the host level, and alert at levels prior to general impact.
Even a failing drive can cause latency spikes which are frustrating, before it starts
returning errors (correctable errors) - the OSD will not see these other than longer
latency on operations. Seeing a change in the smart counters either at a high rate or
above thresholds you define is most certainly something I would suggest ensuring is
covered in whatever host-level monitoring you're already performing for production
usage.
Seems to me that there is no need to reinvent the wheel and create even more GIL problems
for ceph-mgr. In previous year was released production-ready exporter for smartctl data,
with NVMe support [1]
Golang, CI & tested in production with Ceph - ready to go 🙂
[1] https://github.com/prometheus-community/smartctl_exporter
471
days inactive
1121
days old
9 comments
6 participants
participants (6)
-
Anthony D'Atri -
David Orman -
Erik Lindahl -
Konstantin Shalygin -
Kuhring, Mathias -
Szabo, Istvan (Agoda)