[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
Hi, I’m continuously getting scrub errors in my index pool and log pool that I need to repair always. HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 20.19 is active+clean+inconsistent, acting [39,41,37] Why is this? I have no cue at all, no log entry no anything ☹ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Forgot the very minimum entries after the scrub done: 2021-04-01T11:37:43.559539+0700 osd.39 (osd.39) 50 : cluster [DBG] 20.19 repair starts 2021-04-01T11:37:43.889909+0700 osd.39 (osd.39) 51 : cluster [ERR] 20.19 soid 20:990258ea:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.17263260.1.237:head : omap_digest 0x775cd866 != omap_digest 0xda11ecd0 from shard 39 2021-04-01T11:37:43.950318+0700 osd.39 (osd.39) 52 : cluster [ERR] 20.19 soid 20:994159a0:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.17263260.1.35:head : omap_digest 0xb61affda != omap_digest 0xb3467a38 from shard 39 2021-04-01T11:37:45.397338+0700 mgr.sg-cephmon-6s01 (mgr.25028786) 81795 : cluster [DBG] pgmap v81983: 225 pgs: 1 active+clean+scrubbing+deep+inconsistent+repair, 224 active+clean; 4.8 TiB data, 25 TiB used, 506 TiB / 531 TiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 14.28k op/s 2021-04-01T11:37:45.690930+0700 osd.39 (osd.39) 53 : cluster [ERR] 20.19 repair 0 missing, 2 inconsistent objects 2021-04-01T11:37:45.690951+0700 osd.39 (osd.39) 54 : cluster [ERR] 20.19 repair 2 errors, 0 fixed 2021-04-01T11:37:45.762565+0700 osd.39 (osd.39) 55 : cluster [DBG] 20.19 deep-scrub starts Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com --------------------------------------------------- -----Original Message----- From: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Sent: Thursday, April 1, 2021 11:38 AM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] [ERR] OSD_SCRUB_ERRORS: 2 scrub errors Hi, I’m continuously getting scrub errors in my index pool and log pool that I need to repair always. HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 20.19 is active+clean+inconsistent, acting [39,41,37] Why is this? I have no cue at all, no log entry no anything ☹ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey all, I'd like to pick up on this topic, since we also see regular scrub errors recently. Roughly one per week for around six weeks now. It's always a different PG and the repair command always helps after a while. But the regular re-occurrence seems it bit unsettling. How to best troubleshoot this. We are currently on ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable) Best Wishes, Mathias
"dmesg" on all the linux hosts and look for signs of failing drives. Look at smart data, your HBAs/disk controllers, OOB management logs, and so forth. If you're seeing scrub errors, it's probably a bad disk backing an OSD or OSDs. Is there a common OSD in the PGs you've run the repairs on? On Mon, Jan 9, 2023, at 03:37, Kuhring, Mathias wrote:
Hey all,
I'd like to pick up on this topic, since we also see regular scrub errors recently. Roughly one per week for around six weeks now. It's always a different PG and the repair command always helps after a while. But the regular re-occurrence seems it bit unsettling. How to best troubleshoot this.
We are currently on ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)
Best Wishes, Mathias
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, We too kept seeing this until a few months ago in a cluster with ~400 HDDs, while all the drive SMART statistics was always A-OK. Since we use erasure coding each PG involves up to 10 HDDs. It took us a while to realize we shouldn't expect scrub errors on healthy drives, but eventually we decided to track it down, and found documentation suggesting to use rados list-inconsistent-obj <PG> --format=json-pretty ... before you repair the PG. If you look into that (long) output, you are likely going to find a "read_error" for a specific OSD. Then we started to make a note of the HDD that saw the error. This helped us identify two HDDs that had multiple read errors within a few weeks, even though their SMART data was still perfectly fine. Now that *might* just be bad luck, but we have enough drives that we don't care, so we just replaced them, and since then I've only had a single drive report an error. One conclusion (in our case) is that it could be a drive that likely would have failed sooner or later, even though it hadn't yet reached a threshold for SMART to worry, or the alternative might be that it's a drive that just has more frequent read errors, but it's technically within the allowed variation. Assuming you have configured your cluster with reasonable redundancy you shouldn't run any risk of data losses, but for us we figured it's worth replacing a few outlier drives to sleep better. Cheers, Erik -- Erik Lindahl <erik.lindahl@gmail.com> On 9 Jan 2023 at 23:06 +0100, David Orman <ormandj@corenode.com>, wrote:
"dmesg" on all the linux hosts and look for signs of failing drives. Look at smart data, your HBAs/disk controllers, OOB management logs, and so forth. If you're seeing scrub errors, it's probably a bad disk backing an OSD or OSDs.
Is there a common OSD in the PGs you've run the repairs on?
On Mon, Jan 9, 2023, at 03:37, Kuhring, Mathias wrote:
Hey all,
I'd like to pick up on this topic, since we also see regular scrub errors recently. Roughly one per week for around six weeks now. It's always a different PG and the repair command always helps after a while. But the regular re-occurrence seems it bit unsettling. How to best troubleshoot this.
We are currently on ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)
Best Wishes, Mathias
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It's important to note we do not suggest using the SMART "OK" indicator as the drive being valid. We monitor correctable/uncorrectable error counts, as you can see a dramatic rise when the drives start to fail. 'OK' will be reported for SMART health long after the drive is throwing many uncorrectable errors and needs replacement. You have to look at the actual counters, themselves. That said, you will generally see these uncorrectable errors in the kernel output from dmesg, as well. On Mon, Jan 9, 2023, at 16:38, Erik Lindahl wrote:
Hi,
We too kept seeing this until a few months ago in a cluster with ~400 HDDs, while all the drive SMART statistics was always A-OK. Since we use erasure coding each PG involves up to 10 HDDs.
It took us a while to realize we shouldn't expect scrub errors on healthy drives, but eventually we decided to track it down, and found documentation suggesting to use
rados list-inconsistent-obj <PG> --format=json-pretty
... before you repair the PG. If you look into that (long) output, you are likely going to find a "read_error" for a specific OSD. Then we started to make a note of the HDD that saw the error.
This helped us identify two HDDs that had multiple read errors within a few weeks, even though their SMART data was still perfectly fine. Now that *might* just be bad luck, but we have enough drives that we don't care, so we just replaced them, and since then I've only had a single drive report an error.
One conclusion (in our case) is that it could be a drive that likely would have failed sooner or later, even though it hadn't yet reached a threshold for SMART to worry, or the alternative might be that it's a drive that just has more frequent read errors, but it's technically within the allowed variation. Assuming you have configured your cluster with reasonable redundancy you shouldn't run any risk of data losses, but for us we figured it's worth replacing a few outlier drives to sleep better.
Cheers,
Erik
-- Erik Lindahl <erik.lindahl@gmail.com> On 9 Jan 2023 at 23:06 +0100, David Orman <ormandj@corenode.com>, wrote:
"dmesg" on all the linux hosts and look for signs of failing drives. Look at smart data, your HBAs/disk controllers, OOB management logs, and so forth. If you're seeing scrub errors, it's probably a bad disk backing an OSD or OSDs.
Is there a common OSD in the PGs you've run the repairs on?
On Mon, Jan 9, 2023, at 03:37, Kuhring, Mathias wrote:
Hey all,
I'd like to pick up on this topic, since we also see regular scrub errors recently. Roughly one per week for around six weeks now. It's always a different PG and the repair command always helps after a while. But the regular re-occurrence seems it bit unsettling. How to best troubleshoot this.
We are currently on ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)
Best Wishes, Mathias
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Jan 9, 2023, at 17:46, David Orman <ormandj@corenode.com> wrote:
It's important to note we do not suggest using the SMART "OK" indicator as the drive being valid. We monitor correctable/uncorrectable error counts, as you can see a dramatic rise when the drives start to fail. 'OK' will be reported for SMART health long after the drive is throwing many uncorrectable errors and needs replacement. You have to look at the actual counters, themselves.
I strongly agree, especially given personal experience with SSD firmware design flaws. Also, examining UDMA / CRC error rates led to the discovery that certain aftermarket drive carriers had lower tolerances than those from the chassis vendor, resulting in drives that were silently slow. Reseating in most cases restored performance. — aad
Hi, Good points; however, given that ceph already collects all this statistics, isn't there any way to set (?) reasonable thresholds and actually have ceph detect the amount of read errors and suggest that a given drive should be replaced? It seems a bit strange that we all should have to wait for a PG read error, then log into each node to check the number of read errors for each device and keep track of this? Of course it's possible to write scripts for everything, but there must be numerous Ceph sites with hundreds of OSD nodes, so I'm a bit surprised this isn't more automated... Cheers, Erik -- Erik Lindahl <erik.lindahl@gmail.com> On 10 Jan 2023 at 00:09 +0100, Anthony D'Atri <aad@dreamsnake.net>, wrote:
On Jan 9, 2023, at 17:46, David Orman <ormandj@corenode.com> wrote:
It's important to note we do not suggest using the SMART "OK" indicator as the drive being valid. We monitor correctable/uncorrectable error counts, as you can see a dramatic rise when the drives start to fail. 'OK' will be reported for SMART health long after the drive is throwing many uncorrectable errors and needs replacement. You have to look at the actual counters, themselves.
I strongly agree, especially given personal experience with SSD firmware design flaws.
Also, examining UDMA / CRC error rates led to the discovery that certain aftermarket drive carriers had lower tolerances than those from the chassis vendor, resulting in drives that were silently slow. Reseating in most cases restored performance.
— aad
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We ship all of this to our centralized monitoring system (and a lot more) and have dashboards/proactive monitoring/alerting with 100PiB+ of Ceph. If you're running Ceph in production, I believe host-level monitoring is critical, above and beyond Ceph level. Things like inlet/outlet temperature, hardware state of various components, and various other details are probably best served by monitoring external to Ceph itself. I did a quick glance and didn't see this data (OSD errors re: reads/writes) exposed in the Pacific version of Ceph's Prometheus-style exporter, but I may have overlooked it. This would be nice to have, as well, if it does not exist. We collect drive counters at the host level, and alert at levels prior to general impact. Even a failing drive can cause latency spikes which are frustrating, before it starts returning errors (correctable errors) - the OSD will not see these other than longer latency on operations. Seeing a change in the smart counters either at a high rate or above thresholds you define is most certainly something I would suggest ensuring is covered in whatever host-level monitoring you're already performing for production usage. David On Mon, Jan 9, 2023, at 17:46, Erik Lindahl wrote:
Hi,
Good points; however, given that ceph already collects all this statistics, isn't there any way to set (?) reasonable thresholds and actually have ceph detect the amount of read errors and suggest that a given drive should be replaced?
It seems a bit strange that we all should have to wait for a PG read error, then log into each node to check the number of read errors for each device and keep track of this? Of course it's possible to write scripts for everything, but there must be numerous Ceph sites with hundreds of OSD nodes, so I'm a bit surprised this isn't more automated...
Cheers,
Erik
-- Erik Lindahl <erik.lindahl@gmail.com> On 10 Jan 2023 at 00:09 +0100, Anthony D'Atri <aad@dreamsnake.net>, wrote:
On Jan 9, 2023, at 17:46, David Orman <ormandj@corenode.com> wrote:
It's important to note we do not suggest using the SMART "OK" indicator as the drive being valid. We monitor correctable/uncorrectable error counts, as you can see a dramatic rise when the drives start to fail. 'OK' will be reported for SMART health long after the drive is throwing many uncorrectable errors and needs replacement. You have to look at the actual counters, themselves.
I strongly agree, especially given personal experience with SSD firmware design flaws.
Also, examining UDMA / CRC error rates led to the discovery that certain aftermarket drive carriers had lower tolerances than those from the chassis vendor, resulting in drives that were silently slow. Reseating in most cases restored performance.
— aad
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
On 10 Jan 2023, at 07:10, David Orman <ormandj@corenode.com> wrote:
We ship all of this to our centralized monitoring system (and a lot more) and have dashboards/proactive monitoring/alerting with 100PiB+ of Ceph. If you're running Ceph in production, I believe host-level monitoring is critical, above and beyond Ceph level. Things like inlet/outlet temperature, hardware state of various components, and various other details are probably best served by monitoring external to Ceph itself.
I agree with David's suggestions
I did a quick glance and didn't see this data (OSD errors re: reads/writes) exposed in the Pacific version of Ceph's Prometheus-style exporter, but I may have overlooked it. This would be nice to have, as well, if it does not exist.
We collect drive counters at the host level, and alert at levels prior to general impact. Even a failing drive can cause latency spikes which are frustrating, before it starts returning errors (correctable errors) - the OSD will not see these other than longer latency on operations. Seeing a change in the smart counters either at a high rate or above thresholds you define is most certainly something I would suggest ensuring is covered in whatever host-level monitoring you're already performing for production usage.
Seems to me that there is no need to reinvent the wheel and create even more GIL problems for ceph-mgr. In previous year was released production-ready exporter for smartctl data, with NVMe support [1] Golang, CI & tested in production with Ceph - ready to go 🙂 [1] https://github.com/prometheus-community/smartctl_exporter
participants (6)
-
Anthony D'Atri
-
David Orman
-
Erik Lindahl
-
Konstantin Shalygin
-
Kuhring, Mathias
-
Szabo, Istvan (Agoda)