Hi everyone. I got a warning with root@cthulhu1:/etc/ceph# ceph -s cluster: id: 9c5bb196-c212-11ee-84f3-c3f2beae892d health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent So I find the pg with the issue, and launch a pg repair (still waiting) But I try to find «why» so I check all the OSD related on this pg and didn't find anything, no error from osd daemon, no errors from smartctl, no error from the kernel message. So I just like to know if that's «normal» or should I scratch deeper. JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 12 avril 2024 11:51:37 CEST
Hello Albert, Have you check the hardware status of the involved drives other than with smartctl? Like with the manufacturer's tools / WebUI (iDrac / perccli for DELL hardware for example). If these tools don't report any media error (that is bad blocs on disks) then you might just be facing the bit rot phenomenon. But this is very rare and should happen in a sysadmin's lifetime as often as a Royal Flush hand in a professional poker player's lifetime. ;-) If no media error is reported, then you might want to check and update the firmware of all drives. Once you figured it out, you may enable osd_scrub_auto_repair=true to have these inconsistencies repaired automatically on deep-scrubbing, but make sure you're using the alert module [1] so to at least get informed about the scrub errors. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/mgr/alerts/ ----- Le 12 Avr 24, à 11:59, Albert Shih Albert.Shih@obspm.fr a écrit :
Hi everyone.
I got a warning with
root@cthulhu1:/etc/ceph# ceph -s cluster: id: 9c5bb196-c212-11ee-84f3-c3f2beae892d health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent
So I find the pg with the issue, and launch a pg repair (still waiting)
But I try to find «why» so I check all the OSD related on this pg and didn't find anything, no error from osd daemon, no errors from smartctl, no error from the kernel message.
So I just like to know if that's «normal» or should I scratch deeper.
JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 12 avril 2024 11:51:37 CEST _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
check your ceph.log on the mons for "stat mismatch" and grep for the PG in question for potentially more information. Additionally "rados list-inconsistent-obj {pgid}" will often show which OSD and objects are implicated for the inconsistency. If the acting set has changed since the scrub (for example an osd is removed or failed) in which the inconsistency was found this data wont be there any longer and you would need to deep-scrub the PG again to get that information. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Fri, Apr 12, 2024 at 6:56 AM Frédéric Nass < frederic.nass@univ-lorraine.fr> wrote:
Hello Albert,
Have you check the hardware status of the involved drives other than with smartctl? Like with the manufacturer's tools / WebUI (iDrac / perccli for DELL hardware for example). If these tools don't report any media error (that is bad blocs on disks) then you might just be facing the bit rot phenomenon. But this is very rare and should happen in a sysadmin's lifetime as often as a Royal Flush hand in a professional poker player's lifetime. ;-)
If no media error is reported, then you might want to check and update the firmware of all drives.
Once you figured it out, you may enable osd_scrub_auto_repair=true to have these inconsistencies repaired automatically on deep-scrubbing, but make sure you're using the alert module [1] so to at least get informed about the scrub errors.
Regards, Frédéric.
[1] https://docs.ceph.com/en/latest/mgr/alerts/
----- Le 12 Avr 24, à 11:59, Albert Shih Albert.Shih@obspm.fr a écrit :
Hi everyone.
I got a warning with
root@cthulhu1:/etc/ceph# ceph -s cluster: id: 9c5bb196-c212-11ee-84f3-c3f2beae892d health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent
So I find the pg with the issue, and launch a pg repair (still waiting)
But I try to find «why» so I check all the OSD related on this pg and didn't find anything, no error from osd daemon, no errors from smartctl, no error from the kernel message.
So I just like to know if that's «normal» or should I scratch deeper.
JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 12 avril 2024 11:51:37 CEST _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 12/04/2024 à 12:56:12+0200, Frédéric Nass a écrit
Hi,
Have you check the hardware status of the involved drives other than with smartctl? Like with the manufacturer's tools / WebUI (iDrac / perccli for DELL hardware for example).
Yes, all my disk are «under» periodic check with smartctl + icinga.
If these tools don't report any media error (that is bad blocs on disks) then you might just be facing the bit rot phenomenon. But this is very rare and should happen in a sysadmin's lifetime as often as a Royal Flush hand in a professional poker player's lifetime. ;-)
If no media error is reported, then you might want to check and update the firmware of all drives.
You're perfectly right. It's just a newbie error, I check on the «main» osd of the PG (meaning the first in the list) but forget to check on other. On when server I indeed get some error on a disk. But strangely smartctl report nothing. I will add a check with dmesg.
Once you figured it out, you may enable osd_scrub_auto_repair=true to have these inconsistencies repaired automatically on deep-scrubbing, but make sure you're using the alert module [1] so to at least get informed about the scrub errors.
Thanks. I will look into because we got already icinga2 on site so I use icinga2 to check the cluster. Is they are a list of what the alert module going to check ? Regards JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 12 avril 2024 15:13:13 CEST
----- Le 12 Avr 24, à 15:17, Albert Shih Albert.Shih@obspm.fr a écrit :
Le 12/04/2024 à 12:56:12+0200, Frédéric Nass a écrit
Hi,
Have you check the hardware status of the involved drives other than with smartctl? Like with the manufacturer's tools / WebUI (iDrac / perccli for DELL hardware for example).
Yes, all my disk are «under» periodic check with smartctl + icinga.
Actually, I meant lower level tools (drive / server vendor tools).
If these tools don't report any media error (that is bad blocs on disks) then you might just be facing the bit rot phenomenon. But this is very rare and should happen in a sysadmin's lifetime as often as a Royal Flush hand in a professional poker player's lifetime. ;-)
If no media error is reported, then you might want to check and update the firmware of all drives.
You're perfectly right.
It's just a newbie error, I check on the «main» osd of the PG (meaning the first in the list) but forget to check on other.
Ok.
On when server I indeed get some error on a disk.
But strangely smartctl report nothing. I will add a check with dmesg.
That's why I pointed you to the drive / server vendor tools earlier as sometimes smartctl is missing the information you want.
Once you figured it out, you may enable osd_scrub_auto_repair=true to have these inconsistencies repaired automatically on deep-scrubbing, but make sure you're using the alert module [1] so to at least get informed about the scrub errors.
Thanks. I will look into because we got already icinga2 on site so I use icinga2 to check the cluster.
Is they are a list of what the alert module going to check ?
Basically the module checks for ceph status (ceph -s) changes. https://github.com/ceph/ceph/blob/main/src/pybind/mgr/alerts/module.py Regards, Frédéric.
Regards
JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 12 avril 2024 15:13:13 CEST
If you're using an Icinga active check that just looks for SMART overall-health self-assessment test result: PASSED then it's not doing much for you. That bivalue status can be shown for a drive that is decidedly an ex-parrot. Gotta look at specific attributes, which is thorny since they aren't consistently implemented. drivedb.h is a downright mess, which doesn't help.
----- Le 12 Avr 24, à 15:17, Albert Shih Albert.Shih@obspm.fr a écrit :
Le 12/04/2024 à 12:56:12+0200, Frédéric Nass a écrit
Hi,
Have you check the hardware status of the involved drives other than with smartctl? Like with the manufacturer's tools / WebUI (iDrac / perccli for DELL hardware for example).
Yes, all my disk are «under» periodic check with smartctl + icinga.
Actually, I meant lower level tools (drive / server vendor tools).
If these tools don't report any media error (that is bad blocs on disks) then you might just be facing the bit rot phenomenon. But this is very rare and should happen in a sysadmin's lifetime as often as a Royal Flush hand in a professional poker player's lifetime. ;-)
If no media error is reported, then you might want to check and update the firmware of all drives.
You're perfectly right.
It's just a newbie error, I check on the «main» osd of the PG (meaning the first in the list) but forget to check on other.
Ok.
On when server I indeed get some error on a disk.
But strangely smartctl report nothing. I will add a check with dmesg.
That's why I pointed you to the drive / server vendor tools earlier as sometimes smartctl is missing the information you want.
Once you figured it out, you may enable osd_scrub_auto_repair=true to have these inconsistencies repaired automatically on deep-scrubbing, but make sure you're using the alert module [1] so to at least get informed about the scrub errors.
Thanks. I will look into because we got already icinga2 on site so I use icinga2 to check the cluster.
Is they are a list of what the alert module going to check ?
Basically the module checks for ceph status (ceph -s) changes.
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/alerts/module.py
Regards, Frédéric.
Regards
JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 12 avril 2024 15:13:13 CEST
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Albert Shih
-
Anthony D'Atri
-
Frédéric Nass
-
Wesley Dillingham