12 Apr
2024
12 Apr
'24
9:59 a.m.
Hi everyone.
I got a warning with
root@cthulhu1:/etc/ceph# ceph -s
cluster:
id: 9c5bb196-c212-11ee-84f3-c3f2beae892d
health: HEALTH_ERR
1 scrub errors
Possible data damage: 1 pg inconsistent
So I find the pg with the issue, and launch a pg repair (still waiting)
But I try to find «why» so I check all the OSD related on this pg and
didn't find anything, no error from osd daemon, no errors from smartctl, no
error from the kernel message.
So I just like to know if that's «normal» or should I scratch deeper.
JAS
--
Albert SHIH 🦫 🐸
France
Heure locale/Local time:
ven. 12 avril 2024 11:51:37 CEST
12 Apr
12 Apr
10:56 a.m.
Hello Albert,
Have you check the hardware status of the involved drives other than with smartctl? Like
with the manufacturer's tools / WebUI (iDrac / perccli for DELL hardware for
example).
If these tools don't report any media error (that is bad blocs on disks) then you
might just be facing the bit rot phenomenon. But this is very rare and should happen in a
sysadmin's lifetime as often as a Royal Flush hand in a professional poker
player's lifetime. ;-)
If no media error is reported, then you might want to check and update the firmware of all
drives.
Once you figured it out, you may enable osd_scrub_auto_repair=true to have these
inconsistencies repaired automatically on deep-scrubbing, but make sure you're using
the alert module [1] so to at least get informed about the scrub errors.
Regards,
Frédéric.
[1] https://docs.ceph.com/en/latest/mgr/alerts/
----- Le 12 Avr 24, à 11:59, Albert Shih Albert.Shih(a)obspm.fr a écrit :
> Hi everyone.
>
> I got a warning with
>
> root@cthulhu1:/etc/ceph# ceph -s
> cluster:
> id: 9c5bb196-c212-11ee-84f3-c3f2beae892d
> health: HEALTH_ERR
> 1 scrub errors
> Possible data damage: 1 pg inconsistent
>
> So I find the pg with the issue, and launch a pg repair (still waiting)
>
> But I try to find «why» so I check all the OSD related on this pg and
> didn't find anything, no error from osd daemon, no errors from smartctl, no
> error from the kernel message.
>
> So I just like to know if that's «normal» or should I scratch deeper.
>
> JAS
> --
> Albert SHIH 🦫 🐸
> France
> Heure locale/Local time:
> ven. 12 avril 2024 11:51:37 CEST
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:49 p.m.
check your ceph.log on the mons for "stat mismatch" and grep for the PG in
question for potentially more information.
Additionally "rados list-inconsistent-obj {pgid}" will often show which OSD
and objects are implicated for the inconsistency. If the acting set has
changed since the scrub (for example an osd is removed or failed) in which
the inconsistency was found this data wont be there any longer and you
would need to deep-scrub the PG again to get that information.
Respectfully,
*Wes Dillingham*
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
wes(a)wesdillingham.com
On Fri, Apr 12, 2024 at 6:56 AM Frédéric Nass <
frederic.nass(a)univ-lorraine.fr> wrote:
Hello Albert,
Have you check the hardware status of the involved drives other than with
smartctl? Like with the manufacturer's tools / WebUI (iDrac / perccli for
DELL hardware for example).
If these tools don't report any media error (that is bad blocs on disks)
then you might just be facing the bit rot phenomenon. But this is very rare
and should happen in a sysadmin's lifetime as often as a Royal Flush hand
in a professional poker player's lifetime. ;-)
If no media error is reported, then you might want to check and update the
firmware of all drives.
Once you figured it out, you may enable osd_scrub_auto_repair=true to have
these inconsistencies repaired automatically on deep-scrubbing, but make
sure you're using the alert module [1] so to at least get informed about
the scrub errors.
Regards,
Frédéric.
[1] https://docs.ceph.com/en/latest/mgr/alerts/
----- Le 12 Avr 24, à 11:59, Albert Shih Albert.Shih(a)obspm.fr a écrit :
Hi everyone.
I got a warning with
root@cthulhu1:/etc/ceph# ceph -s
cluster:
id: 9c5bb196-c212-11ee-84f3-c3f2beae892d
health: HEALTH_ERR
1 scrub errors
Possible data damage: 1 pg inconsistent
So I find the pg with the issue, and launch a pg repair (still waiting)
But I try to find «why» so I check all the OSD related on this pg and
didn't find anything, no error from osd daemon, no errors from smartctl,
no
error from the kernel message.
So I just like to know if that's «normal» or should I scratch deeper.
JAS
--
Albert SHIH 🦫 🐸
France
Heure locale/Local time:
ven. 12 avril 2024 11:51:37 CEST
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:17 p.m.
Le 12/04/2024 à 12:56:12+0200, Frédéric Nass a écrit
Hi,
> Have you check the hardware status of the
involved drives other than with smartctl? Like with the manufacturer's tools / WebUI
(iDrac / perccli for DELL hardware for example).
Yes, all my disk are «under» periodic check with smartctl + icinga.
> If these tools don't report any media error (that is bad blocs on disks) then you
might just be facing the bit rot phenomenon. But this is very rare and should happen in a
sysadmin's lifetime as often as a Royal Flush hand in a professional poker
player's lifetime. ;-)
> If no media error is reported, then you
might want to check and update the firmware of all drives.
You're perfectly right.
It's just a newbie error, I check on the «main» osd of the PG (meaning the
first in the list) but forget to check on other.
On when server I indeed get some error on a disk.
But strangely smartctl report nothing. I will add a check with dmesg.
> Once you figured it out, you may enable
osd_scrub_auto_repair=true to have these inconsistencies repaired automatically on
deep-scrubbing, but make sure you're using the alert module [1] so to at least get
informed about the scrub errors.
Thanks. I will look into because we got already icinga2 on site so I use
icinga2 to check the cluster.
Is they are a list of what the alert module going to check ?
Regards
JAS
--
Albert SHIH 🦫 🐸
France
Heure locale/Local time:
ven. 12 avril 2024 15:13:13 CEST
2:24 p.m.
----- Le 12 Avr 24, à 15:17, Albert Shih Albert.Shih(a)obspm.fr a écrit :
Ok.
Basically the module checks for ceph status (ceph -s) changes.
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/alerts/module.py
Regards,
Frédéric.
>
>
> Regards
>
> JAS
> --
> Albert SHIH 🦫 🐸
> France
> Heure locale/Local time:
> ven. 12 avril 2024 15:13:13 CEST
Le 12/04/2024 à 12:56:12+0200, Frédéric Nass a écrit
Hi,
> Have you check the hardware status of
the involved drives other than with
> smartctl? Like with the manufacturer's tools / WebUI (iDrac / perccli for DELL
> hardware for example).
Yes, all my disk are «under» periodic check with smartctl + icinga.
Actually, I meant lower level tools (drive / server vendor tools).
If these tools don't report any media error
(that is bad blocs on disks) then
you might just be facing the bit rot phenomenon. But this is very rare and
should happen in a sysadmin's lifetime as often as a Royal Flush hand in a
professional poker player's lifetime. ;-)
If no media error is reported, then you might want to check and update the
firmware of all drives.
You're perfectly right.
It's just a newbie error, I check on the «main» osd of the PG (meaning the
first in the list) but forget to check on other.
On when server I indeed get some error on a disk.
But strangely smartctl report nothing. I will add a check with dmesg.
That's why I pointed you to the drive / server vendor tools earlier as sometimes
smartctl is missing the information you want.
Once you figured it out, you may enable osd_scrub_auto_repair=true to have these
inconsistencies repaired automatically on deep-scrubbing, but make sure you're
using the alert module [1] so to at least get informed about the scrub errors.
Thanks. I will look into because we got already icinga2 on site so I use
icinga2 to check the cluster.
Is they are a list of what the alert module going to check ? 
2:35 p.m.
If you're using an Icinga active check that just looks for
SMART overall-health self-assessment test result: PASSED
then it's not doing much for you. That bivalue status can be shown for a drive that
is decidedly an ex-parrot. Gotta look at specific attributes, which is thorny since they
aren't consistently implemented. drivedb.h is a downright mess, which doesn't
help.
----- Le 12 Avr 24, à 15:17, Albert Shih Albert.Shih(a)obspm.fr a écrit :
Ok.
Basically the module checks for ceph status (ceph -s) changes.
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/alerts/module.py
Regards,
Frédéric.
Le 12/04/2024 à 12:56:12+0200, Frédéric Nass a
écrit
Hi,
> Have you check the hardware status of
the involved drives other than with
> smartctl? Like with the manufacturer's tools / WebUI (iDrac / perccli for DELL
> hardware for example).
Yes, all my disk are «under» periodic check with smartctl + icinga.
Actually, I meant lower level tools (drive / server vendor tools).
If these tools don't report any media error
(that is bad blocs on disks) then
you might just be facing the bit rot phenomenon. But this is very rare and
should happen in a sysadmin's lifetime as often as a Royal Flush hand in a
professional poker player's lifetime. ;-)
If no media error is reported, then you might want to check and update the
firmware of all drives.
You're perfectly right.
It's just a newbie error, I check on the «main» osd of the PG (meaning the
first in the list) but forget to check on other.
On when server I indeed get some error on a
disk.
But strangely smartctl report nothing. I will add a check with dmesg.
That's why I pointed you to the drive / server vendor tools earlier as sometimes
smartctl is missing the information you want.
Once you figured it out, you may enable osd_scrub_auto_repair=true to have these
inconsistencies repaired automatically on deep-scrubbing, but make sure you're
using the alert module [1] so to at least get informed about the scrub errors.
Thanks. I will look into because we got already icinga2 on site so I use
icinga2 to check the cluster.
Is they are a list of what the alert module going to check ?
Regards
JAS
--
Albert SHIH 🦫 🐸
France
Heure locale/Local time:
ven. 12 avril 2024 15:13:13 CEST
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
20
days inactive
20
days old
5 comments
4 participants
participants (4)
-
Albert Shih -
Anthony D'Atri -
Frédéric Nass -
Wesley Dillingham