11 Jan
2023
11 Jan
'23
4:57 a.m.
Dear Ceph users,
my cluster is build with old hardware on a gigabit network, so I often
experience warnings like OSD_SLOW_PING_TIME_BACK. These in turn triggers
alert mails too often, forcing me to disable alerts which is not
sustainable. So my question is: is it possible to tell Ceph to ignore
(or at least to not send alerts for) a given class of warnings?
Thank you,
Nicola
30 Jan
30 Jan
10:27 a.m.
You should be able to do `ceph health mute OSD_SLOW_PING_TIME_BACK --sticky` to mute the
health warn/error state as it flaps.
You can also set a TTL for the mute (1d, 1w, 1m) to have it roll off after a specific
time.
Code here is the warning error such as OSD_SLOW_PING_TIME_BACK or POOL_TOO_FEW_PGS, etc.
But the sticky option will mute the warning even after it clears and comes back.
Monitor commands:
=================
health mute <code> [<ttl>] [--sticky] mute health alert
Reed
On Jan 11, 2023, at 6:57 AM, Nicola Mori
<mori(a)fi.infn.it> wrote:
Dear Ceph users,
my cluster is build with old hardware on a gigabit network, so I often experience
warnings like OSD_SLOW_PING_TIME_BACK. These in turn triggers alert mails too often,
forcing me to disable alerts which is not sustainable. So my question is: is it possible
to tell Ceph to ignore (or at least to not send alerts for) a given class of warnings?
Thank you,
Nicola
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Feb
2 Feb
12:51 a.m.
Thank you Reed. I tried your solution but it didn't work, the warning emails are
arriving anyway. Two possible reasons:
1) I issued `ceph health mute OSD_SLOW_PING_TIME_BACK --sticky` while the warning was not
active, so it had no effect
2) according to this
(https://people.redhat.com/bhubbard/nature/default/rados/operations/monitori…):
```
Most health mutes also disappear if the extent of an alert gets worse. For example, if
there is one OSD down, and the alert is muted, the mute will disappear if one or more
additional OSDs go down. This is true for any health alert that involves a count
indicating how much or how many of something is triggering the warning or error.
```
it could be that since OSD_SLOW_PING_TIME_BACK reports the number of osds that are
experiencing slow heartbeats hen the mute is automatically removed.
Do you have any insight on this?
5:57 a.m.
Well, I guess the mute is now active:
```
# ceph health detail
HEALTH_WARN 4 OSD(s) have spurious read errors; (muted: OSD_SLOW_PING_TIME_BACK
OSD_SLOW_PING_TIME_FRONT)
```
but I still get emails from the alert module reporting about
OSD_SLOW_PING_TIME_BACK/FRONT. Is this expected?
5 Feb
5 Feb
12:38 a.m.
Yesterday I caught the cluster while OSD_SLOW_PING_TIME_FRONT was active:
# ceph health detail
HEALTH_WARN 4 slow ops, oldest one blocked for 9233 sec, daemons [mon.aka,mon.balin] have
slow ops.; (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT)
(MUTED, STICKY) [WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest
1193.125ms)
Slow OSD heartbeats on front from osd.72 [] to osd.43 [] 1193.125 msec
[WRN] SLOW_OPS: 4 slow ops, oldest one blocked for 9233 sec, daemons [mon.aka,mon.balin]
have slow ops.
I really don't understand: the warning is muted, but it shows up anyway, and moreover
with a MUTED label... and still I get warning emails.
9 Feb
9 Feb
12:20 a.m.
I finally found the (hard) way to avoid receiving unwanted email alerts: I modified the
alerts module in order to be able to specify the set of alert codes for which no
notification is sent. If someone is interested I can share it, just let me know.
454
days inactive
483
days old
5 comments
2 participants
participants (2)
-
Nicola Mori -
Reed Dier