Permanently ignore some warning classes
Dear Ceph users, my cluster is build with old hardware on a gigabit network, so I often experience warnings like OSD_SLOW_PING_TIME_BACK. These in turn triggers alert mails too often, forcing me to disable alerts which is not sustainable. So my question is: is it possible to tell Ceph to ignore (or at least to not send alerts for) a given class of warnings? Thank you, Nicola
You should be able to do `ceph health mute OSD_SLOW_PING_TIME_BACK --sticky` to mute the health warn/error state as it flaps. You can also set a TTL for the mute (1d, 1w, 1m) to have it roll off after a specific time. Code here is the warning error such as OSD_SLOW_PING_TIME_BACK or POOL_TOO_FEW_PGS, etc. But the sticky option will mute the warning even after it clears and comes back.
Monitor commands: ================= health mute <code> [<ttl>] [--sticky] mute health alert
Reed
On Jan 11, 2023, at 6:57 AM, Nicola Mori <mori@fi.infn.it> wrote:
Dear Ceph users,
my cluster is build with old hardware on a gigabit network, so I often experience warnings like OSD_SLOW_PING_TIME_BACK. These in turn triggers alert mails too often, forcing me to disable alerts which is not sustainable. So my question is: is it possible to tell Ceph to ignore (or at least to not send alerts for) a given class of warnings? Thank you,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you Reed. I tried your solution but it didn't work, the warning emails are arriving anyway. Two possible reasons: 1) I issued `ceph health mute OSD_SLOW_PING_TIME_BACK --sticky` while the warning was not active, so it had no effect 2) according to this (https://people.redhat.com/bhubbard/nature/default/rados/operations/monitorin...): ``` Most health mutes also disappear if the extent of an alert gets worse. For example, if there is one OSD down, and the alert is muted, the mute will disappear if one or more additional OSDs go down. This is true for any health alert that involves a count indicating how much or how many of something is triggering the warning or error. ``` it could be that since OSD_SLOW_PING_TIME_BACK reports the number of osds that are experiencing slow heartbeats hen the mute is automatically removed. Do you have any insight on this?
Well, I guess the mute is now active: ``` # ceph health detail HEALTH_WARN 4 OSD(s) have spurious read errors; (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT) ``` but I still get emails from the alert module reporting about OSD_SLOW_PING_TIME_BACK/FRONT. Is this expected?
Yesterday I caught the cluster while OSD_SLOW_PING_TIME_FRONT was active: # ceph health detail HEALTH_WARN 4 slow ops, oldest one blocked for 9233 sec, daemons [mon.aka,mon.balin] have slow ops.; (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT) (MUTED, STICKY) [WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 1193.125ms) Slow OSD heartbeats on front from osd.72 [] to osd.43 [] 1193.125 msec [WRN] SLOW_OPS: 4 slow ops, oldest one blocked for 9233 sec, daemons [mon.aka,mon.balin] have slow ops. I really don't understand: the warning is muted, but it shows up anyway, and moreover with a MUTED label... and still I get warning emails.
I finally found the (hard) way to avoid receiving unwanted email alerts: I modified the alerts module in order to be able to specify the set of alert codes for which no notification is sent. If someone is interested I can share it, just let me know.
participants (2)
-
Nicola Mori
-
Reed Dier