Hi, I'm trying to understand the "ceph health mute" behavior. In this case, I'm referring to the warning PG_NOT_DEEP_SCRUBBED. If you mute it for a week and the cluster continues deep-scrubbing, the "mute" will clear at some point although there are still PGs not deep-scrubbed in time warnings. I could verify this in a tiny lab with 19.2.2, setting osd_deep_scrub_interval to 10 minutes, the warning pops up. Then I mute that warning, issue deep-scrubs for several pools, and at some point I see this in the mon log: Jun 25 08:53:28 host1 ceph-mon[823315]: log_channel(cluster) log [WRN] : Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:28 host1 ceph-mon[823315]: Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:29 host1 ceph-mon[823315]: pgmap v164176: 389 pgs: 389 active+clean; 428 MiB data, 57 GiB used, 279 GiB / 336 GiB avail ... Jun 25 08:53:31 host1 ceph-mon[823315]: log_channel(cluster) log [INF] : Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) Jun 25 08:53:31 host1 ceph-mon[823315]: Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) I don't really understand what the code does [0] (I'm not a dev): ---snip--- if (!p->second.sticky) { auto q = all.checks.find(p->first); if (q == all.checks.end()) { mon.clog->info() << "Health alert mute " << p->first << " cleared (health alert cleared)"; p = pending_mutes.erase(p); changed = true; continue; } if (p->second.count) { // count-based mute if (q->second.count > p->second.count) { mon.clog->info() << "Health alert mute " << p->first << " cleared (count increased from " << p->second.count << " to " << q->second.count << ")"; p = pending_mutes.erase(p); changed = true; continue; ---snip--- Could anyone shed some light what I'm not understanding? Why would the mute clear although there are still PGs not deep-scrubbed? Thanks! Eugen [0] https://github.com/ceph/ceph/blob/d78ffd1247d6cef5cbd829e77204185dc0d3a8ba/s...
Hi Eugen, Reading the code, the muted alert was cleared because it was non-sticky and the number of affected PGs increased (which was decided to be a good reason to alert the admin). Have you tried to use the --sticky argument on the 'ceph health mute' command? Cheers, Frédéric. ----- Le 25 Juin 25, à 9:21, Eugen Block eblock@nde.ag a écrit :
Hi,
I'm trying to understand the "ceph health mute" behavior. In this case, I'm referring to the warning PG_NOT_DEEP_SCRUBBED. If you mute it for a week and the cluster continues deep-scrubbing, the "mute" will clear at some point although there are still PGs not deep-scrubbed in time warnings. I could verify this in a tiny lab with 19.2.2, setting osd_deep_scrub_interval to 10 minutes, the warning pops up. Then I mute that warning, issue deep-scrubs for several pools, and at some point I see this in the mon log:
Jun 25 08:53:28 host1 ceph-mon[823315]: log_channel(cluster) log [WRN] : Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:28 host1 ceph-mon[823315]: Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:29 host1 ceph-mon[823315]: pgmap v164176: 389 pgs: 389 active+clean; 428 MiB data, 57 GiB used, 279 GiB / 336 GiB avail ... Jun 25 08:53:31 host1 ceph-mon[823315]: log_channel(cluster) log [INF] : Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) Jun 25 08:53:31 host1 ceph-mon[823315]: Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61)
I don't really understand what the code does [0] (I'm not a dev):
---snip--- if (!p->second.sticky) { auto q = all.checks.find(p->first); if (q == all.checks.end()) { mon.clog->info() << "Health alert mute " << p->first << " cleared (health alert cleared)"; p = pending_mutes.erase(p); changed = true; continue; } if (p->second.count) { // count-based mute if (q->second.count > p->second.count) { mon.clog->info() << "Health alert mute " << p->first << " cleared (count increased from " << p->second.count << " to " << q->second.count << ")"; p = pending_mutes.erase(p); changed = true; continue; ---snip---
Could anyone shed some light what I'm not understanding? Why would the mute clear although there are still PGs not deep-scrubbed?
Thanks! Eugen
[0] https://github.com/ceph/ceph/blob/d78ffd1247d6cef5cbd829e77204185dc0d3a8ba/s...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Frédéric. The customer found the sticky flag, too. I must admit, I haven't used the mute command too often yet, usually I try to get to the bottom of a warning and rather fix the underlying issue. :-D So the mute clears if the number increases:
if (q->second.count > p->second.count)
That makes sense, and I agree that an admin might want to know about that. Then this is resolved for me, thanks for the quick response! Eugen Zitat von Frédéric Nass <frederic.nass@univ-lorraine.fr>:
Hi Eugen,
Reading the code, the muted alert was cleared because it was non-sticky and the number of affected PGs increased (which was decided to be a good reason to alert the admin).
Have you tried to use the --sticky argument on the 'ceph health mute' command?
Cheers, Frédéric.
----- Le 25 Juin 25, à 9:21, Eugen Block eblock@nde.ag a écrit :
Hi,
I'm trying to understand the "ceph health mute" behavior. In this case, I'm referring to the warning PG_NOT_DEEP_SCRUBBED. If you mute it for a week and the cluster continues deep-scrubbing, the "mute" will clear at some point although there are still PGs not deep-scrubbed in time warnings. I could verify this in a tiny lab with 19.2.2, setting osd_deep_scrub_interval to 10 minutes, the warning pops up. Then I mute that warning, issue deep-scrubs for several pools, and at some point I see this in the mon log:
Jun 25 08:53:28 host1 ceph-mon[823315]: log_channel(cluster) log [WRN] : Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:28 host1 ceph-mon[823315]: Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:29 host1 ceph-mon[823315]: pgmap v164176: 389 pgs: 389 active+clean; 428 MiB data, 57 GiB used, 279 GiB / 336 GiB avail ... Jun 25 08:53:31 host1 ceph-mon[823315]: log_channel(cluster) log [INF] : Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) Jun 25 08:53:31 host1 ceph-mon[823315]: Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61)
I don't really understand what the code does [0] (I'm not a dev):
---snip--- if (!p->second.sticky) { auto q = all.checks.find(p->first); if (q == all.checks.end()) { mon.clog->info() << "Health alert mute " << p->first << " cleared (health alert cleared)"; p = pending_mutes.erase(p); changed = true; continue; } if (p->second.count) { // count-based mute if (q->second.count > p->second.count) { mon.clog->info() << "Health alert mute " << p->first << " cleared (count increased from " << p->second.count << " to " << q->second.count << ")"; p = pending_mutes.erase(p); changed = true; continue; ---snip---
Could anyone shed some light what I'm not understanding? Why would the mute clear although there are still PGs not deep-scrubbed?
Thanks! Eugen
[0] https://github.com/ceph/ceph/blob/d78ffd1247d6cef5cbd829e77204185dc0d3a8ba/s...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Looks like I'm not alone in drop off scrub performance after last update? :) Łukasz Borek lukasz@borek.org.pl On Wed, 25 Jun 2025 at 11:58, Eugen Block <eblock@nde.ag> wrote:
Thanks Frédéric. The customer found the sticky flag, too. I must admit, I haven't used the mute command too often yet, usually I try to get to the bottom of a warning and rather fix the underlying issue. :-D So the mute clears if the number increases:
if (q->second.count > p->second.count)
That makes sense, and I agree that an admin might want to know about that. Then this is resolved for me, thanks for the quick response!
Eugen
Zitat von Frédéric Nass <frederic.nass@univ-lorraine.fr>:
Hi Eugen,
Reading the code, the muted alert was cleared because it was non-sticky and the number of affected PGs increased (which was decided to be a good reason to alert the admin).
Have you tried to use the --sticky argument on the 'ceph health mute' command?
Cheers, Frédéric.
----- Le 25 Juin 25, à 9:21, Eugen Block eblock@nde.ag a écrit :
Hi,
I'm trying to understand the "ceph health mute" behavior. In this case, I'm referring to the warning PG_NOT_DEEP_SCRUBBED. If you mute it for a week and the cluster continues deep-scrubbing, the "mute" will clear at some point although there are still PGs not deep-scrubbed in time warnings. I could verify this in a tiny lab with 19.2.2, setting osd_deep_scrub_interval to 10 minutes, the warning pops up. Then I mute that warning, issue deep-scrubs for several pools, and at some point I see this in the mon log:
Jun 25 08:53:28 host1 ceph-mon[823315]: log_channel(cluster) log [WRN] : Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:28 host1 ceph-mon[823315]: Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:29 host1 ceph-mon[823315]: pgmap v164176: 389 pgs: 389 active+clean; 428 MiB data, 57 GiB used, 279 GiB / 336 GiB avail ... Jun 25 08:53:31 host1 ceph-mon[823315]: log_channel(cluster) log [INF] : Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) Jun 25 08:53:31 host1 ceph-mon[823315]: Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61)
I don't really understand what the code does [0] (I'm not a dev):
---snip--- if (!p->second.sticky) { auto q = all.checks.find(p->first); if (q == all.checks.end()) { mon.clog->info() << "Health alert mute " << p->first << " cleared (health alert cleared)"; p = pending_mutes.erase(p); changed = true; continue; } if (p->second.count) { // count-based mute if (q->second.count > p->second.count) { mon.clog->info() << "Health alert mute " << p->first << " cleared (count increased from " << p->second.count << " to " << q->second.count << ")"; p = pending_mutes.erase(p); changed = true; continue; ---snip---
Could anyone shed some light what I'm not understanding? Why would the mute clear although there are still PGs not deep-scrubbed?
Thanks! Eugen
[0]
https://github.com/ceph/ceph/blob/d78ffd1247d6cef5cbd829e77204185dc0d3a8ba/s...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Actually, this is not the result of an upgrade but of two disk failures and the resulting backfill. The scrub performance is alright. :-) Zitat von Lukasz Borek <lukasz@borek.org.pl>:
Looks like I'm not alone in drop off scrub performance after last update? :)
Łukasz Borek lukasz@borek.org.pl
On Wed, 25 Jun 2025 at 11:58, Eugen Block <eblock@nde.ag> wrote:
Thanks Frédéric. The customer found the sticky flag, too. I must admit, I haven't used the mute command too often yet, usually I try to get to the bottom of a warning and rather fix the underlying issue. :-D So the mute clears if the number increases:
if (q->second.count > p->second.count)
That makes sense, and I agree that an admin might want to know about that. Then this is resolved for me, thanks for the quick response!
Eugen
Zitat von Frédéric Nass <frederic.nass@univ-lorraine.fr>:
Hi Eugen,
Reading the code, the muted alert was cleared because it was non-sticky and the number of affected PGs increased (which was decided to be a good reason to alert the admin).
Have you tried to use the --sticky argument on the 'ceph health mute' command?
Cheers, Frédéric.
----- Le 25 Juin 25, à 9:21, Eugen Block eblock@nde.ag a écrit :
Hi,
I'm trying to understand the "ceph health mute" behavior. In this case, I'm referring to the warning PG_NOT_DEEP_SCRUBBED. If you mute it for a week and the cluster continues deep-scrubbing, the "mute" will clear at some point although there are still PGs not deep-scrubbed in time warnings. I could verify this in a tiny lab with 19.2.2, setting osd_deep_scrub_interval to 10 minutes, the warning pops up. Then I mute that warning, issue deep-scrubs for several pools, and at some point I see this in the mon log:
Jun 25 08:53:28 host1 ceph-mon[823315]: log_channel(cluster) log [WRN] : Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:28 host1 ceph-mon[823315]: Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:29 host1 ceph-mon[823315]: pgmap v164176: 389 pgs: 389 active+clean; 428 MiB data, 57 GiB used, 279 GiB / 336 GiB avail ... Jun 25 08:53:31 host1 ceph-mon[823315]: log_channel(cluster) log [INF] : Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) Jun 25 08:53:31 host1 ceph-mon[823315]: Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61)
I don't really understand what the code does [0] (I'm not a dev):
---snip--- if (!p->second.sticky) { auto q = all.checks.find(p->first); if (q == all.checks.end()) { mon.clog->info() << "Health alert mute " << p->first << " cleared (health alert cleared)"; p = pending_mutes.erase(p); changed = true; continue; } if (p->second.count) { // count-based mute if (q->second.count > p->second.count) { mon.clog->info() << "Health alert mute " << p->first << " cleared (count increased from " << p->second.count << " to " << q->second.count << ")"; p = pending_mutes.erase(p); changed = true; continue; ---snip---
Could anyone shed some light what I'm not understanding? Why would the mute clear although there are still PGs not deep-scrubbed?
Thanks! Eugen
[0]
https://github.com/ceph/ceph/blob/d78ffd1247d6cef5cbd829e77204185dc0d3a8ba/s...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Jun 25, 2025, 3:54 PM Eugen Block <eblock@nde.ag> wrote:
Actually, this is not the result of an upgrade but of two disk failures and the resulting backfill. The scrub performance is alright. :-)
Zitat von Lukasz Borek <lukasz@borek.org.pl>:
Looks like I'm not alone in drop off scrub performance after last update? :)
Łukasz Borek lukasz@borek.org.pl
On Wed, 25 Jun 2025 at 11:58, Eugen Block <eblock@nde.ag> wrote:
Thanks Frédéric. The customer found the sticky flag, too. I must admit, I haven't used the mute command too often yet, usually I try to get to the bottom of a warning and rather fix the underlying issue. :-D So the mute clears if the number increases:
if (q->second.count > p->second.count)
That makes sense, and I agree that an admin might want to know about that. Then this is resolved for me, thanks for the quick response!
Eugen
Zitat von Frédéric Nass <frederic.nass@univ-lorraine.fr>:
Hi Eugen,
Reading the code, the muted alert was cleared because it was non-sticky and the number of affected PGs increased (which was decided to be a good reason to alert the admin).
Have you tried to use the --sticky argument on the 'ceph health mute' command?
Cheers, Frédéric.
----- Le 25 Juin 25, à 9:21, Eugen Block eblock@nde.ag a écrit :
Hi,
I'm trying to understand the "ceph health mute" behavior. In this case, I'm referring to the warning PG_NOT_DEEP_SCRUBBED. If you mute it for a week and the cluster continues deep-scrubbing, the "mute" will clear at some point although there are still PGs not deep-scrubbed in time warnings. I could verify this in a tiny lab with 19.2.2, setting osd_deep_scrub_interval to 10 minutes, the warning pops up. Then I mute that warning, issue deep-scrubs for several pools, and at some point I see this in the mon log:
Jun 25 08:53:28 host1 ceph-mon[823315]: log_channel(cluster) log [WRN] : Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:28 host1 ceph-mon[823315]: Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:29 host1 ceph-mon[823315]: pgmap v164176: 389 pgs: 389 active+clean; 428 MiB data, 57 GiB used, 279 GiB / 336 GiB avail ... Jun 25 08:53:31 host1 ceph-mon[823315]: log_channel(cluster) log [INF] : Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) Jun 25 08:53:31 host1 ceph-mon[823315]: Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61)
I don't really understand what the code does [0] (I'm not a dev):
---snip--- if (!p->second.sticky) { auto q = all.checks.find(p->first); if (q == all.checks.end()) { mon.clog->info() << "Health alert mute " << p->first << " cleared (health alert cleared)"; p = pending_mutes.erase(p); changed = true; continue; } if (p->second.count) { // count-based mute if (q->second.count > p->second.count) { mon.clog->info() << "Health alert mute " << p->first << " cleared (count increased from " << p->second.count << " to " << q->second.count << ")"; p = pending_mutes.erase(p); changed = true; continue; ---snip---
Could anyone shed some light what I'm not understanding? Why would the mute clear although there are still PGs not deep-scrubbed?
Thanks! Eugen
[0]
https://github.com/ceph/ceph/blob/d78ffd1247d6cef5cbd829e77204185dc0d3a8ba/s...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If you don't pass the --sticky flag to health mute, then the alert will be considered unmuted when the condition clears (e.g. if all PGs becomes scrubbed in a sufficient timespan). If you pass the --sticky flag, then the alarm remains muted irrespective of whether or not the condition clears at some point or not in the future until you explicitly unmute it. Cheers, Tyler
participants (4)
-
Eugen Block
-
Frédéric Nass
-
Lukasz Borek
-
Tyler Stachecki