I'm also struggling with those alerts after upgrading. Anyone was able to figure out which metric reflects the alert? From OSD log: root@/var/log/ceph/f3e63d9e-2f4c-11ef-87a2-0f1170f55ed5# cat ceph-osd.207.log | grep 'txc_state_proc slow aio_wait' 2025-06-18T01:10:09.902+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x558721570600, latency = 6.289815426s 2025-06-18T01:10:09.906+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x5586ea1fc900, latency = 6.285336018s 2025-06-18T01:10:09.910+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x5585988f2300, latency = 6.285313129s 2025-06-18T01:10:09.914+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x5585987a1200, latency = 6.287529469s 2025-06-18T01:10:09.922+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x558720399b00, latency = 6.305826664s Metrics with txc in name: ceph_bluestore_txc_commit_lat_sum{ceph_daemon="osd.207"} 3827.241650 ceph_bluestore_txc_submit_lat_sum{ceph_daemon="osd.207"} 547.903859 ceph_bluestore_txc_throttle_lat_sum{ceph_daemon="osd.207"} 49.293641 Should I summarize them to get values above 5 from the log? avg( rate(ceph_bluestore_txc_commit_lat_sum{job=~"$cluster",ceph_daemon='osd.208' }[1m]) / on (ceph_daemon) rate(ceph_bluestore_txc_commit_lat_count{job=~"$cluster",ceph_daemon='osd.208' }[1m]) * 1000 ) On Wed, 18 Jun 2025 at 12:05, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Jan,
Maybe ceph health mute is what you're looking for:
ceph health mute BLUESTORE_SLOW_OP_ALERT 24h
You could add '--sticky' at the end if you want to keep this alert muted even after the last occurence disappears.
Regards, Frédéric.
----- Le 29 Mai 25, à 16:25, Jan Kasprzak kas@fi.muni.cz a écrit :
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Łukasz Borek lukasz@borek.org.pl