How to clear the "slow operations" warning?
Hello, Ceph users, TL;DR: how to clear/acknowledge the following warning from ceph -s? health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD. # ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
If this is the case, the drive might warrant replacement. A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive. Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33 does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello again, I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle). When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors. How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time? Thanks! -Yenya Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall
Hi Jan, There is a parameter (I don't remember the name but it is in the documentation!) to define how long the warning must remain after the condition has been cleared. By default it is 24h. Cheers, Michel Sent from my mobile Le 18 juin 2025 08:36:23 Jan Kasprzak <kas@fi.muni.cz> a écrit :
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in
BlueStore
osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, This one? $ ceph config help bluestore_slow_ops_warn_lifetime bluestore_slow_ops_warn_lifetime - Set the time period during which a BlueStore slow ops warning will be raised when the `bluestore_slow_ops_warn_threshold` is exceeded. This is not the same as `osd_op_complaint_time`, which is about RADOS ops at the OSD level. (uint, advanced) Default: 86400 Can update at runtime: true See also: [bluestore_slow_ops_warn_threshold,osd_op_complaint_time] Found it here [1] as not in the documentation yet. Cheers, Frédéric. [1] https://github.com/ceph/ceph/blob/f484edf976c350b2f4b42fe15e0498fb30cc449a/s... ----- Le 18 Juin 25, à 8:43, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
Hi Jan,
There is a parameter (I don't remember the name but it is in the documentation!) to define how long the warning must remain after the condition has been cleared. By default it is 24h.
Cheers,
Michel Sent from my mobile Le 18 juin 2025 08:36:23 Jan Kasprzak <kas@fi.muni.cz> a écrit :
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in
BlueStore
osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think it's well explained on the docs: https://docs.ceph.com/en/reef/rados/operations/health-checks/#bluestore-slow... The parameters that modify that are these 2, which the defaults are a bit aggressive in my opinion: bluestore_slow_ops_warn_lifetime -> Interval to check (default 1 day) bluestore_slow_ops_warn_threshold -> number slow ops in the interval to trigger alert ( default 1) This means that 1 single slow op will maintain the alert for 1 day. To update: ceph config set global bluestore_slow_ops_warn_lifetime 10 ceph config set global bluestore_slow_ops_warn_threshold 5 To clean the alert, you can also reset bluestore_slow_ops_lifetime to any value so will restart the evaluation of the alert. ----------------------------------------------------------------- Roberto VALVERDE CAMESELLE IT Storage And Data Management CERN, European Organization for Nuclear Research Esplanade des Particules 1, Geneve (Switzerland) ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, June 18, 2025 11:58 AM To: Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> Cc: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: How to clear the "slow operations" warning? Hi, This one? $ ceph config help bluestore_slow_ops_warn_lifetime bluestore_slow_ops_warn_lifetime - Set the time period during which a BlueStore slow ops warning will be raised when the `bluestore_slow_ops_warn_threshold` is exceeded. This is not the same as `osd_op_complaint_time`, which is about RADOS ops at the OSD level. (uint, advanced) Default: 86400 Can update at runtime: true See also: [bluestore_slow_ops_warn_threshold,osd_op_complaint_time] Found it here [1] as not in the documentation yet. Cheers, Frédéric. [1] https://github.com/ceph/ceph/blob/f484edf976c350b2f4b42fe15e0498fb30cc449a/s... ----- Le 18 Juin 25, à 8:43, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
Hi Jan,
There is a parameter (I don't remember the name but it is in the documentation!) to define how long the warning must remain after the condition has been cleared. By default it is 24h.
Cheers,
Michel Sent from my mobile Le 18 juin 2025 08:36:23 Jan Kasprzak <kas@fi.muni.cz> a écrit :
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in
BlueStore
osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Besides Michels response regarding the default of 24 hours after which the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components. Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow
operations in BlueStore
osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net -
private}> |
| https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I should have said that we also experience this from time to time, each time a different OSD and not related to any problem logged on the server, for various reasons that are transient problem not related to the OSD itself. It is documented at https://docs.ceph.com/en/latest/rados/operations/health-checks/#bluestore-sl... (sorry I was not connected when replying previously) which states explicitely that there can be false positive. Personnally, I would not worry too much about this warning if you have it once, if it disappears after the warning lifetime and if it is not always affecting the same OSD or OSD server... Michel Le 18/06/2025 à 08:51, Eugen Block a écrit :
Besides Michels response regarding the default of 24 hours after which the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components.
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of
that HDD.
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow
operations in BlueStore
osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net -
private}> |
| https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It’s very early in the morning for me, but I kinda want to suggest OSD compaction, either “tell osd.1701 compact” online, or set the compact_on_start option and restart it.
On Jun 18, 2025, at 2:51 AM, Eugen Block <eblock@nde.ag> wrote:
Besides Michels response regarding the default of 24 hours after which the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components.
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, You mention the osd_compact_on_start option which is false by default. I checked quicly the documentation but this option seems under-documented: I found it mentioned only in the Pacific release notes when it was introduced. Would you recommend to set it to true on production clusters? What are the possible side effects, in particular in terms of OSD start time? Best regards, Michel Le 18/06/2025 à 14:01, Anthony D'Atri a écrit :
It’s very early in the morning for me, but I kinda want to suggest OSD compaction, either “tell osd.1701 compact” online, or set the compact_on_start option and restart it.
On Jun 18, 2025, at 2:51 AM, Eugen Block <eblock@nde.ag> wrote:
Besides Michels response regarding the default of 24 hours after which the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components.
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD. If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning? ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Michel, I would definitely not recommend using osd_compact_on_start=true in production clusters, because cluster-wide power outages do happen, as well as "restarted the wrong OSD" mistakes, and this option extends the downtime by the amount of time it takes for all OSDs to compact. On Wed, Jun 18, 2025 at 8:54 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
Hi Anthony,
You mention the osd_compact_on_start option which is false by default. I checked quicly the documentation but this option seems under-documented: I found it mentioned only in the Pacific release notes when it was introduced.
Would you recommend to set it to true on production clusters? What are the possible side effects, in particular in terms of OSD start time?
Best regards,
Michel
It’s very early in the morning for me, but I kinda want to suggest OSD compaction, either “tell osd.1701 compact” online, or set the compact_on_start option and restart it.
On Jun 18, 2025, at 2:51 AM, Eugen Block <eblock@nde.ag> wrote:
Besides Michels response regarding the default of 24 hours after which
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying
cause
was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more
Or if there are SATA downshifts or other errors.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations
in BlueStore
osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net -
| https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net -
Le 18/06/2025 à 14:01, Anthony D'Atri a écrit : the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components. that HDD. than, say, 10 then I’d replace the drive. private}> | private}> |
| https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
We keep it on in production, just for the convenience of keeping the OSDs compacted during maintenance. Startup time used to be a slight concern, but a recent release (Squid?) changed the RockdDB parameters, making startup with compaction several times faster than before. YMMV. On 18/06/2025 13:57, Alexander Patrakov wrote:
Hello Michel,
I would definitely not recommend using osd_compact_on_start=true in production clusters, because cluster-wide power outages do happen, as well as "restarted the wrong OSD" mistakes, and this option extends the downtime by the amount of time it takes for all OSDs to compact.
On Wed, Jun 18, 2025 at 8:54 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
Hi Anthony,
You mention the osd_compact_on_start option which is false by default. I checked quicly the documentation but this option seems under-documented: I found it mentioned only in the Pacific release notes when it was introduced.
Would you recommend to set it to true on production clusters? What are the possible side effects, in particular in terms of OSD start time?
Best regards,
Michel
It’s very early in the morning for me, but I kinda want to suggest OSD compaction, either “tell osd.1701 compact” online, or set the compact_on_start option and restart it.
On Jun 18, 2025, at 2:51 AM, Eugen Block <eblock@nde.ag> wrote:
Besides Michels response regarding the default of 24 hours after which
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
> Hello, Ceph users, > > TL;DR: how to clear/acknowledge the following warning from ceph -s? > > health: HEALTH_WARN > 1 OSD(s) experiencing slow operations in BlueStore > > Details: > This is caused by a bad sector on the physical HDD - the time of this > warning is the same as the most recent error in the SMART log of
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more
Or if there are SATA downshifts or other errors.
> # ceph health detail > HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore > [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore > osd.33 observed slow operation indications in BlueStore > > When I restart osd.33 using "systemctl restart ceph-osd@33.service" > on the OSD host, the warning disappears. But is there a "softer" way > how to acknowledge this warning? ceph osd down 33
does not touch the process, only the maps and peering, which often suffices. > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net -
> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > We all agree on the necessity of compromise. We just can't agree on > when it's necessary to compromise. --Larry Wall > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net -
Le 18/06/2025 à 14:01, Anthony D'Atri a écrit : the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components. that HDD. than, say, 10 then I’d replace the drive. private}> | private}> |
| https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Indeed, unset it after cycling though once.
On Jun 18, 2025, at 9:00 AM, Alexander Patrakov <patrakov@gmail.com> wrote:
Hello Michel,
I would definitely not recommend using osd_compact_on_start=true in production clusters, because cluster-wide power outages do happen, as well as "restarted the wrong OSD" mistakes, and this option extends the downtime by the amount of time it takes for all OSDs to compact.
On Wed, Jun 18, 2025 at 8:54 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
Hi Anthony,
You mention the osd_compact_on_start option which is false by default. I checked quicly the documentation but this option seems under-documented: I found it mentioned only in the Pacific release notes when it was introduced.
Would you recommend to set it to true on production clusters? What are the possible side effects, in particular in terms of OSD start time?
Best regards,
Michel
Le 18/06/2025 à 14:01, Anthony D'Atri a écrit : It’s very early in the morning for me, but I kinda want to suggest OSD compaction, either “tell osd.1701 compact” online, or set the compact_on_start option and restart it.
On Jun 18, 2025, at 2:51 AM, Eugen Block <eblock@nde.ag> wrote:
Besides Michels response regarding the default of 24 hours after which the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components.
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote:
> Hello, Ceph users, > > TL;DR: how to clear/acknowledge the following warning from ceph -s? > > health: HEALTH_WARN > 1 OSD(s) experiencing slow operations in BlueStore > > Details: > This is caused by a bad sector on the physical HDD - the time of this > warning is the same as the most recent error in the SMART log of
If this is the case, the drive might warrant replacement.
A few grown defects are not entirely alarming, but if there are more
that HDD. than, say, 10 then I’d replace the drive.
Or if there are SATA downshifts or other errors.
> # ceph health detail > HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore > [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations
> osd.33 observed slow operation indications in BlueStore > > When I restart osd.33 using "systemctl restart ceph-osd@33.service" > on the OSD host, the warning disappears. But is there a "softer" way > how to acknowledge this warning? ceph osd down 33
does not touch the process, only the maps and peering, which often suffices.
> Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net -
> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > We all agree on the necessity of compromise. We just can't agree on > when it's necessary to compromise. --Larry Wall > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net -
in BlueStore private}> | private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Be cognizant of the offline compaction taking longer than mon_osd_down_out_interval and causing restarted OSDs to go "out" and recovery to begin before the compaction finishes. Prevent this with usage of noout flags or modifying the down out interval. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Wed, Jun 18, 2025 at 9:14 AM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Indeed, unset it after cycling though once.
On Jun 18, 2025, at 9:00 AM, Alexander Patrakov <patrakov@gmail.com> wrote:
Hello Michel,
I would definitely not recommend using osd_compact_on_start=true in production clusters, because cluster-wide power outages do happen, as well as "restarted the wrong OSD" mistakes, and this option extends the downtime by the amount of time it takes for all OSDs to compact.
On Wed, Jun 18, 2025 at 8:54 PM Michel Jouvin < michel.jouvin@ijclab.in2p3.fr> wrote:
Hi Anthony,
You mention the osd_compact_on_start option which is false by default. I checked quicly the documentation but this option seems under-documented: I found it mentioned only in the Pacific release notes when it was introduced.
Would you recommend to set it to true on production clusters? What are the possible side effects, in particular in terms of OSD start time?
Best regards,
Michel
Le 18/06/2025 à 14:01, Anthony D'Atri a écrit : It’s very early in the morning for me, but I kinda want to suggest OSD compaction, either “tell osd.1701 compact” online, or set the compact_on_start option and restart it.
On Jun 18, 2025, at 2:51 AM, Eugen Block <eblock@nde.ag> wrote:
Besides Michels response regarding the default of 24 hours after which the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components.
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote: > >> Hello, Ceph users, >> >> TL;DR: how to clear/acknowledge the following warning from ceph -s? >> >> health: HEALTH_WARN >> 1 OSD(s) experiencing slow operations in BlueStore >> >> Details: >> This is caused by a bad sector on the physical HDD - the time of this >> warning is the same as the most recent error in the SMART log of that HDD. > If this is the case, the drive might warrant replacement. > > A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive. > > Or if there are SATA downshifts or other errors. > > >> # ceph health detail >> HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore >> [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore >> osd.33 observed slow operation indications in BlueStore >> >> When I restart osd.33 using "systemctl restart ceph-osd@33.service " >> on the OSD host, the warning disappears. But is there a "softer" way >> how to acknowledge this warning? > ceph osd down 33 > > does not touch the process, only the maps and peering, which often suffices. > >> Thanks! >> >> -Yenya >> >> -- >> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >> We all agree on the necessity of compromise. We just can't agree on >> when it's necessary to compromise. --Larry Wall >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Jan, Maybe ceph health mute is what you're looking for: ceph health mute BLUESTORE_SLOW_OP_ALERT 24h You could add '--sticky' at the end if you want to keep this alert muted even after the last occurence disappears. Regards, Frédéric. ----- Le 29 Mai 25, à 16:25, Jan Kasprzak kas@fi.muni.cz a écrit :
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm also struggling with those alerts after upgrading. Anyone was able to figure out which metric reflects the alert? From OSD log: root@/var/log/ceph/f3e63d9e-2f4c-11ef-87a2-0f1170f55ed5# cat ceph-osd.207.log | grep 'txc_state_proc slow aio_wait' 2025-06-18T01:10:09.902+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x558721570600, latency = 6.289815426s 2025-06-18T01:10:09.906+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x5586ea1fc900, latency = 6.285336018s 2025-06-18T01:10:09.910+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x5585988f2300, latency = 6.285313129s 2025-06-18T01:10:09.914+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x5585987a1200, latency = 6.287529469s 2025-06-18T01:10:09.922+0000 7fa677cdd640 0 bluestore(/var/lib/ceph/osd/ceph-207) _txc_state_proc slow aio_wait, txc = 0x558720399b00, latency = 6.305826664s Metrics with txc in name: ceph_bluestore_txc_commit_lat_sum{ceph_daemon="osd.207"} 3827.241650 ceph_bluestore_txc_submit_lat_sum{ceph_daemon="osd.207"} 547.903859 ceph_bluestore_txc_throttle_lat_sum{ceph_daemon="osd.207"} 49.293641 Should I summarize them to get values above 5 from the log? avg( rate(ceph_bluestore_txc_commit_lat_sum{job=~"$cluster",ceph_daemon='osd.208' }[1m]) / on (ceph_daemon) rate(ceph_bluestore_txc_commit_lat_count{job=~"$cluster",ceph_daemon='osd.208' }[1m]) * 1000 ) On Wed, 18 Jun 2025 at 12:05, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Jan,
Maybe ceph health mute is what you're looking for:
ceph health mute BLUESTORE_SLOW_OP_ALERT 24h
You could add '--sticky' at the end if you want to keep this alert muted even after the last occurence disappears.
Regards, Frédéric.
----- Le 29 Mai 25, à 16:25, Jan Kasprzak kas@fi.muni.cz a écrit :
Hello, Ceph users,
TL;DR: how to clear/acknowledge the following warning from ceph -s?
health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore
Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD.
# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore
When I restart osd.33 using "systemctl restart ceph-osd@33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning?
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Łukasz Borek lukasz@borek.org.pl
participants (11)
-
Alexander Patrakov
-
Anthony D'Atri
-
Anthony D'Atri
-
Chris Palmer
-
Eugen Block
-
Frédéric Nass
-
Jan Kasprzak
-
Lukasz Borek
-
Michel Jouvin
-
Roberto Valverde Cameselle
-
Wesley Dillingham