Be cognizant of the offline compaction taking longer than mon_osd_down_out_interval and causing restarted OSDs to go "out" and recovery to begin before the compaction finishes. Prevent this with usage of noout flags or modifying the down out interval. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Wed, Jun 18, 2025 at 9:14 AM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Indeed, unset it after cycling though once.
On Jun 18, 2025, at 9:00 AM, Alexander Patrakov <patrakov@gmail.com> wrote:
Hello Michel,
I would definitely not recommend using osd_compact_on_start=true in production clusters, because cluster-wide power outages do happen, as well as "restarted the wrong OSD" mistakes, and this option extends the downtime by the amount of time it takes for all OSDs to compact.
On Wed, Jun 18, 2025 at 8:54 PM Michel Jouvin < michel.jouvin@ijclab.in2p3.fr> wrote:
Hi Anthony,
You mention the osd_compact_on_start option which is false by default. I checked quicly the documentation but this option seems under-documented: I found it mentioned only in the Pacific release notes when it was introduced.
Would you recommend to set it to true on production clusters? What are the possible side effects, in particular in terms of OSD start time?
Best regards,
Michel
Le 18/06/2025 à 14:01, Anthony D'Atri a écrit : It’s very early in the morning for me, but I kinda want to suggest OSD compaction, either “tell osd.1701 compact” online, or set the compact_on_start option and restart it.
On Jun 18, 2025, at 2:51 AM, Eugen Block <eblock@nde.ag> wrote:
Besides Michels response regarding the default of 24 hours after which the warning usually would disappear, I wanted to mention that we also saw this warning during some network issues we had. So if the disks seem okay, I'd recommend to check the network components.
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hello again,
I still have problem with occasional "slow operations in BlueStore" warning, which I don't know how to clear/acknowledge except restarting that OSD process (or doihg "ceph osd down/up" cycle).
When I previously asked about this several weeks ago, the underlying cause was HDD with SMART errors logged. I replaced that disk eventually, but now I am occasionally getting the same warning from different OSDs, which do not have any SMART errors or pending/reallocated sectors.
How can I see whether it was one-time problem and now it is OK, and how can I clear the warning manually or automatically after some time?
Thanks!
-Yenya
Anthony D'Atri wrote: > >> Hello, Ceph users, >> >> TL;DR: how to clear/acknowledge the following warning from ceph -s? >> >> health: HEALTH_WARN >> 1 OSD(s) experiencing slow operations in BlueStore >> >> Details: >> This is caused by a bad sector on the physical HDD - the time of this >> warning is the same as the most recent error in the SMART log of that HDD. > If this is the case, the drive might warrant replacement. > > A few grown defects are not entirely alarming, but if there are more than, say, 10 then I’d replace the drive. > > Or if there are SATA downshifts or other errors. > > >> # ceph health detail >> HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore >> [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore >> osd.33 observed slow operation indications in BlueStore >> >> When I restart osd.33 using "systemctl restart ceph-osd@33.service " >> on the OSD host, the warning disappears. But is there a "softer" way >> how to acknowledge this warning? > ceph osd down 33 > > does not touch the process, only the maps and peering, which often suffices. > >> Thanks! >> >> -Yenya >> >> -- >> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >> We all agree on the necessity of compromise. We just can't agree on >> when it's necessary to compromise. --Larry Wall >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io