[ceph-users] Re: Spurious Read Errors: 0x6706be76

Thanks, Igor. That's super helpful! I'm trying to temporarily disable the BLUESTORE_SPURIOUS_READ_ERRORS warning while we further investigate our underlying cause. In short: I'm trying to keep this cluster from going into HEALTH_WARN every day or so. I'm afraid that my attention to monitoring will lose sensitivity and I'll miss something more actionable (like a drive failure). https://docs.ceph.com/en/latest/rados/operations/health-checks/#bluestore-s… I found the above page in the documentation and ran "ceph config set osd bluestore_warn_on_spurious_read_errors false". However, I'm still seeing the cluster report on spurious read errors even after restarting all OSD, MON, and MGR services. # ceph config dump | grep spur osd advanced bluestore_warn_on_spurious_read_errors false Am I missing a step in how to fully apply this setting? Thanks! ~Jay -----Original Message----- From: Igor Fedotov &lt;ifedotov(a)suse.de&gt; Sent: Tuesday, June 22, 2021 7:46 AM To: Jay Sullivan &lt;jpspgd(a)rit.edu&gt;du>; ceph-users(a)ceph.io Subject: Re: [ceph-users] Spurious Read Errors: 0x6706be76 Hi Jay, this alert was introduced in Pacific indeed. That's probably why you haven't seen it before. And it definitely implies read retries, the following output mentions that explicitly: HEALTH_WARN 1 OSD(s) have spurious read errors [WRN] BLUESTORE_SPURIOUS_READ_ERRORS: 1 OSD(s) have spurious read errors osd.117 reads with retries: 1 "reads with retries" is actually a replica of "bluestore_reads_with_retries" perf counter at the corresponding OSD hence one can monitor it directly with "ceph daemon osd.N perf dump" command. Additionally one can increase "debug bluestore" log level to 5 to get relevant logging output in OSD log, here is the code line to print it:     dout(5) << __func__ << " read at 0x" << std::hex << offset << "~" << length             << " failed " << std::dec << retry_count << " times before succeeding" << dendl; Thanks, Igor On 6/22/2021 2:10 AM, Jay Sullivan wrote: