We experienced one week ago problems with a dying OSD resulting in OSD_TOO_MANY_REPAIRS, unfortunately unnoticed (it seems our monitoring system is not notifying these errors properly). When we realized the error, we removed the problematic OSD (ceph orch osd rm --replace), despite the scrubbing errors: the resulting backfills succeeded but did not fix the scrub errors. The collegue who took care of this problem decided to lauch a `ceph pg repair` on the 3 PGs with reported inconsistencies but it doesn't seem to converge. 'ceph -s' still reports :
3 active+clean+scrubbing+deep+inconsistent+repair
after a few hours and for at least one of the PG, there is the following message every 3s:
Sep 18 12:30:55 ceph-76212 ceph-mon[2506]: osd.72 pg 11.e2d Deep scrub errors, upgrading scrub to deep-scrub
Not sure if it is the sign of a problem or just because the operation is ongoing. I'm looking for advices on what to do to move forward. There was not yet a report from users of an impact of this but it doesn't mean there is none... The affected pool is storing RBD volumes (from OpenStack Cinder).
Can't say how long it should take, but repairs can take a while. For me, it's usually takes a long while until the "active+clean+scrubbing+deep+inconsistent+repair" status appears, then I guess it is dependant on disk (and possibly wpq -vs- mclock?) perf. I would stay calm for a while to let the cluster try to get itself right. -- May the most significant bit of your life be positive.