Hi, Does this help? https://github.com/cernceph/ceph-scripts/blob/master/tools/scrubbing/autore… Cheers, Dan On Mon, May 25, 2020 at 5:18 PM Daniel Aberger - Profihost AG <d.aberger(a)profihost.ag> wrote: > > Hello, > > we are currently experiencing problems with ceph pg repair not working > on Ceph Nautilus 14.2.8. > > ceph health detail is showing us an inconsistent pg: > > [aaaaax-yyyy ~]# ceph health detail > HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent > OSD_SCRUB_ERRORS 1 scrub errors > PG_DAMAGED Possible data damage: 1 pg inconsistent > pg 18.19a is active+clean+inconsistent+snaptrim_wait, acting > [21,15,39,18,0,9] > > when we try to repair it, nothing happens. > > [aaaaax-yyyy ~]# ceph pg repair 18.19a > instructing pg 18.19as0 on osd.21 to repair > > There are no new entries in OSD 21's log file. > > We have no trouble repairing pgs in our other clusters so I assume it > might have to be something related to this cluster using Erasure > Codings. But this is just a wild guess. > > I found a similar problem in this mailing list - > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-April/026304.html > > Unfortunately the solution of waiting more than a week until it fixes > itself isn't quite satisfying. > > Is there anyone who has had similar issues and knows how to repair these > inconsistent pgs or what is causing the delay? > > > -- > Mit freundlichen Grüßen > Daniel Aberger > Ihr Profihost Team > > ------------------------------- > Profihost AG > Expo Plaza 1 > 30539 Hannover > Deutschland > > Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 > URL: http://www.profihost.com | E-Mail: info(a)profihost.com > > Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 > Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 > Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe > Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender) > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi, I'm not sure if the repair waits for snaptrim; but it does need a scrub reservation on all the related OSDs, hence our script. And I've also observed that the repair req isn't queued up -- if the OSDs are busy with other scrubs, the repair req is forgotten. -- Dan On Wed, May 27, 2020 at 11:28 AM Daniel Aberger - Profihost AG <d.aberger(a)profihost.ag> wrote: > > Hi, > > (un)fortunately I can't test it because I managed to repair the pg. > > snaptrim and snaptrim_wait have been a part of this particular pg's > status. As I was trying to look deeper into the case I had a watch on > ceph health detail and noticed that snaptrim/snaptrim_wait was suddenly > not a part of the status anymore. > > So I gave it another try with ceph pg repair 18.19a and suddenly the > pg's status changed to active+clean+inconsistent+repair. It repaired > successfully. > > Is snaptrim somehow blocking repair instructions? I would have thought > that repair instructions will be queued up until they can be performed > but it does not seem to work as I expected it to. > > Anyway I'll keep your script in mind and give it a shot if it happens > again. Thank you :) > > Daniel > > Am 25.05.20 um 17:40 schrieb Dan van der Ster: > > Hi, > > > > Does this help? > > > > https://github.com/cernceph/ceph-scripts/blob/master/tools/scrubbing/autore… > > > > Cheers, Dan > > > > On Mon, May 25, 2020 at 5:18 PM Daniel Aberger - Profihost AG > > <d.aberger(a)profihost.ag> wrote: > >> > >> Hello, > >> > >> we are currently experiencing problems with ceph pg repair not working > >> on Ceph Nautilus 14.2.8. > >> > >> ceph health detail is showing us an inconsistent pg: > >> > >> [aaaaax-yyyy ~]# ceph health detail > >> HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent > >> OSD_SCRUB_ERRORS 1 scrub errors > >> PG_DAMAGED Possible data damage: 1 pg inconsistent > >> pg 18.19a is active+clean+inconsistent+snaptrim_wait, acting > >> [21,15,39,18,0,9] > >> > >> when we try to repair it, nothing happens. > >> > >> [aaaaax-yyyy ~]# ceph pg repair 18.19a > >> instructing pg 18.19as0 on osd.21 to repair > >> > >> There are no new entries in OSD 21's log file. > >> > >> We have no trouble repairing pgs in our other clusters so I assume it > >> might have to be something related to this cluster using Erasure > >> Codings. But this is just a wild guess. > >> > >> I found a similar problem in this mailing list - > >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-April/026304.html > >> > >> Unfortunately the solution of waiting more than a week until it fixes > >> itself isn't quite satisfying. > >> > >> Is there anyone who has had similar issues and knows how to repair these > >> inconsistent pgs or what is causing the delay? > >> > >> > >> -- > >> Mit freundlichen Grüßen > >> Daniel Aberger > >> Ihr Profihost Team > >> > >> ------------------------------- > >> Profihost AG > >> Expo Plaza 1 > >> 30539 Hannover > >> Deutschland > >> > >> Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 > >> URL: http://www.profihost.com | E-Mail: info(a)profihost.com > >> > >> Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 > >> Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 > >> Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe > >> Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender) > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io