Hi Chad, In case it's relevant we are on Nautilus 14.2.6, not Mimic. I've followed Paul's advice and issued a "ceph osd down XXX" command for the primary osd in each affected pg. I've also tried doing a systemctl restart for several of the primary osd's, again with no apparent effect. Unfortunately, we still have the same number of unfound objects. ceph health shows: HEALTH_ERR 23/435429429 objects unfound (0.000%); Possible data damage: 14 pgs recovery_unfound; Degraded data redundancy: 46/4349729009 objects degraded (0.000%), 14 pgs degraded; 9 pgs not deep-scrubbed in time; 9 pgs not scrubbed in time If the unfound items were caused by a faulty script of mine*, rather than a fault with ceph, then presumably the objects are lost forever? If so, can we identify which files are lost from the OID? And should we now clean up by running "ceph pg xxxxx mark_unfound_lost delete" on each affected pg? * Script was a "rolling upgrade" used after upgrading ceph, that should have restarted all osds on each node sequentially. Due to an error of mine, the script didn't wait for the all osds to *properly* start before moving on to the next node. "norecover" was set, which may have aggravated the situation. thanks again, Jake On 2/4/20 4:35 PM, Chad William Seys wrote: -- Jake Grimmett MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.