[ceph-users] Unpurgeable rbd image from trash

Hello everyone, We have an unpurgeable image living in the trash of one of our clusters: # rbd --pool volumes trash ls 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b If we try to purge the whole trash it says the image is being restored but we have never tried to do that: # rbd --pool volumes trash purge Removing images: 0% complete...failed. 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: error: image is pending restoration. When trying to delete manually, it says there are some watchers, but this is actually not the case: # rbd --pool volumes trash remove 5afa5e5a07b8bc rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 -1 librbd::api::Trash: remove: error: image is pending restoration. This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. Removing image: 0% complete...failed. # rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc # We have tried to stat the first 10 rbd_data objects and they were all deleted. We know we can manually delete the omapkey from rbd_trash but we though it would be better to understand how an image might get in this state. Has anyone seen this before? Many thanks! Cheers, Enrico -- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland