Unpurgeable rbd image from trash
Hello everyone, We have an unpurgeable image living in the trash of one of our clusters: # rbd --pool volumes trash ls 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b If we try to purge the whole trash it says the image is being restored but we have never tried to do that: # rbd --pool volumes trash purge Removing images: 0% complete...failed. 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: error: image is pending restoration. When trying to delete manually, it says there are some watchers, but this is actually not the case: # rbd --pool volumes trash remove 5afa5e5a07b8bc rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 -1 librbd::api::Trash: remove: error: image is pending restoration. This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. Removing image: 0% complete...failed. # rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc # We have tried to stat the first 10 rbd_data objects and they were all deleted. We know we can manually delete the omapkey from rbd_trash but we though it would be better to understand how an image might get in this state. Has anyone seen this before? Many thanks! Cheers, Enrico -- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland
Can you provide the output from "rados -p volumes listomapvals rbd_trash"? On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi@cern.ch> wrote:
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters: # rbd --pool volumes trash ls 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored but we have never tried to do that: # rbd --pool volumes trash purge Removing images: 0% complete...failed. 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 -1 librbd::api::Trash: remove: error: image is pending restoration. This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. Removing image: 0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc #
We have tried to stat the first 10 rbd_data objects and they were all deleted. We know we can manually delete the omapkey from rbd_trash but we though it would be better to understand how an image might get in this state. Has anyone seen this before?
Many thanks! Cheers, Enrico
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Hello Jason, # rados -p volumes listomapvals rbd_trash id_5afa5e5a07b8bc value (71 bytes) : 00000000 02 01 41 00 00 00 00 2b 00 00 00 76 6f 6c 75 6d |..A....+...volum| 00000010 65 2d 30 32 64 39 35 39 66 65 2d 61 36 39 33 2d |e-02d959fe-a693-| 00000020 34 61 63 62 2d 39 35 65 32 2d 63 61 30 34 62 39 |4acb-95e2-ca04b9| 00000030 36 35 33 38 39 62 12 05 2a 60 09 c5 d4 16 12 05 |65389b..*`......| 00000040 2a 60 09 c5 d4 16 01 |*`.....| 00000047 Hope that helps. Cheers, Enrico On 10/03/2021 14:31, Jason Dillaman wrote:
Can you provide the output from "rados -p volumes listomapvals rbd_trash"?
On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi@cern.ch> wrote:
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters: # rbd --pool volumes trash ls 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored but we have never tried to do that: # rbd --pool volumes trash purge Removing images: 0% complete...failed. 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 -1 librbd::api::Trash: remove: error: image is pending restoration. This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. Removing image: 0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc #
We have tried to stat the first 10 rbd_data objects and they were all deleted. We know we can manually delete the omapkey from rbd_trash but we though it would be better to understand how an image might get in this state. Has anyone seen this before?
Many thanks! Cheers, Enrico
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland
Odd, it looks like it's stuck in the "MOVING" state. Perhaps the "rbd trash mv" command was aborted mid-operation? The way to work around this issue is as follows: $ rados -p volumes getomapval rbd_trash id_5afa5e5a07b8bc key_file $ hexedit key_file ## CHANGE LAST BYTE FROM '01' to '00' $ rados -p volumes setomapval rbd_trash id_5afa5e5a07b8bc --input-file key_file $ rbd trash rm --pool volumes 5afa5e5a07b8bc I'll open a ticket to expose the ability to run "rbd trash mv --image-id <XYZ>" to workaround an interrupted move operation in the future and to cleanup the error text. On Wed, Mar 10, 2021 at 8:37 AM Enrico Bocchi <enrico.bocchi@cern.ch> wrote:
Hello Jason,
# rados -p volumes listomapvals rbd_trash id_5afa5e5a07b8bc value (71 bytes) : 00000000 02 01 41 00 00 00 00 2b 00 00 00 76 6f 6c 75 6d |..A....+...volum| 00000010 65 2d 30 32 64 39 35 39 66 65 2d 61 36 39 33 2d |e-02d959fe-a693-| 00000020 34 61 63 62 2d 39 35 65 32 2d 63 61 30 34 62 39 |4acb-95e2-ca04b9| 00000030 36 35 33 38 39 62 12 05 2a 60 09 c5 d4 16 12 05 |65389b..*`......| 00000040 2a 60 09 c5 d4 16 01 |*`.....| 00000047
Hope that helps. Cheers, Enrico
On 10/03/2021 14:31, Jason Dillaman wrote:
Can you provide the output from "rados -p volumes listomapvals rbd_trash"?
On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi@cern.ch> wrote:
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters: # rbd --pool volumes trash ls 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored but we have never tried to do that: # rbd --pool volumes trash purge Removing images: 0% complete...failed. 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 -1 librbd::api::Trash: remove: error: image is pending restoration. This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. Removing image: 0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc #
We have tried to stat the first 10 rbd_data objects and they were all deleted. We know we can manually delete the omapkey from rbd_trash but we though it would be better to understand how an image might get in this state. Has anyone seen this before?
Many thanks! Cheers, Enrico
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland
-- Jason
Thanks a lot, that fixed the problem. For the record, we are running nautilus 14.2.11 and there is no such '--input-file' option for setomapval. `#rados -p volumes setomapval rbd_trash id_5afa5e5a07b8bc < key_file` does the trick. Cheers, Enrico On 10/03/2021 17:17, Jason Dillaman wrote:
Odd, it looks like it's stuck in the "MOVING" state. Perhaps the "rbd trash mv" command was aborted mid-operation? The way to work around this issue is as follows:
$ rados -p volumes getomapval rbd_trash id_5afa5e5a07b8bc key_file $ hexedit key_file ## CHANGE LAST BYTE FROM '01' to '00' $ rados -p volumes setomapval rbd_trash id_5afa5e5a07b8bc --input-file key_file $ rbd trash rm --pool volumes 5afa5e5a07b8bc
I'll open a ticket to expose the ability to run "rbd trash mv --image-id <XYZ>" to workaround an interrupted move operation in the future and to cleanup the error text.
On Wed, Mar 10, 2021 at 8:37 AM Enrico Bocchi <enrico.bocchi@cern.ch> wrote:
Hello Jason,
# rados -p volumes listomapvals rbd_trash id_5afa5e5a07b8bc value (71 bytes) : 00000000 02 01 41 00 00 00 00 2b 00 00 00 76 6f 6c 75 6d |..A....+...volum| 00000010 65 2d 30 32 64 39 35 39 66 65 2d 61 36 39 33 2d |e-02d959fe-a693-| 00000020 34 61 63 62 2d 39 35 65 32 2d 63 61 30 34 62 39 |4acb-95e2-ca04b9| 00000030 36 35 33 38 39 62 12 05 2a 60 09 c5 d4 16 12 05 |65389b..*`......| 00000040 2a 60 09 c5 d4 16 01 |*`.....| 00000047
Hope that helps. Cheers, Enrico
On 10/03/2021 14:31, Jason Dillaman wrote:
Can you provide the output from "rados -p volumes listomapvals rbd_trash"?
On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi@cern.ch> wrote:
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters: # rbd --pool volumes trash ls 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored but we have never tried to do that: # rbd --pool volumes trash purge Removing images: 0% complete...failed. 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 -1 librbd::api::Trash: remove: error: image is pending restoration. This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. Removing image: 0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc #
We have tried to stat the first 10 rbd_data objects and they were all deleted. We know we can manually delete the omapkey from rbd_trash but we though it would be better to understand how an image might get in this state. Has anyone seen this before?
Many thanks! Cheers, Enrico
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland
participants (2)
-
Enrico Bocchi
-
Jason Dillaman