Ceph Snapshot Children not exists / children relation broken
Hello all, I`ve a problem with an undeletable Image. Let my try to explain. There is a storage with a Snapshot, and the snapshot thinks he has a Children: rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd' 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16 2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16rbd: unprotecting snap failed: (16) Device or resource busy rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error looking up name for image id f907bc6b8b4567 in pool rbd rbd: listing children failed: (2) No such file or directory So i am not able to unprotect and delete the Snapshot. Is there a way to fix this issue? I know this happens if deep-flatten is not enabled, and this is the root cause I think. We`ve activated this feature now but this s artifact from a time where it was disabled. Best regards & thx for your help. Torsten
On Fri, Jul 31, 2020 at 3:37 AM Torsten Ennenbach <tennenbach@gridscale.io> wrote:
Hello all,
I`ve a problem with an undeletable Image. Let my try to explain.
There is a storage with a Snapshot, and the snapshot thinks he has a Children:
rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd' 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16 2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16rbd: unprotecting snap failed: (16) Device or resource busy
rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error looking up name for image id f907bc6b8b4567 in pool rbd rbd: listing children failed: (2) No such file or directory
So i am not able to unprotect and delete the Snapshot. Is there a way to fix this issue?
Is the image in the trash? "rbd trash ls --all --long"
I know this happens if deep-flatten is not enabled, and this is the root cause I think. We`ve activated this feature now but this s artifact from a time where it was disabled.
Best regards & thx for your help.
Torsten _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Hey Jason Nope it is not (Anymore) Even if it is in the trash it can’t be deleted (same error like mentioned earlier) I tried to move this to trash as a solution, but this aint working also. Best regards Torsten
Am 31.07.2020 um 13:58 schrieb Jason Dillaman <jdillama@redhat.com>:
On Fri, Jul 31, 2020 at 3:37 AM Torsten Ennenbach <tennenbach@gridscale.io <mailto:tennenbach@gridscale.io>> wrote:
Hello all,
I`ve a problem with an undeletable Image. Let my try to explain.
There is a storage with a Snapshot, and the snapshot thinks he has a Children:
rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd' 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16 2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16rbd: unprotecting snap failed: (16) Device or resource busy
rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error looking up name for image id f907bc6b8b4567 in pool rbd rbd: listing children failed: (2) No such file or directory
So i am not able to unprotect and delete the Snapshot. Is there a way to fix this issue?
Is the image in the trash? "rbd trash ls --all --long"
I know this happens if deep-flatten is not enabled, and this is the root cause I think. We`ve activated this feature now but this s artifact from a time where it was disabled.
Best regards & thx for your help.
Torsten _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Jason
On Fri, Jul 31, 2020 at 8:03 AM Torsten Ennenbach <tennenbach@gridscale.io> wrote:
Hey Jason Nope it is not (Anymore)
Even if it is in the trash it can’t be deleted (same error like mentioned earlier)
I was asking about the child image (f907bc6b8b4567). What does "rados -p rbd listomapvals rbd_header.f907bc6b8b4567" return?
I tried to move this to trash as a solution, but this aint working also.
Best regards Torsten
Am 31.07.2020 um 13:58 schrieb Jason Dillaman <jdillama@redhat.com>:
On Fri, Jul 31, 2020 at 3:37 AM Torsten Ennenbach <tennenbach@gridscale.io <mailto:tennenbach@gridscale.io>> wrote:
Hello all,
I`ve a problem with an undeletable Image. Let my try to explain.
There is a storage with a Snapshot, and the snapshot thinks he has a Children:
rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd' 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16 2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16rbd: unprotecting snap failed: (16) Device or resource busy
rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error looking up name for image id f907bc6b8b4567 in pool rbd rbd: listing children failed: (2) No such file or directory
So i am not able to unprotect and delete the Snapshot. Is there a way to fix this issue?
Is the image in the trash? "rbd trash ls --all --long"
I know this happens if deep-flatten is not enabled, and this is the root cause I think. We`ve activated this feature now but this s artifact from a time where it was disabled.
Best regards & thx for your help.
Torsten _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Jason
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
On Fri, Jul 31, 2020 at 8:10 AM Torsten Ennenbach <tennenbach@gridscale.io> wrote:
Hi Jason
Am 31.07.2020 um 14:08 schrieb Jason Dillaman <jdillama@redhat.com>:
rados -p rbd listomapvals rbd_header.f907bc6b8b4567
rados -p rbd listomapvals rbd_header.f907bc6b8b4567 error getting omap keys rbd/rbd_header.f907bc6b8b4567: (2) No such file or directory
Ack. This can be fixed, but do you have any idea how the child image was removed? The removal process should delete its link to the parent before it's actually deleted (same w/ the flatten process). If there is some bug allowing images to slip through, I would like to fix it. First step to removing that link manually will be to get the image id of the "delete-me-please" image via "rbd info". You can then run the following to find the matching key entry for that image id (it's buried in a binary blob): (looking for 106e8126b1f2 in my example) $ rados -p rbd listomapvals rbd_children <..... snip ....> key (32 bytes): 00000000 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 |............106e| 00000010 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 |8126b1f2........| 00000020 value (19 bytes) : 00000000 01 00 00 00 0b 00 00 00 31 30 37 37 64 32 36 64 |........1077d26d| 00000010 38 66 64 |8fd| 00000013 < .... snip .... > You will then need to use hexedit (or something similar) to create a file w/ that exact binary key value that matches your image id. Below I've tweaked the hex from above key to match the xxd import format: $ cat <<EOF > key.txt
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e 00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........ EOF $ xxd -r -g 1 key.txt key.bin $ xxd -g 1 key.bin 00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e 00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
You can now provide that binary key file to rados to remove the offending key: $ rados -p rbd rmomapkey rbd_children --omap-key-file key.bin
Torsten _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Wow, Thx I will try this asap. Thats a … solution… Unfortunately I can’t, tell you how this is happened :( As far as I remember, this images was a cloned snapshot, without deep-flatten active as feature. Thx Jason. --
Am 31.07.2020 um 14:43 schrieb Jason Dillaman <jdillama@redhat.com>:
On Fri, Jul 31, 2020 at 8:10 AM Torsten Ennenbach <tennenbach@gridscale.io> wrote:
Hi Jason
Am 31.07.2020 um 14:08 schrieb Jason Dillaman <jdillama@redhat.com>:
rados -p rbd listomapvals rbd_header.f907bc6b8b4567
rados -p rbd listomapvals rbd_header.f907bc6b8b4567 error getting omap keys rbd/rbd_header.f907bc6b8b4567: (2) No such file or directory
Ack. This can be fixed, but do you have any idea how the child image was removed? The removal process should delete its link to the parent before it's actually deleted (same w/ the flatten process). If there is some bug allowing images to slip through, I would like to fix it.
First step to removing that link manually will be to get the image id of the "delete-me-please" image via "rbd info". You can then run the following to find the matching key entry for that image id (it's buried in a binary blob):
(looking for 106e8126b1f2 in my example) $ rados -p rbd listomapvals rbd_children <..... snip ....> key (32 bytes): 00000000 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 |............106e| 00000010 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 |8126b1f2........| 00000020
value (19 bytes) : 00000000 01 00 00 00 0b 00 00 00 31 30 37 37 64 32 36 64 |........1077d26d| 00000010 38 66 64 |8fd| 00000013 < .... snip .... >
You will then need to use hexedit (or something similar) to create a file w/ that exact binary key value that matches your image id. Below I've tweaked the hex from above key to match the xxd import format:
$ cat <<EOF > key.txt
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e 00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........ EOF $ xxd -r -g 1 key.txt key.bin $ xxd -g 1 key.bin 00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e 00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
You can now provide that binary key file to rados to remove the offending key:
$ rados -p rbd rmomapkey rbd_children --omap-key-file key.bin
Torsten _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Hi Jason. Well, I don't tried that, because I am afraid to break something :/ I don’t really understand what are you doing there :( Thanks anyways. Regards Torsten
Am 31.07.2020 um 16:46 schrieb Torsten Ennenbach <tennenbach@gridscale.io>:
Wow,
Thx I will try this asap. Thats a … solution… Unfortunately I can’t, tell you how this is happened :(
As far as I remember, this images was a cloned snapshot, without deep-flatten active as feature.
Thx Jason.
--
Am 31.07.2020 um 14:43 schrieb Jason Dillaman <jdillama@redhat.com <mailto:jdillama@redhat.com>>:
On Fri, Jul 31, 2020 at 8:10 AM Torsten Ennenbach <tennenbach@gridscale.io <mailto:tennenbach@gridscale.io>> wrote:
Hi Jason
Am 31.07.2020 um 14:08 schrieb Jason Dillaman <jdillama@redhat.com <mailto:jdillama@redhat.com>>:
rados -p rbd listomapvals rbd_header.f907bc6b8b4567
rados -p rbd listomapvals rbd_header.f907bc6b8b4567 error getting omap keys rbd/rbd_header.f907bc6b8b4567: (2) No such file or directory
Ack. This can be fixed, but do you have any idea how the child image was removed? The removal process should delete its link to the parent before it's actually deleted (same w/ the flatten process). If there is some bug allowing images to slip through, I would like to fix it.
First step to removing that link manually will be to get the image id of the "delete-me-please" image via "rbd info". You can then run the following to find the matching key entry for that image id (it's buried in a binary blob):
(looking for 106e8126b1f2 in my example) $ rados -p rbd listomapvals rbd_children <..... snip ....> key (32 bytes): 00000000 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 |............106e| 00000010 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 |8126b1f2........| 00000020
value (19 bytes) : 00000000 01 00 00 00 0b 00 00 00 31 30 37 37 64 32 36 64 |........1077d26d| 00000010 38 66 64 |8fd| 00000013 < .... snip .... >
You will then need to use hexedit (or something similar) to create a file w/ that exact binary key value that matches your image id. Below I've tweaked the hex from above key to match the xxd import format:
$ cat <<EOF > key.txt
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e 00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........ EOF $ xxd -r -g 1 key.txt key.bin $ xxd -g 1 key.bin 00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e 00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
You can now provide that binary key file to rados to remove the offending key:
$ rados -p rbd rmomapkey rbd_children --omap-key-file key.bin
Torsten _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Jason
On 8/3/20 2:07 PM, Torsten Ennenbach wrote:
Hi Jason.
Well, I don't tried that, because I am afraid to break something :/ I don’t really understand what are you doing there:(
Thanks anyways.
May be you catch this [1] bug? I have how-to solution [2] to resolve this, please try again. [1] https://tracker.ceph.com/issues/19413 [2] https://k0ste.ru/how-to-delete-rbd-snapshot-in-luminous-ceph-cluster-kraken-... k
participants (3)
-
Jason Dillaman
-
Konstantin Shalygin
-
Torsten Ennenbach