31 Jul
2020
31 Jul
'20
12:37 a.m.
Hello all,
I`ve a problem with an undeletable Image.
Let my try to explain.
There is a storage with a Snapshot, and the snapshot thinks he has a Children:
rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot
unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd'
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered
error: (16) Device or resource busy
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80
should_complete_error: ret_val=-16
2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80
should_complete_error: ret_val=-16rbd: unprotecting snap failed:
(16) Device or resource busy
rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09
2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error
looking up name for image id f907bc6b8b4567 in pool rbd
rbd: listing children failed: (2) No such file or directory
So i am not able to unprotect and delete the Snapshot.
Is there a way to fix this issue?
I know this happens if deep-flatten is not enabled, and this is the root cause I think.
We`ve activated this feature now but this s artifact from a time where it was disabled.
Best regards & thx for your help.
Torsten
31 Jul
31 Jul
4:58 a.m.
On Fri, Jul 31, 2020 at 3:37 AM Torsten Ennenbach
<tennenbach(a)gridscale.io> wrote:
Hello all,
I`ve a problem with an undeletable Image.
Let my try to explain.
There is a storage with a Snapshot, and the snapshot thinks he has a Children:
rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot
unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd'
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered
error: (16) Device or resource busy
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80
should_complete_error: ret_val=-16
2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80
should_complete_error: ret_val=-16rbd: unprotecting snap failed:
(16) Device or resource busy
rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09
2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error
looking up name for image id f907bc6b8b4567 in pool rbd
rbd: listing children failed: (2) No such file or directory
So i am not able to unprotect and delete the Snapshot.
Is there a way to fix this issue?
Is the image in the trash? "rbd trash ls --all --long"
I know this happens if deep-flatten is not enabled,
and this is the root cause I think. We`ve activated this feature now but this s artifact
from a time where it was disabled.
Best regards & thx for your help.
Torsten
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
5:03 a.m.
Hey Jason
Nope it is not (Anymore)
Even if it is in the trash it can’t be deleted (same error like mentioned earlier)
I tried to move this to trash as a solution, but this aint working also.
Best regards
Torsten
Am 31.07.2020 um 13:58 schrieb Jason Dillaman
<jdillama(a)redhat.com>om>:
On Fri, Jul 31, 2020 at 3:37 AM Torsten Ennenbach
<tennenbach(a)gridscale.io <mailto:tennenbach@gridscale.io>> wrote:
Hello all,
I`ve a problem with an undeletable Image.
Let my try to explain.
There is a storage with a Snapshot, and the snapshot thinks he has a Children:
rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot
unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd'
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered
error: (16) Device or resource busy
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80
should_complete_error: ret_val=-16
2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80
should_complete_error: ret_val=-16rbd: unprotecting snap failed:
(16) Device or resource busy
rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09
2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error
looking up name for image id f907bc6b8b4567 in pool rbd
rbd: listing children failed: (2) No such file or directory
So i am not able to unprotect and delete the Snapshot.
Is there a way to fix this issue?
Is the image in the trash? "rbd trash ls --all --long"
I know this happens if deep-flatten is not
enabled, and this is the root cause I think. We`ve activated this feature now but this s
artifact from a time where it was disabled.
Best regards & thx for your help.
Torsten
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
--
Jason
5:08 a.m.
On Fri, Jul 31, 2020 at 8:03 AM Torsten Ennenbach
<tennenbach(a)gridscale.io> wrote:
Hey Jason
Nope it is not (Anymore)
Even if it is in the trash it can’t be deleted (same error like mentioned earlier)
I was asking about the child image (f907bc6b8b4567). What does "rados
-p rbd listomapvals rbd_header.f907bc6b8b4567" return?
I tried to move this to trash as a solution, but this
aint working also.
Best regards
Torsten
--
Jason
Am 31.07.2020 um 13:58 schrieb Jason Dillaman
<jdillama(a)redhat.com>om>:
On Fri, Jul 31, 2020 at 3:37 AM Torsten Ennenbach
<tennenbach(a)gridscale.io <mailto:tennenbach@gridscale.io>> wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hello all,
I`ve a problem with an undeletable Image.
Let my try to explain.
There is a storage with a Snapshot, and the snapshot thinks he has a Children:
rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot
unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd'
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered
error: (16) Device or resource busy
2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80
should_complete_error: ret_val=-16
2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80
should_complete_error: ret_val=-16rbd: unprotecting snap failed:
(16) Device or resource busy
rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09
2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error
looking up name for image id f907bc6b8b4567 in pool rbd
rbd: listing children failed: (2) No such file or directory
So i am not able to unprotect and delete the Snapshot.
Is there a way to fix this issue?
Is the image in the trash? "rbd trash ls --all --long"
I know this happens if deep-flatten is not
enabled, and this is the root cause I think. We`ve activated this feature now but this s
artifact from a time where it was disabled.
Best regards & thx for your help.
Torsten
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
--
Jason
5:10 a.m.
Hi Jason
Am 31.07.2020 um 14:08 schrieb Jason Dillaman
<jdillama(a)redhat.com>om>:
rados
-p rbd listomapvals rbd_header.f907bc6b8b4567
rados -p rbd listomapvals rbd_header.f907bc6b8b4567
error getting omap keys rbd/rbd_header.f907bc6b8b4567: (2) No such file or directory
Torsten
5:43 a.m.
On Fri, Jul 31, 2020 at 8:10 AM Torsten Ennenbach
<tennenbach(a)gridscale.io> wrote:
Hi Jason
Ack. This can be fixed, but do you have any idea how the child image
was removed? The removal process should delete its link to the parent
before it's actually deleted (same w/ the flatten process). If there
is some bug allowing images to slip through, I would like to fix it.
First step to removing that link manually will be to get the image id
of the "delete-me-please" image via "rbd info". You can then run the
following to find the matching key entry for that image id (it's
buried in a binary blob):
(looking for 106e8126b1f2 in my example)
$ rados -p rbd listomapvals rbd_children
<..... snip ....>
key (32 bytes):
00000000 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 |............106e|
00000010 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 |8126b1f2........|
00000020
value (19 bytes) :
00000000 01 00 00 00 0b 00 00 00 31 30 37 37 64 32 36 64 |........1077d26d|
00000010 38 66 64 |8fd|
00000013
< .... snip .... >
You will then need to use hexedit (or something similar) to create a
file w/ that exact binary key value that matches your image id. Below
I've tweaked the hex from above key to match the xxd import format:
$ cat <<EOF > key.txt
Am 31.07.2020 um 14:08 schrieb Jason Dillaman
<jdillama(a)redhat.com>om>:
rados
-p rbd listomapvals rbd_header.f907bc6b8b4567
rados -p rbd listomapvals rbd_header.f907bc6b8b4567
error getting omap keys rbd/rbd_header.f907bc6b8b4567: (2) No such file or directory
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36
65 ............106e
00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
EOF
$ xxd -r -g 1 key.txt key.bin
$ xxd -g 1 key.bin
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e
00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
You can now provide that binary key file to rados to remove the offending key:
$ rados -p rbd rmomapkey rbd_children --omap-key-file key.bin
Torsten
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
7:46 a.m.
Wow,
Thx I will try this asap. Thats a … solution…
Unfortunately I can’t, tell you how this is happened :(
As far as I remember, this images was a cloned snapshot, without deep-flatten active as
feature.
Thx Jason.
--
Am 31.07.2020 um 14:43 schrieb Jason Dillaman
<jdillama(a)redhat.com>om>:
On Fri, Jul 31, 2020 at 8:10 AM Torsten Ennenbach
<tennenbach(a)gridscale.io> wrote:
Hi Jason
Ack. This can be fixed, but do you have any idea how the child image
was removed? The removal process should delete its link to the parent
before it's actually deleted (same w/ the flatten process). If there
is some bug allowing images to slip through, I would like to fix it.
First step to removing that link manually will be to get the image id
of the "delete-me-please" image via "rbd info". You can then run the
following to find the matching key entry for that image id (it's
buried in a binary blob):
(looking for 106e8126b1f2 in my example)
$ rados -p rbd listomapvals rbd_children
<..... snip ....>
key (32 bytes):
00000000 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 |............106e|
00000010 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 |8126b1f2........|
00000020
value (19 bytes) :
00000000 01 00 00 00 0b 00 00 00 31 30 37 37 64 32 36 64 |........1077d26d|
00000010 38 66 64 |8fd|
00000013
< .... snip .... >
You will then need to use hexedit (or something similar) to create a
file w/ that exact binary key value that matches your image id. Below
I've tweaked the hex from above key to match the xxd import format:
$ cat <<EOF > key.txt
Am 31.07.2020 um 14:08 schrieb Jason Dillaman
<jdillama(a)redhat.com>om>:
rados
-p rbd listomapvals rbd_header.f907bc6b8b4567
rados -p rbd listomapvals rbd_header.f907bc6b8b4567
error getting omap keys rbd/rbd_header.f907bc6b8b4567: (2) No such file or directory
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31
30 36 65 ............106e
00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
EOF
$ xxd -r -g 1 key.txt key.bin
$ xxd -g 1 key.bin
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e
00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
You can now provide that binary key file to rados to remove the offending key:
$ rados -p rbd rmomapkey rbd_children --omap-key-file key.bin
Torsten
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
3 Aug
3 Aug
12:07 a.m.
Hi Jason.
Well, I don't tried that, because I am afraid to break something :/ I don’t really
understand what are you doing there :(
Thanks anyways.
Regards
Torsten
Am 31.07.2020 um 16:46 schrieb Torsten Ennenbach
<tennenbach(a)gridscale.io>io>:
Wow,
Thx I will try this asap. Thats a … solution…
Unfortunately I can’t, tell you how this is happened :(
As far as I remember, this images was a cloned snapshot, without deep-flatten active as
feature.
Thx Jason.
--
Am 31.07.2020 um 14:43 schrieb Jason Dillaman
<jdillama(a)redhat.com <mailto:jdillama@redhat.com>>:
On Fri, Jul 31, 2020 at 8:10 AM Torsten Ennenbach
<tennenbach(a)gridscale.io <mailto:tennenbach@gridscale.io>> wrote:
Hi Jason
Ack. This can be fixed, but do you have any idea how the child image
was removed? The removal process should delete its link to the parent
before it's actually deleted (same w/ the flatten process). If there
is some bug allowing images to slip through, I would like to fix it.
First step to removing that link manually will be to get the image id
of the "delete-me-please" image via "rbd info". You can then run the
following to find the matching key entry for that image id (it's
buried in a binary blob):
(looking for 106e8126b1f2 in my example)
$ rados -p rbd listomapvals rbd_children
<..... snip ....>
key (32 bytes):
00000000 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 |............106e|
00000010 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 |8126b1f2........|
00000020
value (19 bytes) :
00000000 01 00 00 00 0b 00 00 00 31 30 37 37 64 32 36 64 |........1077d26d|
00000010 38 66 64 |8fd|
00000013
< .... snip .... >
You will then need to use hexedit (or something similar) to create a
file w/ that exact binary key value that matches your image id. Below
I've tweaked the hex from above key to match the xxd import format:
$ cat <<EOF > key.txt
Am 31.07.2020 um 14:08 schrieb Jason Dillaman
<jdillama(a)redhat.com <mailto:jdillama@redhat.com>>:
rados
-p rbd listomapvals rbd_header.f907bc6b8b4567
rados -p rbd listomapvals rbd_header.f907bc6b8b4567
error getting omap keys rbd/rbd_header.f907bc6b8b4567: (2) No such file or directory
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31
30 36 65 ............106e
00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
EOF
$ xxd -r -g 1 key.txt key.bin
$ xxd -g 1 key.bin
00000000: 02 00 00 00 00 00 00 00 0c 00 00 00 31 30 36 65 ............106e
00000010: 38 31 32 36 62 31 66 32 04 00 00 00 00 00 00 00 8126b1f2........
You can now provide that binary key file to rados to remove the offending key:
$ rados -p rbd rmomapkey rbd_children --omap-key-file key.bin
Torsten
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
--
Jason
20 Aug
20 Aug
1:19 a.m.
On 8/3/20 2:07 PM, Torsten Ennenbach wrote:
Hi Jason.
Well, I don't tried that, because I am afraid to break something :/ I don’t really
understand what are you doing there:(
Thanks anyways.
May be you catch this [1] bug? I have how-to solution [2] to resolve
this, please try again.
[1] https://tracker.ceph.com/issues/19413
[2]
https://k0ste.ru/how-to-delete-rbd-snapshot-in-luminous-ceph-cluster-kraken…
k
1357
days inactive
1377
days old
8 comments
3 participants
participants (3)
-
Jason Dillaman -
Konstantin Shalygin -
Torsten Ennenbach