10 Mar
2021
10 Mar
'21
6:32 p.m.
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters:
# rbd --pool volumes trash ls
5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored
but we have never tried to do that:
# rbd --pool volumes trash purge
Removing images: 0% complete...failed.
2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove:
error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but
this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc
rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80
-1 librbd::api::Trash: remove: error: image is pending restoration.
This means the image is still open or the client using it crashed. Try
again after closing/unmapping it or waiting 30s for the crashed client
to timeout.
Removing image:
0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc
#
We have tried to stat the first 10 rbd_data objects and they were all
deleted.
We know we can manually delete the omapkey from rbd_trash but we though
it would be better to understand how an image might get in this state.
Has anyone seen this before?
Many thanks!
Cheers,
Enrico
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
10 Mar
10 Mar
7:01 p.m.
Can you provide the output from "rados -p volumes listomapvals rbd_trash"?
On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi(a)cern.ch> wrote:
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters:
# rbd --pool volumes trash ls
5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored
but we have never tried to do that:
# rbd --pool volumes trash purge
Removing images: 0% complete...failed.
2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove:
error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but
this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc
rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80
-1 librbd::api::Trash: remove: error: image is pending restoration.
This means the image is still open or the client using it crashed. Try
again after closing/unmapping it or waiting 30s for the crashed client
to timeout.
Removing image:
0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc
#
We have tried to stat the first 10 rbd_data objects and they were all
deleted.
We know we can manually delete the omapkey from rbd_trash but we though
it would be better to understand how an image might get in this state.
Has anyone seen this before?
Many thanks!
Cheers,
Enrico
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
7:07 p.m.
Hello Jason,
# rados -p volumes listomapvals rbd_trash
id_5afa5e5a07b8bc
value (71 bytes) :
00000000 02 01 41 00 00 00 00 2b 00 00 00 76 6f 6c 75 6d
|..A....+...volum|
00000010 65 2d 30 32 64 39 35 39 66 65 2d 61 36 39 33 2d
|e-02d959fe-a693-|
00000020 34 61 63 62 2d 39 35 65 32 2d 63 61 30 34 62 39
|4acb-95e2-ca04b9|
00000030 36 35 33 38 39 62 12 05 2a 60 09 c5 d4 16 12 05
|65389b..*`......|
00000040 2a 60 09 c5 d4 16 01 |*`.....|
00000047
Hope that helps.
Cheers,
Enrico
On 10/03/2021 14:31, Jason Dillaman wrote:
Can you provide the output from "rados -p volumes
listomapvals rbd_trash"?
On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi(a)cern.ch> wrote:
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters:
# rbd --pool volumes trash ls
5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored
but we have never tried to do that:
# rbd --pool volumes trash purge
Removing images: 0% complete...failed.
2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove:
error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but
this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc
rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80
-1 librbd::api::Trash: remove: error: image is pending restoration.
This means the image is still open or the client using it crashed. Try
again after closing/unmapping it or waiting 30s for the crashed client
to timeout.
Removing image:
0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc
#
We have tried to stat the first 10 rbd_data objects and they were all
deleted.
We know we can manually delete the omapkey from rbd_trash but we though
it would be better to understand how an image might get in this state.
Has anyone seen this before?
Many thanks!
Cheers,
Enrico
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:47 p.m.
Odd, it looks like it's stuck in the "MOVING" state. Perhaps the "rbd
trash mv" command was aborted mid-operation? The way to work around
this issue is as follows:
$ rados -p volumes getomapval rbd_trash id_5afa5e5a07b8bc key_file
$ hexedit key_file ## CHANGE LAST BYTE FROM '01' to '00'
$ rados -p volumes setomapval rbd_trash id_5afa5e5a07b8bc --input-file key_file
$ rbd trash rm --pool volumes 5afa5e5a07b8bc
I'll open a ticket to expose the ability to run "rbd trash mv
--image-id <XYZ>" to workaround an interrupted move operation in the
future and to cleanup the error text.
On Wed, Mar 10, 2021 at 8:37 AM Enrico Bocchi <enrico.bocchi(a)cern.ch> wrote:
Hello Jason,
# rados -p volumes listomapvals rbd_trash
id_5afa5e5a07b8bc
value (71 bytes) :
00000000 02 01 41 00 00 00 00 2b 00 00 00 76 6f 6c 75 6d
|..A....+...volum|
00000010 65 2d 30 32 64 39 35 39 66 65 2d 61 36 39 33 2d
|e-02d959fe-a693-|
00000020 34 61 63 62 2d 39 35 65 32 2d 63 61 30 34 62 39
|4acb-95e2-ca04b9|
00000030 36 35 33 38 39 62 12 05 2a 60 09 c5 d4 16 12 05
|65389b..*`......|
00000040 2a 60 09 c5 d4 16 01 |*`.....|
00000047
Hope that helps.
Cheers,
Enrico
On 10/03/2021 14:31, Jason Dillaman wrote:
--
Jason
Can you provide the output from "rados -p
volumes listomapvals rbd_trash"?
On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi(a)cern.ch> wrote:
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters:
# rbd --pool volumes trash ls
5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored
but we have never tried to do that:
# rbd --pool volumes trash purge
Removing images: 0% complete...failed.
2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove:
error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but
this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc
rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80
-1 librbd::api::Trash: remove: error: image is pending restoration.
This means the image is still open or the client using it crashed. Try
again after closing/unmapping it or waiting 30s for the crashed client
to timeout.
Removing image:
0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc
#
We have tried to stat the first 10 rbd_data objects and they were all
deleted.
We know we can manually delete the omapkey from rbd_trash but we though
it would be better to understand how an image might get in this state.
Has anyone seen this before?
Many thanks!
Cheers,
Enrico
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11 Mar
11 Mar
3:25 p.m.
Thanks a lot, that fixed the problem.
For the record, we are running nautilus 14.2.11 and there is no such
'--input-file' option for setomapval.
`#rados -p volumes setomapval rbd_trash id_5afa5e5a07b8bc < key_file`
does the trick.
Cheers,
Enrico
On 10/03/2021 17:17, Jason Dillaman wrote:
Odd, it looks like it's stuck in the
"MOVING" state. Perhaps the "rbd
trash mv" command was aborted mid-operation? The way to work around
this issue is as follows:
$ rados -p volumes getomapval rbd_trash id_5afa5e5a07b8bc key_file
$ hexedit key_file ## CHANGE LAST BYTE FROM '01' to '00'
$ rados -p volumes setomapval rbd_trash id_5afa5e5a07b8bc --input-file key_file
$ rbd trash rm --pool volumes 5afa5e5a07b8bc
I'll open a ticket to expose the ability to run "rbd trash mv
--image-id <XYZ>" to workaround an interrupted move operation in the
future and to cleanup the error text.
On Wed, Mar 10, 2021 at 8:37 AM Enrico Bocchi <enrico.bocchi(a)cern.ch> wrote:
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
Hello Jason,
# rados -p volumes listomapvals rbd_trash
id_5afa5e5a07b8bc
value (71 bytes) :
00000000 02 01 41 00 00 00 00 2b 00 00 00 76 6f 6c 75 6d
|..A....+...volum|
00000010 65 2d 30 32 64 39 35 39 66 65 2d 61 36 39 33 2d
|e-02d959fe-a693-|
00000020 34 61 63 62 2d 39 35 65 32 2d 63 61 30 34 62 39
|4acb-95e2-ca04b9|
00000030 36 35 33 38 39 62 12 05 2a 60 09 c5 d4 16 12 05
|65389b..*`......|
00000040 2a 60 09 c5 d4 16 01 |*`.....|
00000047
Hope that helps.
Cheers,
Enrico
On 10/03/2021 14:31, Jason Dillaman wrote:
Can you provide the output from "rados -p
volumes listomapvals rbd_trash"?
On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi(a)cern.ch> wrote:
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
Hello everyone,
We have an unpurgeable image living in the trash of one of our clusters:
# rbd --pool volumes trash ls
5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b
If we try to purge the whole trash it says the image is being restored
but we have never tried to do that:
# rbd --pool volumes trash purge
Removing images: 0% complete...failed.
2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove:
error: image is pending restoration.
When trying to delete manually, it says there are some watchers, but
this is actually not the case:
# rbd --pool volumes trash remove 5afa5e5a07b8bc
rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80
-1 librbd::api::Trash: remove: error: image is pending restoration.
This means the image is still open or the client using it crashed. Try
again after closing/unmapping it or waiting 30s for the crashed client
to timeout.
Removing image:
0% complete...failed.
# rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc
#
We have tried to stat the first 10 rbd_data objects and they were all
deleted.
We know we can manually delete the omapkey from rbd_trash but we though
it would be better to understand how an image might get in this state.
Has anyone seen this before?
Many thanks!
Cheers,
Enrico
--
Enrico Bocchi
CERN European Laboratory for Particle Physics
IT - Storage Group - General Storage Services
Mailbox: G20500 - Office: 31-2-010
1211 Genève 23
Switzerland
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1146
days inactive
1147
days old
4 comments
2 participants
participants (2)
-
Enrico Bocchi -
Jason Dillaman