Re: RGW unable to delete a bucket
Hi Folks, I don't know of a downstream issue that looks like this, and we've upstreamed every fix for bucket listing and cleanup we have. We are pursuing a space leak believed to arise in "radosgw-admin bucket rm --purge-objects" but not a non-terminating listing. The only upstream release not planned to get a backport of orphans list tools is Luminous. I thought backport to Octopus was already done by the backport team? regards, Matt On Thu, Aug 6, 2020 at 2:40 PM EDH - Manuel Rios <mriosfer@easydatahost.com> wrote:
You'r not the only one affected by this issue
As far as i know several huge companies hitted this bug too, but private patches or tools are not public released.
This is caused for the a resharding process during upload in previous versions.
Workarround for us.:
- Delete objects of the bucket at rados level. - Delete the index file of the bucket.
Pray to god to not happen again.
Still pending backporting to Nautilus of the new experimental tool to find orphans in RGW
Maybe @Matt Benjamin can give us and ETA for get ready that tool backported...
Regards
-----Mensaje original----- De: Andrei Mikhailovsky <andrei@arhont.com> Enviado el: jueves, 6 de agosto de 2020 13:55 Para: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Re: RGW unable to delete a bucket
BUMP...
----- Original Message -----
From: "Andrei Mikhailovsky" <andrei@arhont.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, 4 August, 2020 17:16:28 Subject: [ceph-users] RGW unable to delete a bucket
Hi
I am trying to delete a bucket using the following command:
# radosgw-admin bucket rm --bucket=<bucket-name> --purge-objects
However, in console I get the following messages. About 100+ of those messages per second.
2020-08-04T17:11:06.411+0100 7fe64cacf080 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
The command has been running for about 35 days days and it still hasn't finished. The size of the bucket is under 1TB for sure. Probably around 500GB.
I have recently removed about a dozen of old buckets without any issues. It's this particular bucket that is being very stubborn.
Anything I can do to remove it, including it's objects and any orphans it might have?
Thanks
Andrei _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Hi Gents, thanks for your replies. I have stopped the removal command and reinitiated it again. To my surprise it has removed the bucket pretty quickly this time around. Not too sure if it has removed all the objects or orphans. I will need to further investigate this matter. Andrei ----- Original Message -----
From: "Matt Benjamin" <mbenjami@redhat.com> To: "EDH" <mriosfer@easydatahost.com> Cc: "Andrei Mikhailovsky" <andrei@arhont.com>, "ceph-users" <ceph-users@ceph.io>, "Ivancich, Eric" <ivancich@redhat.com> Sent: Thursday, 6 August, 2020 21:48:53 Subject: Re: RGW unable to delete a bucket
Hi Folks,
I don't know of a downstream issue that looks like this, and we've upstreamed every fix for bucket listing and cleanup we have. We are pursuing a space leak believed to arise in "radosgw-admin bucket rm --purge-objects" but not a non-terminating listing.
The only upstream release not planned to get a backport of orphans list tools is Luminous. I thought backport to Octopus was already done by the backport team?
regards,
Matt
On Thu, Aug 6, 2020 at 2:40 PM EDH - Manuel Rios <mriosfer@easydatahost.com> wrote:
You'r not the only one affected by this issue
As far as i know several huge companies hitted this bug too, but private patches or tools are not public released.
This is caused for the a resharding process during upload in previous versions.
Workarround for us.:
- Delete objects of the bucket at rados level. - Delete the index file of the bucket.
Pray to god to not happen again.
Still pending backporting to Nautilus of the new experimental tool to find orphans in RGW
Maybe @Matt Benjamin can give us and ETA for get ready that tool backported...
Regards
-----Mensaje original----- De: Andrei Mikhailovsky <andrei@arhont.com> Enviado el: jueves, 6 de agosto de 2020 13:55 Para: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Re: RGW unable to delete a bucket
BUMP...
----- Original Message -----
From: "Andrei Mikhailovsky" <andrei@arhont.com> To: "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, 4 August, 2020 17:16:28 Subject: [ceph-users] RGW unable to delete a bucket
Hi
I am trying to delete a bucket using the following command:
# radosgw-admin bucket rm --bucket=<bucket-name> --purge-objects
However, in console I get the following messages. About 100+ of those messages per second.
2020-08-04T17:11:06.411+0100 7fe64cacf080 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
The command has been running for about 35 days days and it still hasn't finished. The size of the bucket is under 1TB for sure. Probably around 500GB.
I have recently removed about a dozen of old buckets without any issues. It's this particular bucket that is being very stubborn.
Anything I can do to remove it, including it's objects and any orphans it might have?
Thanks
Andrei _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
participants (2)
-
Andrei Mikhailovsky
-
Matt Benjamin