Undo "radosgw-admin bi purge"
Hi, There is an operation "radosgw-admin bi purge" that removes all bucket index objects for one bucket in the rados gateway. What is the undo operation for this? After this operation the bucket cannot be listed or removed any more. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
When the admin runs “bi purge” they have the option of supplying a bucket_id with the “--bucket-id” command-line argument. This was useful back when resharding did not automatically remove the older bucket index shards (which it now does), which had a different bucket_id from the current bucket index shards. If the admin doesn't supply a bucket_id it assumes the current bucket index shards are to be purged. Because this is generally not wanted, the admin is required to supply the "--yes-i-really-mean-it” command-line argument to verify that they know that this is generally not done. There is no “undo” for “bi purge”, because it removes the metadata objects that contain the bucket listing. The objects should still be readable if you know their names.
After this operation the bucket cannot be listed or removed any more.
Knowing the above, what is your goal? Removal/clean-up? Recovery of as much as possible? Both are possible to a degree (not 100%) but the processes are not simple and highly manual. Eric (he/him)
On Feb 20, 2023, at 10:01 AM, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
There is an operation "radosgw-admin bi purge" that removes all bucket index objects for one bucket in the rados gateway.
What is the undo operation for this?
After this operation the bucket cannot be listed or removed any more.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Robert, A colleague and I ran into this a few weeks ago. The way we managed to get access back to delete the bucket properly (using radosgw-admin bucket rm) was to reshard the bucket. This created a new bucket index and therefore it was then possible to delete it. If you are looking to get access back to the objects, then as Eric said there's no way to get those indexes back but the objects will still be there in the pool. sudo radosgw-admin bucket reshard --bucket={bucket_name} --num-shards {number} If you are doing multi-site replication, resharding can cause some issues on earlier versions of ceph so check that out if that applies to you. If you're just going to delete the bucket anyway it may not be an issue. Regards, Richard On Wed, 22 Feb 2023 at 07:43, J. Eric Ivancich <ivancich@redhat.com> wrote:
When the admin runs “bi purge” they have the option of supplying a bucket_id with the “--bucket-id” command-line argument. This was useful back when resharding did not automatically remove the older bucket index shards (which it now does), which had a different bucket_id from the current bucket index shards.
If the admin doesn't supply a bucket_id it assumes the current bucket index shards are to be purged. Because this is generally not wanted, the admin is required to supply the "--yes-i-really-mean-it” command-line argument to verify that they know that this is generally not done.
There is no “undo” for “bi purge”, because it removes the metadata objects that contain the bucket listing.
The objects should still be readable if you know their names.
After this operation the bucket cannot be listed or removed any more.
Knowing the above, what is your goal? Removal/clean-up? Recovery of as much as possible? Both are possible to a degree (not 100%) but the processes are not simple and highly manual.
Eric (he/him)
On Feb 20, 2023, at 10:01 AM, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
There is an operation "radosgw-admin bi purge" that removes all bucket index objects for one bucket in the rados gateway.
What is the undo operation for this?
After this operation the bucket cannot be listed or removed any more.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 21.02.23 22:52, Richard Bade wrote:
A colleague and I ran into this a few weeks ago. The way we managed to get access back to delete the bucket properly (using radosgw-admin bucket rm) was to reshard the bucket.
This created a new bucket index and therefore it was then possible to delete it. If you are looking to get access back to the objects, then as Eric said there's no way to get those indexes back but the objects will still be there in the pool.
Thanks for the answers so far. The issue we faced was a corrupt bucket index object. We thought about strategies to repair that but found none. I tried different things on a test cluster in a test bucket, one of them was "bi purge". And then I thought: Why is there such an operation when there is no way to get the index back and a working bucket? Resharding after a "bi prune" seems to work but as a result the bucket is empty when listing via S3. A bucket remove is successful but leaves all the RADOS objects in the index and data pools. Why is there no operation to rebuild the index for a bucket based on the existing RADOS objects in the data pool? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
If it's a test cluster, you could try: root@ceph01:/# radosgw-admin bucket check -h |grep -A1 check-objects --check-objects bucket check: rebuilds bucket index according to actual objects state On Wed, Feb 22, 2023, at 02:22, Robert Sander wrote:
On 21.02.23 22:52, Richard Bade wrote:
A colleague and I ran into this a few weeks ago. The way we managed to get access back to delete the bucket properly (using radosgw-admin bucket rm) was to reshard the bucket.
This created a new bucket index and therefore it was then possible to delete it. If you are looking to get access back to the objects, then as Eric said there's no way to get those indexes back but the objects will still be there in the pool.
Thanks for the answers so far.
The issue we faced was a corrupt bucket index object.
We thought about strategies to repair that but found none.
I tried different things on a test cluster in a test bucket, one of them was "bi purge". And then I thought: Why is there such an operation when there is no way to get the index back and a working bucket?
Resharding after a "bi prune" seems to work but as a result the bucket is empty when listing via S3. A bucket remove is successful but leaves all the RADOS objects in the index and data pools.
Why is there no operation to rebuild the index for a bucket based on the existing RADOS objects in the data pool?
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 22.02.23 14:42, David Orman wrote:
If it's a test cluster, you could try:
root@ceph01:/# radosgw-admin bucket check -h |grep -A1 check-objects --check-objects bucket check: rebuilds bucket index according to actual objects state
After a "bi purge" a "bucket check" returns an error: # radosgw-admin bi purge --bucket=testbucket --yes-i-really-mean-it # radosgw-admin bi list --bucket=testbucket ERROR: bi_list(): (2) No such file or directory # radosgw-admin bucket check --bucket=testbucket --check-objects 2023-02-22T16:51:11.970+0100 7fdcc6093e40 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for :testbucket[471f26a3-ff89-4b02-911a-0c89e2e295fa.104944180.1]) failed Adding --fix does not change anything. I can still download the one S3 object I put in the bucket because I know its name, but: # s3cmd ls s3://testbucket/ ERROR: S3 error: 404 (NoSuchKey) A "bucket reshard" recreates index objects: # radosgw-admin bucket reshard --bucket=testbucket --num-shards=12 tenant: bucket name: testbucket old bucket instance id: 471f26a3-ff89-4b02-911a-0c89e2e295fa.104944180.1 new bucket instance id: 471f26a3-ff89-4b02-911a-0c89e2e295fa.105128491.1 total entries: 0 2023-02-22T16:58:34.496+0100 7f52360dce40 1 execute INFO: reshard of bucket "testbucket" from "testbucket:471f26a3-ff89-4b02-911a-0c89e2e295fa.104944180.1" to "testbucket:471f26a3-ff89-4b02-911a-0c89e2e295fa.105128491.1" completed successfully After that "bucket check" runs without error but cannot fix the situation: # radosgw-admin bucket check --bucket=testbucket --check-objects --fix [] {} { "existing_header": { "usage": {} }, "calculated_header": { "usage": {} } } "s3cmd ls s3://testbucket/" shows nothing. "s3cmd rb s3://testbucket/" removes the bucket but the RADOS objects of the S3 objects remain in the data pool. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Everything you say is to be expected. I was not aware `reshard` could be run when the prior shards are removed, but apparently it can, and it creates new bucket index shards that are empty. Normally `reshard` reads entries from the old shards and copies their data to the new shards but since the old shards no longer exist, there’s nothing to copy over. But I presume the reason the reshard was suggested by other respondents was to allow for a bucket removal, which you verified. You’re correct in that the objects in the data pool still exist. To list those you could run `rgw-orphan-list`. It will output objects in the data pool that are not referenced by any bucket index. Note: for large clusters it can take a while to run. If after reviewing the list of objects you believe (have verified) they’re not used, you can then remove them via `rados` commands. rgw-orphan-list is still considered experimental, but it has successfully helped clean up large clusters. You also asked why there’s not a command to scan the data pool and recreate the bucket index. I think the concept would work as all head objects include the bucket marker in their names. There might be some corner cases where it’d partially fail, such as (possibly) transactional changes that were underway when the bucket index was purged. And there is metadata in the bucket index that’s not stored in the objects, so it would have to be recreated somehow. But no one has written it yet. Eric (he/him)
On Feb 22, 2023, at 11:04 AM, Robert Sander <r.sander@heinlein-support.de> wrote:
On 22.02.23 14:42, David Orman wrote:
If it's a test cluster, you could try: root@ceph01:/# radosgw-admin bucket check -h |grep -A1 check-objects --check-objects bucket check: rebuilds bucket index according to actual objects state
After a "bi purge" a "bucket check" returns an error:
# radosgw-admin bi purge --bucket=testbucket --yes-i-really-mean-it # radosgw-admin bi list --bucket=testbucket ERROR: bi_list(): (2) No such file or directory # radosgw-admin bucket check --bucket=testbucket --check-objects 2023-02-22T16:51:11.970+0100 7fdcc6093e40 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for :testbucket[471f26a3-ff89-4b02-911a-0c89e2e295fa.104944180.1]) failed
Adding --fix does not change anything.
I can still download the one S3 object I put in the bucket because I know its name, but:
# s3cmd ls s3://testbucket/ ERROR: S3 error: 404 (NoSuchKey)
A "bucket reshard" recreates index objects:
# radosgw-admin bucket reshard --bucket=testbucket --num-shards=12 tenant: bucket name: testbucket old bucket instance id: 471f26a3-ff89-4b02-911a-0c89e2e295fa.104944180.1 new bucket instance id: 471f26a3-ff89-4b02-911a-0c89e2e295fa.105128491.1 total entries: 0 2023-02-22T16:58:34.496+0100 7f52360dce40 1 execute INFO: reshard of bucket "testbucket" from "testbucket:471f26a3-ff89-4b02-911a-0c89e2e295fa.104944180.1" to "testbucket:471f26a3-ff89-4b02-911a-0c89e2e295fa.105128491.1" completed successfully
After that "bucket check" runs without error but cannot fix the situation:
# radosgw-admin bucket check --bucket=testbucket --check-objects --fix [] {} { "existing_header": { "usage": {} }, "calculated_header": { "usage": {} } }
"s3cmd ls s3://testbucket/" shows nothing.
"s3cmd rb s3://testbucket/" removes the bucket but the RADOS objects of the S3 objects remain in the data pool.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 22.02.23 17:45, J. Eric Ivancich wrote:
You also asked why there’s not a command to scan the data pool and recreate the bucket index. I think the concept would work as all head objects include the bucket marker in their names. There might be some corner cases where it’d partially fail, such as (possibly) transactional changes that were underway when the bucket index was purged. And there is metadata in the bucket index that’s not stored in the objects, so it would have to be recreated somehow. But no one has written it yet.
I am not in an urgent need to get such a feature. How would the process look to get development started in this direction? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Off the top of my head: 1. The command would take a bucket marker and a bucket names as arguments. It might also need some additional metadata to fill in gaps. 2. Scan the data pool for head objects that refer to that bucket marker. 3. Based on the number of such objects found, create a bucket index with an appropriate number of shards (approximately that number divided by 50,000). 4. For each such object: a. Read the manifest in the head object and see if all tail objects exit. b. If any tail objects are missing, maybe report that object on the console as non-recoverable? c. If all tail objects present, resolve the name of the object from the head object and add the bucket index entry to the appropriate shard. Note 1: Slight variations may be needed depending on whether or not the bucket entry object exists and the bucket instance object exists. Note 2: Versioned buckets will likely require some additional steps, but I’d need to refresh my memory on some of the details. Eric (he/him)
On Feb 23, 2023, at 4:51 AM, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
On 22.02.23 17:45, J. Eric Ivancich wrote:
You also asked why there’s not a command to scan the data pool and recreate the bucket index. I think the concept would work as all head objects include the bucket marker in their names. There might be some corner cases where it’d partially fail, such as (possibly) transactional changes that were underway when the bucket index was purged. And there is metadata in the bucket index that’s not stored in the objects, so it would have to be recreated somehow. But no one has written it yet.
I am not in an urgent need to get such a feature.
How would the process look to get development started in this direction?
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
A PR that adds experimental support for restoring a bucket index was merged into main. It’ll need back ports to reef, quincy, and pacific. https://tracker.ceph.com/issues/59053 <https://tracker.ceph.com/issues/59053?issue_count=48&issue_position=1&next_issue_id=58780> Currently it does not work for versioned buckets. And it is experimental. If anyone is able to try it I’d be curious about your experiences. Eric (he/him)
On Feb 23, 2023, at 11:20 AM, J. Eric Ivancich <ivancich@redhat.com> wrote:
Off the top of my head:
1. The command would take a bucket marker and a bucket names as arguments. It might also need some additional metadata to fill in gaps. 2. Scan the data pool for head objects that refer to that bucket marker. 3. Based on the number of such objects found, create a bucket index with an appropriate number of shards (approximately that number divided by 50,000). 4. For each such object: a. Read the manifest in the head object and see if all tail objects exit. b. If any tail objects are missing, maybe report that object on the console as non-recoverable? c. If all tail objects present, resolve the name of the object from the head object and add the bucket index entry to the appropriate shard.
Note 1: Slight variations may be needed depending on whether or not the bucket entry object exists and the bucket instance object exists. Note 2: Versioned buckets will likely require some additional steps, but I’d need to refresh my memory on some of the details.
Eric (he/him)
On Feb 23, 2023, at 4:51 AM, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
On 22.02.23 17:45, J. Eric Ivancich wrote:
You also asked why there’s not a command to scan the data pool and recreate the bucket index. I think the concept would work as all head objects include the bucket marker in their names. There might be some corner cases where it’d partially fail, such as (possibly) transactional changes that were underway when the bucket index was purged. And there is metadata in the bucket index that’s not stored in the objects, so it would have to be recreated somehow. But no one has written it yet.
I am not in an urgent need to get such a feature.
How would the process look to get development started in this direction?
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
David Orman
-
J. Eric Ivancich
-
Richard Bade
-
Robert Sander