Hi Robert, A colleague and I ran into this a few weeks ago. The way we managed to get access back to delete the bucket properly (using radosgw-admin bucket rm) was to reshard the bucket. This created a new bucket index and therefore it was then possible to delete it. If you are looking to get access back to the objects, then as Eric said there's no way to get those indexes back but the objects will still be there in the pool. sudo radosgw-admin bucket reshard --bucket={bucket_name} --num-shards {number} If you are doing multi-site replication, resharding can cause some issues on earlier versions of ceph so check that out if that applies to you. If you're just going to delete the bucket anyway it may not be an issue. Regards, Richard On Wed, 22 Feb 2023 at 07:43, J. Eric Ivancich <ivancich(a)redhat.com> wrote: > > When the admin runs “bi purge” they have the option of supplying a bucket_id with the “--bucket-id” command-line argument. This was useful back when resharding did not automatically remove the older bucket index shards (which it now does), which had a different bucket_id from the current bucket index shards. > > If the admin doesn't supply a bucket_id it assumes the current bucket index shards are to be purged. Because this is generally not wanted, the admin is required to supply the "--yes-i-really-mean-it” command-line argument to verify that they know that this is generally not done. > > There is no “undo” for “bi purge”, because it removes the metadata objects that contain the bucket listing. > > The objects should still be readable if you know their names. > > > After this operation the bucket cannot be listed or removed any more. > > > Knowing the above, what is your goal? Removal/clean-up? Recovery of as much as possible? Both are possible to a degree (not 100%) but the processes are not simple and highly manual. > > Eric > (he/him) > > > On Feb 20, 2023, at 10:01 AM, Robert Sander <r.sander(a)heinlein-support.de> wrote: > > > > Hi, > > > > There is an operation "radosgw-admin bi purge" that removes all bucket index objects for one bucket in the rados gateway. > > > > What is the undo operation for this? > > > > After this operation the bucket cannot be listed or removed any more. > > > > Regards > > -- > > Robert Sander > > Heinlein Consulting GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > http://www.heinlein-support.de > > > > Tel: 030 / 405051-43 > > Fax: 030 / 405051-19 > > > > Zwangsangaben lt. §35a GmbHG: > > HRB 220009 B / Amtsgericht Berlin-Charlottenburg, > > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

If it's a test cluster, you could try: root@ceph01:/# radosgw-admin bucket check -h |grep -A1 check-objects --check-objects bucket check: rebuilds bucket index according to actual objects state On Wed, Feb 22, 2023, at 02:22, Robert Sander wrote: > On 21.02.23 22:52, Richard Bade wrote: > >> A colleague and I ran into this a few weeks ago. The way we managed to >> get access back to delete the bucket properly (using radosgw-admin >> bucket rm) was to reshard the bucket. > >> This created a new bucket index and therefore it was then possible to delete it. >> If you are looking to get access back to the objects, then as Eric >> said there's no way to get those indexes back but the objects will >> still be there in the pool. > > Thanks for the answers so far. > > The issue we faced was a corrupt bucket index object. > > We thought about strategies to repair that but found none. > > I tried different things on a test cluster in a test bucket, one of them > was "bi purge". And then I thought: Why is there such an operation when > there is no way to get the index back and a working bucket? > > Resharding after a "bi prune" seems to work but as a result the bucket > is empty when listing via S3. A bucket remove is successful but leaves > all the RADOS objects in the index and data pools. > > Why is there no operation to rebuild the index for a bucket based on the > existing RADOS objects in the data pool? > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Everything you say is to be expected. I was not aware `reshard` could be run when the prior shards are removed, but apparently it can, and it creates new bucket index shards that are empty. Normally `reshard` reads entries from the old shards and copies their data to the new shards but since the old shards no longer exist, there’s nothing to copy over. But I presume the reason the reshard was suggested by other respondents was to allow for a bucket removal, which you verified. You’re correct in that the objects in the data pool still exist. To list those you could run `rgw-orphan-list`. It will output objects in the data pool that are not referenced by any bucket index. Note: for large clusters it can take a while to run. If after reviewing the list of objects you believe (have verified) they’re not used, you can then remove them via `rados` commands. rgw-orphan-list is still considered experimental, but it has successfully helped clean up large clusters. You also asked why there’s not a command to scan the data pool and recreate the bucket index. I think the concept would work as all head objects include the bucket marker in their names. There might be some corner cases where it’d partially fail, such as (possibly) transactional changes that were underway when the bucket index was purged. And there is metadata in the bucket index that’s not stored in the objects, so it would have to be recreated somehow. But no one has written it yet. Eric (he/him)

Off the top of my head: 1. The command would take a bucket marker and a bucket names as arguments. It might also need some additional metadata to fill in gaps. 2. Scan the data pool for head objects that refer to that bucket marker. 3. Based on the number of such objects found, create a bucket index with an appropriate number of shards (approximately that number divided by 50,000). 4. For each such object: a. Read the manifest in the head object and see if all tail objects exit. b. If any tail objects are missing, maybe report that object on the console as non-recoverable? c. If all tail objects present, resolve the name of the object from the head object and add the bucket index entry to the appropriate shard. Note 1: Slight variations may be needed depending on whether or not the bucket entry object exists and the bucket instance object exists. Note 2: Versioned buckets will likely require some additional steps, but I’d need to refresh my memory on some of the details. Eric (he/him)