When you say that you can't list it with boto or radosgw-admin, what happens? Does it give you an error, or just hang/timeout? How many shards does the bucket have? On 10/17/19 6:00 AM, Paul Emmerich wrote:
Listing large buckets is slow due to S3 ordering requirements, it's approximately O(n^2). However, I wouldn't consider 5M to be a large bucket, it should go to only ~50 shards which should still perform reasonable. How fast are your metadata OSDs?
I just wanted to share that recent work by Eric and Mark is showing huge improvements in sharded listing performance: https://github.com/ceph/ceph/pull/30853#issuecomment-541964967
Try --allow-unordered in radosgw-admin to get an unordered result which is only O(n) as you'd expect.
For boto3: I'm not sure if v2 object listing is available yet (I think it has only been merged into master but has not yet made it into a release?). It doesn't support unordered listing but there has been some work to implement it there, not sure about the current state.
Paul