Re: RadosGW cant list objects when there are too many of them
Listing large buckets is slow due to S3 ordering requirements, it's approximately O(n^2). However, I wouldn't consider 5M to be a large bucket, it should go to only ~50 shards which should still perform reasonable. How fast are your metadata OSDs? Try --allow-unordered in radosgw-admin to get an unordered result which is only O(n) as you'd expect. For boto3: I'm not sure if v2 object listing is available yet (I think it has only been merged into master but has not yet made it into a release?). It doesn't support unordered listing but there has been some work to implement it there, not sure about the current state. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh@hotmail.com> wrote:
Dear All
I have a bucket with 5 million Objects and I cant list objects with radosgw-admin bucket list --bucket=bucket | jq .[].name or listing files using boto3
s3 = boto3.client('s3', endpoint_url=credentials['endpoint_url'], aws_access_key_id=credentials['access_key'], aws_secret_access_key=credentials['secret_key'])
response = s3.list_objects_v2(Bucket=bucket_name) for item in response['Contents']: print(item['Key'])
what is the solution ? how can I find list of my objects ?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
When you say that you can't list it with boto or radosgw-admin, what happens? Does it give you an error, or just hang/timeout? How many shards does the bucket have? On 10/17/19 6:00 AM, Paul Emmerich wrote:
Listing large buckets is slow due to S3 ordering requirements, it's approximately O(n^2). However, I wouldn't consider 5M to be a large bucket, it should go to only ~50 shards which should still perform reasonable. How fast are your metadata OSDs?
I just wanted to share that recent work by Eric and Mark is showing huge improvements in sharded listing performance: https://github.com/ceph/ceph/pull/30853#issuecomment-541964967
Try --allow-unordered in radosgw-admin to get an unordered result which is only O(n) as you'd expect.
For boto3: I'm not sure if v2 object listing is available yet (I think it has only been merged into master but has not yet made it into a release?). It doesn't support unordered listing but there has been some work to implement it there, not sure about the current state.
Paul
Thanks paul Yes listing v2 is not supported yet. I checked metadata osds and all of them are 600gb 10k hdd I dont think this was the issue. I will test the --allow-unordered Regards ________________________________ From: Paul Emmerich <paul.emmerich@croit.io> Sent: Thursday, October 17, 2019 10:00 AM To: Arash Shams <ara4sh@hotmail.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] RadosGW cant list objects when there are too many of them Listing large buckets is slow due to S3 ordering requirements, it's approximately O(n^2). However, I wouldn't consider 5M to be a large bucket, it should go to only ~50 shards which should still perform reasonable. How fast are your metadata OSDs? Try --allow-unordered in radosgw-admin to get an unordered result which is only O(n) as you'd expect. For boto3: I'm not sure if v2 object listing is available yet (I think it has only been merged into master but has not yet made it into a release?). It doesn't support unordered listing but there has been some work to implement it there, not sure about the current state. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io<http://www.croit.io> Tel: +49 89 1896585 90 On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh@hotmail.com> wrote:
Dear All
I have a bucket with 5 million Objects and I cant list objects with radosgw-admin bucket list --bucket=bucket | jq .[].name or listing files using boto3
s3 = boto3.client('s3', endpoint_url=credentials['endpoint_url'], aws_access_key_id=credentials['access_key'], aws_secret_access_key=credentials['secret_key'])
response = s3.list_objects_v2(Bucket=bucket_name) for item in response['Contents']: print(item['Key'])
what is the solution ? how can I find list of my objects ?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Arash Shams
-
Casey Bodley
-
Paul Emmerich