Thanks paul Yes listing v2 is not supported yet. I checked metadata osds and all of them are 600gb 10k hdd I dont think this was the issue. I will test the --allow-unordered Regards ________________________________ From: Paul Emmerich <paul.emmerich@croit.io> Sent: Thursday, October 17, 2019 10:00 AM To: Arash Shams <ara4sh@hotmail.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] RadosGW cant list objects when there are too many of them Listing large buckets is slow due to S3 ordering requirements, it's approximately O(n^2). However, I wouldn't consider 5M to be a large bucket, it should go to only ~50 shards which should still perform reasonable. How fast are your metadata OSDs? Try --allow-unordered in radosgw-admin to get an unordered result which is only O(n) as you'd expect. For boto3: I'm not sure if v2 object listing is available yet (I think it has only been merged into master but has not yet made it into a release?). It doesn't support unordered listing but there has been some work to implement it there, not sure about the current state. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io<http://www.croit.io> Tel: +49 89 1896585 90 On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh@hotmail.com> wrote:
Dear All
I have a bucket with 5 million Objects and I cant list objects with radosgw-admin bucket list --bucket=bucket | jq .[].name or listing files using boto3
s3 = boto3.client('s3', endpoint_url=credentials['endpoint_url'], aws_access_key_id=credentials['access_key'], aws_secret_access_key=credentials['secret_key'])
response = s3.list_objects_v2(Bucket=bucket_name) for item in response['Contents']: print(item['Key'])
what is the solution ? how can I find list of my objects ?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io