On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett <thomas@tsolo.io> wrote:
Hi Jonas,
Thanks :) that solved my issue.
It would seem to me that this is heading towards something that the clients s3 should paginate, but I couldn't find any documentation on how to paginate bucket listings.
the s3 ListBuckets API (https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html) doesn't support pagination, so there's no way for clients to do that but rgw itself should be able to paginate over the 'chunks' to return more than rgw_list_buckets_max_chunk entries in a single ListBuckets request. i opened a bug report for this at https://tracker.ceph.com/issues/63080
All the information points to paginating object listing - which makes sense.
Just for competition of this thread:
The rgw parameters are found at: Quincy radosgw config ref <https://docs.ceph.com/en/quincy/radosgw/config-ref/>
I ran the following command to update the parameter for all running rgw daemons: ceph config set client.rgw rgw_list_buckets_max_chunk 10000
And then confirmed the running daemons were configured: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep rgw_list_buckets_max_chunk "rgw_list_buckets_max_chunk": "10000",
Kind regards, Tom
On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@gmail.com> wrote:
Hi,
You should increase these default settings:
rgw_list_buckets_max_chunk // for buckets rgw_max_listing_results // for objects
On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@tsolo.io> wrote:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io