Thanks for all the responses, much appreciated. Upping the chunk size fixes my problem in the short term but I upgrade to 17.2.6 :) Kind regards, Tom On Tue, 3 Oct 2023 at 15:28, Matt Benjamin <mbenjami@redhat.com> wrote:
Hi Thomas,
If I'm not mistaken, the RGW will paginate ListBuckets essentially like ListObjectsv1 if the S3 client provides the appropriate "marker" parameter values. COS does this too, I noticed. I'm not sure which S3 clients can be relied on to do this, though.
Matt
On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett <thomas@tsolo.io> wrote:
Hi Jonas,
Thanks :) that solved my issue.
It would seem to me that this is heading towards something that the clients s3 should paginate, but I couldn't find any documentation on how to paginate bucket listings. All the information points to paginating object listing - which makes sense.
Just for competition of this thread:
The rgw parameters are found at: Quincy radosgw config ref <https://docs.ceph.com/en/quincy/radosgw/config-ref/>
I ran the following command to update the parameter for all running rgw daemons: ceph config set client.rgw rgw_list_buckets_max_chunk 10000
And then confirmed the running daemons were configured: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep rgw_list_buckets_max_chunk "rgw_list_buckets_max_chunk": "10000",
Kind regards, Tom
On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@gmail.com> wrote:
Hi,
You should increase these default settings:
rgw_list_buckets_max_chunk // for buckets rgw_max_listing_results // for objects
On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@tsolo.io> wrote:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309