Hi Jonas, Thanks :) that solved my issue. It would seem to me that this is heading towards something that the clients s3 should paginate, but I couldn't find any documentation on how to paginate bucket listings. All the information points to paginating object listing - which makes sense. Just for competition of this thread: The rgw parameters are found at: Quincy radosgw config ref <https://docs.ceph.com/en/quincy/radosgw/config-ref/> I ran the following command to update the parameter for all running rgw daemons: ceph config set client.rgw rgw_list_buckets_max_chunk 10000 And then confirmed the running daemons were configured: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep rgw_list_buckets_max_chunk "rgw_list_buckets_max_chunk": "10000", Kind regards, Tom On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@gmail.com> wrote:
Hi,
You should increase these default settings:
rgw_list_buckets_max_chunk // for buckets rgw_max_listing_results // for objects
On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@tsolo.io> wrote:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas