S3 user with more than 1000 buckets
Hi, I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets. When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command. Have I missed a pagination limit for listing user buckets in the rados gateway? Thanks, Tom
Hi, You should increase these default settings: rgw_list_buckets_max_chunk // for buckets rgw_max_listing_results // for objects On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@tsolo.io> wrote:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas
Hi Jonas, Thanks :) that solved my issue. It would seem to me that this is heading towards something that the clients s3 should paginate, but I couldn't find any documentation on how to paginate bucket listings. All the information points to paginating object listing - which makes sense. Just for competition of this thread: The rgw parameters are found at: Quincy radosgw config ref <https://docs.ceph.com/en/quincy/radosgw/config-ref/> I ran the following command to update the parameter for all running rgw daemons: ceph config set client.rgw rgw_list_buckets_max_chunk 10000 And then confirmed the running daemons were configured: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep rgw_list_buckets_max_chunk "rgw_list_buckets_max_chunk": "10000", Kind regards, Tom On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@gmail.com> wrote:
Hi,
You should increase these default settings:
rgw_list_buckets_max_chunk // for buckets rgw_max_listing_results // for objects
On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@tsolo.io> wrote:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas
On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett <thomas@tsolo.io> wrote:
Hi Jonas,
Thanks :) that solved my issue.
It would seem to me that this is heading towards something that the clients s3 should paginate, but I couldn't find any documentation on how to paginate bucket listings.
the s3 ListBuckets API (https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html) doesn't support pagination, so there's no way for clients to do that but rgw itself should be able to paginate over the 'chunks' to return more than rgw_list_buckets_max_chunk entries in a single ListBuckets request. i opened a bug report for this at https://tracker.ceph.com/issues/63080
All the information points to paginating object listing - which makes sense.
Just for competition of this thread:
The rgw parameters are found at: Quincy radosgw config ref <https://docs.ceph.com/en/quincy/radosgw/config-ref/>
I ran the following command to update the parameter for all running rgw daemons: ceph config set client.rgw rgw_list_buckets_max_chunk 10000
And then confirmed the running daemons were configured: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep rgw_list_buckets_max_chunk "rgw_list_buckets_max_chunk": "10000",
Kind regards, Tom
On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@gmail.com> wrote:
Hi,
You should increase these default settings:
rgw_list_buckets_max_chunk // for buckets rgw_max_listing_results // for objects
On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@tsolo.io> wrote:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Thomas, If I'm not mistaken, the RGW will paginate ListBuckets essentially like ListObjectsv1 if the S3 client provides the appropriate "marker" parameter values. COS does this too, I noticed. I'm not sure which S3 clients can be relied on to do this, though. Matt On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett <thomas@tsolo.io> wrote:
Hi Jonas,
Thanks :) that solved my issue.
It would seem to me that this is heading towards something that the clients s3 should paginate, but I couldn't find any documentation on how to paginate bucket listings. All the information points to paginating object listing - which makes sense.
Just for competition of this thread:
The rgw parameters are found at: Quincy radosgw config ref <https://docs.ceph.com/en/quincy/radosgw/config-ref/>
I ran the following command to update the parameter for all running rgw daemons: ceph config set client.rgw rgw_list_buckets_max_chunk 10000
And then confirmed the running daemons were configured: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep rgw_list_buckets_max_chunk "rgw_list_buckets_max_chunk": "10000",
Kind regards, Tom
On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@gmail.com> wrote:
Hi,
You should increase these default settings:
rgw_list_buckets_max_chunk // for buckets rgw_max_listing_results // for objects
On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@tsolo.io> wrote:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Thanks for all the responses, much appreciated. Upping the chunk size fixes my problem in the short term but I upgrade to 17.2.6 :) Kind regards, Tom On Tue, 3 Oct 2023 at 15:28, Matt Benjamin <mbenjami@redhat.com> wrote:
Hi Thomas,
If I'm not mistaken, the RGW will paginate ListBuckets essentially like ListObjectsv1 if the S3 client provides the appropriate "marker" parameter values. COS does this too, I noticed. I'm not sure which S3 clients can be relied on to do this, though.
Matt
On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett <thomas@tsolo.io> wrote:
Hi Jonas,
Thanks :) that solved my issue.
It would seem to me that this is heading towards something that the clients s3 should paginate, but I couldn't find any documentation on how to paginate bucket listings. All the information points to paginating object listing - which makes sense.
Just for competition of this thread:
The rgw parameters are found at: Quincy radosgw config ref <https://docs.ceph.com/en/quincy/radosgw/config-ref/>
I ran the following command to update the parameter for all running rgw daemons: ceph config set client.rgw rgw_list_buckets_max_chunk 10000
And then confirmed the running daemons were configured: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep rgw_list_buckets_max_chunk "rgw_list_buckets_max_chunk": "10000",
Kind regards, Tom
On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@gmail.com> wrote:
Hi,
You should increase these default settings:
rgw_list_buckets_max_chunk // for buckets rgw_max_listing_results // for objects
On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@tsolo.io> wrote:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Den tis 3 okt. 2023 kl 11:59 skrev Thomas Bennett <thomas@tsolo.io>:
Hi,
I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets.
When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command.
Have I missed a pagination limit for listing user buckets in the rados gateway?
There is/was this bug that made the list not tell clients that there are more than 1000 buckets, so the clients would not ask for next list of pagination: https://tracker.ceph.com/issues/57901 For Quincy, it was in 17.2.6 so upgrading to that version would also fix it. https://docs.ceph.com/en/latest/releases/quincy/ search for - rgw: Fix truncated ListBuckets response (pr#49525 <https://github.com/ceph/ceph/pull/49525>, Joshua Baergen) -- May the most significant bit of your life be positive.
participants (5)
-
Casey Bodley
-
Janne Johansson
-
Jonas Nemeiksis
-
Matt Benjamin
-
Thomas Bennett