How to organize data in S3
Hi all, We need to store millions of files using S3 protocol in Ceph (version Nautilus), but have projects where isn't appropriate or possible to create a lot of S3 accounts. Is it better to have multiple S3 buckets or one bucket with sub folders? For example AWS service from Amazon allows you to create up to 100 buckets in each of your AWS cloud accounts. You can request more buckets, up to a maximum quota of 1,000, by submitting a service limit increase. There is no limit on the number of objects you can store in a bucket, but in the Ceph we run into a problem with listing and resharding with a millions of files in one bucket. Thank you Michal
Many buckets. Den sön 23 maj 2021 kl 20:53 skrev Michal Strnad <michal.strnad@cesnet.cz>:
Hi all,
We need to store millions of files using S3 protocol in Ceph (version Nautilus), but have projects where isn't appropriate or possible to create a lot of S3 accounts. Is it better to have multiple S3 buckets or one bucket with sub folders?
For example AWS service from Amazon allows you to create up to 100 buckets in each of your AWS cloud accounts. You can request more buckets, up to a maximum quota of 1,000, by submitting a service limit increase. There is no limit on the number of objects you can store in a bucket, but in the Ceph we run into a problem with listing and resharding with a millions of files in one bucket.
Thank you
Michal
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- May the most significant bit of your life be positive.
Thank you. So we can create millions of buckets associated to only one S3 account without any limitation or side effect? Does anyone use it this way? Many thanks in advance. Michal On 5/23/21 9:42 PM, Janne Johansson wrote:
Many buckets.
Den sön 23 maj 2021 kl 20:53 skrev Michal Strnad <michal.strnad@cesnet.cz>:
Hi all,
We need to store millions of files using S3 protocol in Ceph (version Nautilus), but have projects where isn't appropriate or possible to create a lot of S3 accounts. Is it better to have multiple S3 buckets or one bucket with sub folders?
For example AWS service from Amazon allows you to create up to 100 buckets in each of your AWS cloud accounts. You can request more buckets, up to a maximum quota of 1,000, by submitting a service limit increase. There is no limit on the number of objects you can store in a bucket, but in the Ceph we run into a problem with listing and resharding with a millions of files in one bucket.
Thank you
Michal
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
By default when account is creating max buckets for account is 1000. You can change this default and any time set max buckets setting for any already created account k Sent from my iPhone
On 24 May 2021, at 08:52, Michal Strnad <michal.strnad@cesnet.cz> wrote:
Thank you. So we can create millions of buckets associated to only one S3 account without any limitation or side effect? Does anyone use it this way?
Den mån 24 maj 2021 kl 07:51 skrev Michal Strnad <michal.strnad@cesnet.cz>:
Thank you. So we can create millions of buckets associated to only one S3 account without any limitation or side effect? Does anyone use it this way?
I'm not sure listing buckets will be fun if you literally create "millions of buckets" either, but it you preshard buckets to .. 11 shards and then stick around 1M objects per bucket, you would not get any dynamic reshardings, and the bucket listings will be reasonable, and you would not get millions of buckets either.
Hi all,
We need to store millions of files using S3 protocol in Ceph (version
-- May the most significant bit of your life be positive.
The one side effect you might see if you actually create a million or more bucklets for a single user, is a large OMAP warning, as the mapping of buckets to owners is not sharded. Matt On Mon, May 24, 2021 at 1:51 AM Michal Strnad <michal.strnad@cesnet.cz> wrote:
Thank you. So we can create millions of buckets associated to only one S3 account without any limitation or side effect? Does anyone use it this way?
Many thanks in advance. Michal
On 5/23/21 9:42 PM, Janne Johansson wrote:
Many buckets.
Den sön 23 maj 2021 kl 20:53 skrev Michal Strnad <michal.strnad@cesnet.cz>:
Hi all,
We need to store millions of files using S3 protocol in Ceph (version Nautilus), but have projects where isn't appropriate or possible to create a lot of S3 accounts. Is it better to have multiple S3 buckets or one bucket with sub folders?
For example AWS service from Amazon allows you to create up to 100 buckets in each of your AWS cloud accounts. You can request more buckets, up to a maximum quota of 1,000, by submitting a service limit increase. There is no limit on the number of objects you can store in a bucket, but in the Ceph we run into a problem with listing and resharding with a millions of files in one bucket.
Thank you
Michal
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
participants (4)
-
Janne Johansson
-
Konstantin Shalygin
-
Matt Benjamin
-
Michal Strnad