23 May
2021
23 May
'21
7:52 p.m.
Hi all,
We need to store millions of files using S3 protocol in Ceph (version
Nautilus), but have projects where isn't appropriate or possible to
create a lot of S3 accounts. Is it better to have multiple S3 buckets or
one bucket with sub folders?
For example AWS service from Amazon allows you to create up to 100
buckets in each of your AWS cloud accounts. You can request more
buckets, up to a maximum quota of 1,000, by submitting a service limit
increase. There is no limit on the number of objects you can store in a
bucket, but in the Ceph we run into a problem with listing and
resharding with a millions of files in one bucket.
Thank you
Michal
23 May
23 May
8:42 p.m.
Many buckets.
Den sön 23 maj 2021 kl 20:53 skrev Michal Strnad <michal.strnad(a)cesnet.cz>cz>:
Hi all,
We need to store millions of files using S3 protocol in Ceph (version
Nautilus), but have projects where isn't appropriate or possible to
create a lot of S3 accounts. Is it better to have multiple S3 buckets or
one bucket with sub folders?
For example AWS service from Amazon allows you to create up to 100
buckets in each of your AWS cloud accounts. You can request more
buckets, up to a maximum quota of 1,000, by submitting a service limit
increase. There is no limit on the number of objects you can store in a
bucket, but in the Ceph we run into a problem with listing and
resharding with a millions of files in one bucket.
Thank you
Michal
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
May the most significant bit of your life be positive.
24 May
24 May
6:51 a.m.
Thank you. So we can create millions of buckets associated to only one
S3 account without any limitation or side effect? Does anyone use it
this way?
Many thanks in advance.
Michal
On 5/23/21 9:42 PM, Janne Johansson wrote:
Many buckets.
Den sön 23 maj 2021 kl 20:53 skrev Michal Strnad <michal.strnad(a)cesnet.cz>cz>:
Hi all,
We need to store millions of files using S3 protocol in Ceph (version
Nautilus), but have projects where isn't appropriate or possible to
create a lot of S3 accounts. Is it better to have multiple S3 buckets or
one bucket with sub folders?
For example AWS service from Amazon allows you to create up to 100
buckets in each of your AWS cloud accounts. You can request more
buckets, up to a maximum quota of 1,000, by submitting a service limit
increase. There is no limit on the number of objects you can store in a
bucket, but in the Ceph we run into a problem with listing and
resharding with a millions of files in one bucket.
Thank you
Michal
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:24 a.m.
By default when account is creating max buckets for account is 1000. You can change this
default and any time set max buckets setting for any already created account
k
Sent from my iPhone
On 24 May 2021, at 08:52, Michal Strnad
<michal.strnad(a)cesnet.cz> wrote:
Thank you. So we can create millions of buckets associated to only one
S3 account without any limitation or side effect? Does anyone use it
this way?
8:43 a.m.
Den mån 24 maj 2021 kl 07:51 skrev Michal Strnad <michal.strnad(a)cesnet.cz>cz>:
Thank you. So we can create millions of buckets associated to only one
S3 account without any limitation or side effect? Does anyone use it
this way?
I'm not sure listing buckets will be fun if you literally create
"millions of buckets" either, but it you preshard buckets to .. 11
shards and then stick around 1M objects per bucket, you would not get
any dynamic reshardings, and the bucket listings will be reasonable,
and you would not get millions of buckets either.
>> Hi all,
>>
>> We need to store millions of files using S3 protocol in Ceph (version
--
May the most significant bit of your life be positive.
12:59 p.m.
The one side effect you might see if you actually create a million or
more bucklets for a single user, is a large OMAP warning, as the
mapping of buckets to owners is not sharded.
Matt
On Mon, May 24, 2021 at 1:51 AM Michal Strnad <michal.strnad(a)cesnet.cz> wrote:
Thank you. So we can create millions of buckets associated to only one
S3 account without any limitation or side effect? Does anyone use it
this way?
Many thanks in advance.
Michal
On 5/23/21 9:42 PM, Janne Johansson wrote:
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
Many buckets.
Den sön 23 maj 2021 kl 20:53 skrev Michal Strnad <michal.strnad(a)cesnet.cz>cz>:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi all,
We need to store millions of files using S3 protocol in Ceph (version
Nautilus), but have projects where isn't appropriate or possible to
create a lot of S3 accounts. Is it better to have multiple S3 buckets or
one bucket with sub folders?
For example AWS service from Amazon allows you to create up to 100
buckets in each of your AWS cloud accounts. You can request more
buckets, up to a maximum quota of 1,000, by submitting a service limit
increase. There is no limit on the number of objects you can store in a
bucket, but in the Ceph we run into a problem with listing and
resharding with a millions of files in one bucket.
Thank you
Michal
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1089
days inactive
1090
days old
5 comments
4 participants
participants (4)
-
Janne Johansson -
Konstantin Shalygin -
Matt Benjamin -
Michal Strnad