Hi all, We have several nfs servers providing file storage. There is a nginx in front of nfs servers in order to serve the clients. The files are mostly small files and nearly about 30TB in total. I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate to do so. The data migrating from nfs to rgw is a huge job. Besides I'm not sure whether ceph rgw is suitable in this scenario or not. Thanks
On 5/12/20 4:22 AM, Zhenshi Zhou wrote:
Hi all,
We have several nfs servers providing file storage. There is a nginx in front of nfs servers in order to serve the clients. The files are mostly small files and nearly about 30TB in total.
What is small? How many objects/files are you talking about?
I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate to do so. The data migrating from nfs to rgw is a huge job. Besides I'm not sure whether ceph rgw is suitable in this scenario or not.
Yes, it is. But make sure you don't put millions of objects into a single bucket. Make sure that you spread them out so that you have let's say 1M of objects per bucket at max. Wido
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Wido, I did a research on the nfs files. I found that it contains much pictures about 50KB, and much video files around 30MB. The amount of the files is more than 1 million. Maybe I can find a way to seperate the files in more buckets so that there is no more than 1M objects in each bucket. But how about the small files around 50KB. Does rgw serve well on small files? Wido den Hollander <wido@42on.com> 于2020年5月12日周二 下午2:41写道：
On 5/12/20 4:22 AM, Zhenshi Zhou wrote:
Hi all,
We have several nfs servers providing file storage. There is a nginx in front of nfs servers in order to serve the clients. The files are mostly small files and nearly about 30TB in total.
What is small? How many objects/files are you talking about?
I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate to do so. The data migrating from nfs to rgw is a huge job. Besides I'm not sure whether ceph rgw is suitable in this scenario or not.
Yes, it is. But make sure you don't put millions of objects into a single bucket. Make sure that you spread them out so that you have let's say 1M of objects per bucket at max.
Wido
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 5/18/20 1:51 PM, Zhenshi Zhou wrote:
Hi Wido,
I did a research on the nfs files. I found that it contains much pictures about 50KB, and much video files around 30MB. The amount of the files is more than 1 million. Maybe I can find a way to seperate the files in more buckets so that there is no more than 1M objects in each bucket. But how about the small files around 50KB. Does rgw serve well on small files?
I would recommend using different buckets. What I've done in such cases is use the year+month for sharding. For example: video-2020-05 RGW can serve objects which are 50kB in size, but there is overhead involved. Storing a lot of such small objects comes at a price of overhead. Wido
Wido den Hollander <wido@42on.com <mailto:wido@42on.com>> 于2020年5月12 日周二 下午2:41写道：
On 5/12/20 4:22 AM, Zhenshi Zhou wrote: > Hi all, > > We have several nfs servers providing file storage. There is a nginx in > front of > nfs servers in order to serve the clients. The files are mostly small files > and > nearly about 30TB in total. >
What is small? How many objects/files are you talking about?
> I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate to > do so. > The data migrating from nfs to rgw is a huge job. Besides I'm not sure > whether > ceph rgw is suitable in this scenario or not. >
Yes, it is. But make sure you don't put millions of objects into a single bucket. Make sure that you spread them out so that you have let's say 1M of objects per bucket at max.
Wido
> Thanks > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> >
On Mon, May 18, 2020 at 1:52 PM Zhenshi Zhou <deaderzzs@gmail.com> wrote:
50KB, and much video files around 30MB. The amount of the files is more than 1 million. Maybe I can find a way to seperate the files in more buckets so that there is no more than 1M objects in each bucket. But how about the small files around 50KB. Does rgw serve well on small files?
1 million files is usually the point where you first need to start thinking about some optimizations, but that's mostly just making sure that the index is on SSD and it'll happily work up to ~10 million files. Then you might need to start thinking about the index being on *good* SSDs (and/or on many SSDs/DB devices). It starts the get interesting if you need to go beyond 100 million files, that's the point where you need to start tuning shard sizes and the types of index queries that you send... I've found that a few hundred million objects per bucket are no problem if you run with large shard sizes (500k - 1 million); however, there are some index-queries that can be really expensive like filtering on prefixes in some pathological cases... Small files: sure, works well, but can be challenging for erasure coding on HDDs, but that's unrelated to rgw/you'd have the same problem with CephFS Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
Wido den Hollander <wido@42on.com> 于2020年5月12日周二 下午2:41写道：
On 5/12/20 4:22 AM, Zhenshi Zhou wrote:
Hi all,
We have several nfs servers providing file storage. There is a nginx in front of nfs servers in order to serve the clients. The files are mostly small files and nearly about 30TB in total.
What is small? How many objects/files are you talking about?
I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate to do so. The data migrating from nfs to rgw is a huge job. Besides I'm not sure whether ceph rgw is suitable in this scenario or not.
Yes, it is. But make sure you don't put millions of objects into a single bucket. Make sure that you spread them out so that you have let's say 1M of objects per bucket at max.
Wido
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Awesome, thanks a lot ! I'll try it. Paul Emmerich <paul.emmerich@croit.io> 于2020年5月18日周一 下午8:53写道：
On Mon, May 18, 2020 at 1:52 PM Zhenshi Zhou <deaderzzs@gmail.com> wrote:
50KB, and much video files around 30MB. The amount of the files is more than 1 million. Maybe I can find a way to seperate the files in more buckets so that there is no more than 1M objects in each bucket. But how about the small files around 50KB. Does rgw serve well on small files?
1 million files is usually the point where you first need to start thinking about some optimizations, but that's mostly just making sure that the index is on SSD and it'll happily work up to ~10 million files. Then you might need to start thinking about the index being on *good* SSDs (and/or on many SSDs/DB devices).
It starts the get interesting if you need to go beyond 100 million files, that's the point where you need to start tuning shard sizes and the types of index queries that you send...
I've found that a few hundred million objects per bucket are no problem if you run with large shard sizes (500k - 1 million); however, there are some index-queries that can be really expensive like filtering on prefixes in some pathological cases...
Small files: sure, works well, but can be challenging for erasure coding on HDDs, but that's unrelated to rgw/you'd have the same problem with CephFS
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
Wido den Hollander <wido@42on.com> 于2020年5月12日周二 下午2:41写道：
On 5/12/20 4:22 AM, Zhenshi Zhou wrote:
Hi all,
We have several nfs servers providing file storage. There is a nginx
in
front of nfs servers in order to serve the clients. The files are mostly small files and nearly about 30TB in total.
What is small? How many objects/files are you talking about?
I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate to do so. The data migrating from nfs to rgw is a huge job. Besides I'm not sure whether ceph rgw is suitable in this scenario or not.
Yes, it is. But make sure you don't put millions of objects into a single bucket. Make sure that you spread them out so that you have let's say 1M of objects per bucket at max.
Wido
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Paul Emmerich
-
Wido den Hollander
-
Zhenshi Zhou