On 5/18/20 1:51 PM, Zhenshi Zhou wrote:
Hi Wido,
I did a research on the nfs files. I found that it contains much pictures about 50KB, and much video files around 30MB. The amount of the files is more than 1 million. Maybe I can find a way to seperate the files in more buckets so that there is no more than 1M objects in each bucket. But how about the small files around 50KB. Does rgw serve well on small files?
I would recommend using different buckets. What I've done in such cases is use the year+month for sharding. For example: video-2020-05 RGW can serve objects which are 50kB in size, but there is overhead involved. Storing a lot of such small objects comes at a price of overhead. Wido
Wido den Hollander <wido@42on.com <mailto:wido@42on.com>> 于2020年5月12 日周二 下午2:41写道：
On 5/12/20 4:22 AM, Zhenshi Zhou wrote: > Hi all, > > We have several nfs servers providing file storage. There is a nginx in > front of > nfs servers in order to serve the clients. The files are mostly small files > and > nearly about 30TB in total. >
What is small? How many objects/files are you talking about?
> I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate to > do so. > The data migrating from nfs to rgw is a huge job. Besides I'm not sure > whether > ceph rgw is suitable in this scenario or not. >
Yes, it is. But make sure you don't put millions of objects into a single bucket. Make sure that you spread them out so that you have let's say 1M of objects per bucket at max.
Wido
> Thanks > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> >