This may be too broad of a topic, or opening a can of worms, but we are running a CEPH environment and I was wondering if there's any guidance about this question: Given that some group would like to store 50-100 TBs of data on CEPH and use it from a linux environment, are there any advantages or disadvantages in terms of performance/ease of use/learning curve to using cephfs vs using a block device thru rbd vs using object storage thru rgw? Here are my general thoughts: cephfs - Until recently, you were not allowed to have multiple filesystems. Not sure about performance. rbd - Can only be mounted on one system at a time, but I guess that filesystem could then be served using NFS. rgw - A different usage model from regular linux file/directory structure. Are there advantages to forcing people to use this interface? I'm tempted to set up 3 separate areas and try them and compare the results, but I'm wondering if somebody has done some similar experiment in the past. Thanks for any help you can provide! Jorge
Hi Jorge, I think it depends on your workload. On Tue, May 25, 2021 at 7:43 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
This may be too broad of a topic, or opening a can of worms, but we are running a CEPH environment and I was wondering if there's any guidance about this question:
Given that some group would like to store 50-100 TBs of data on CEPH and use it from a linux environment, are there any advantages or disadvantages in terms of performance/ease of use/learning curve to using cephfs vs using a block device thru rbd vs using object storage thru rgw? Here are my general thoughts:
cephfs - Until recently, you were not allowed to have multiple filesystems. Not sure about performance.
I/O performance can be /very/ good. Metadata performance has can vary. If you need shared POSIX access ("native" or NFS or SMB), you need cephfs.
rbd - Can only be mounted on one system at a time, but I guess that filesystem could then be served using NFS.
Yes, but it's single attach.
rgw - A different usage model from regular linux file/directory structure. Are there advantages to forcing people to use this interface?
There are advantages. S3 has become a preferred interface for some applications, especially analytics (e.g., Hadoop, Spark, PrestoSql)).
I'm tempted to set up 3 separate areas and try them and compare the results, but I'm wondering if somebody has done some similar experiment in the past.
Not sure, good question. Matt
Thanks for any help you can provide!
Jorge _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Yeah, agreed. My first question would be how is your user going to consume the storage? You'll struggle to run VM's on RadosGW and if they are doing archival backups then RBD is likely not the best solution. Each has very different requirements at the hardware level, for example if you are talking about running dozens of VM's then an SSD\NVME based cluster exposing RBD is a good solution, if you want to store large amounts of video files for a security system then a SATA based cluster with some NVME cache exposing S3 via RadosGW could be a good solution. On Wed, May 26, 2021 at 9:21 AM Matt Benjamin <mbenjami@redhat.com> wrote:
Hi Jorge,
I think it depends on your workload.
On Tue, May 25, 2021 at 7:43 PM Jorge Garcia <jgarcia@soe.ucsc.edu> wrote:
This may be too broad of a topic, or opening a can of worms, but we are running a CEPH environment and I was wondering if there's any guidance about this question:
Given that some group would like to store 50-100 TBs of data on CEPH and use it from a linux environment, are there any advantages or disadvantages in terms of performance/ease of use/learning curve to using cephfs vs using a block device thru rbd vs using object storage thru rgw? Here are my general thoughts:
cephfs - Until recently, you were not allowed to have multiple filesystems. Not sure about performance.
I/O performance can be /very/ good. Metadata performance has can vary. If you need shared POSIX access ("native" or NFS or SMB), you need cephfs.
rbd - Can only be mounted on one system at a time, but I guess that filesystem could then be served using NFS.
Yes, but it's single attach.
rgw - A different usage model from regular linux file/directory structure. Are there advantages to forcing people to use this interface?
There are advantages. S3 has become a preferred interface for some applications, especially analytics (e.g., Hadoop, Spark, PrestoSql)).
I'm tempted to set up 3 separate areas and try them and compare the results, but I'm wondering if somebody has done some similar experiment in the past.
Not sure, good question.
Matt
Thanks for any help you can provide!
Jorge _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The quick answer, is they are optimized for different use cases. Things like relational databases (mysql, postgresql) benefit from the performance that a dedicated filesystem can provide (rbd). Shared filesystems are usually counter indicated with such software. Shared filesystems like cephfs are nice but can't scale quite as well in number of filesystems as something like rbd. Latency in certain operations can be worse. Posix network filesystems have their drawbacks. Posix wasn't really designed around network fs's. But super useful when you need to share filesystems across nodes. A lot of existing software assumes shared filesystems. Can get pretty good scaling easily out of some software with it. rgw is a very different protocol (webby). A lot of existing software doesn't work with it. So comparability is not as good. But thats changing. Also has some assumptions around how data is read/written. Can be scaled quite large. http clients are very easy to come by to speak to it though, so for new software, its pretty nice. So, its not necessarily a "which one should I support". One of cephs great features is you can support all 3 with the same storage and use them all as needed. ________________________________________ From: Jorge Garcia <jgarcia@soe.ucsc.edu> Sent: Tuesday, May 25, 2021 4:43 PM To: ceph-users@ceph.io Subject: [ceph-users] cephfs vs rbd vs rgw Check twice before you click! This email originated from outside PNNL. This may be too broad of a topic, or opening a can of worms, but we are running a CEPH environment and I was wondering if there's any guidance about this question: Given that some group would like to store 50-100 TBs of data on CEPH and use it from a linux environment, are there any advantages or disadvantages in terms of performance/ease of use/learning curve to using cephfs vs using a block device thru rbd vs using object storage thru rgw? Here are my general thoughts: cephfs - Until recently, you were not allowed to have multiple filesystems. Not sure about performance. rbd - Can only be mounted on one system at a time, but I guess that filesystem could then be served using NFS. rgw - A different usage model from regular linux file/directory structure. Are there advantages to forcing people to use this interface? I'm tempted to set up 3 separate areas and try them and compare the results, but I'm wondering if somebody has done some similar experiment in the past. Thanks for any help you can provide! Jorge _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So, its not necessarily a "which one should I support". One of cephs great features is you can support all 3 with the same storage and use them all as needed.
..with the caveat that you can't serve the same files over them, but it is quite true that you can have all three served from the same cluster, stored on the same drives and so on. -- May the most significant bit of your life be positive.
participants (5)
-
Cory Hawkvelt
-
Fox, Kevin M
-
Janne Johansson
-
Jorge Garcia
-
Matt Benjamin