The quick answer, is they are optimized for different use cases. Things like relational databases (mysql, postgresql) benefit from the performance that a dedicated filesystem can provide (rbd). Shared filesystems are usually counter indicated with such software. Shared filesystems like cephfs are nice but can't scale quite as well in number of filesystems as something like rbd. Latency in certain operations can be worse. Posix network filesystems have their drawbacks. Posix wasn't really designed around network fs's. But super useful when you need to share filesystems across nodes. A lot of existing software assumes shared filesystems. Can get pretty good scaling easily out of some software with it. rgw is a very different protocol (webby). A lot of existing software doesn't work with it. So comparability is not as good. But thats changing. Also has some assumptions around how data is read/written. Can be scaled quite large. http clients are very easy to come by to speak to it though, so for new software, its pretty nice. So, its not necessarily a "which one should I support". One of cephs great features is you can support all 3 with the same storage and use them all as needed. ________________________________________ From: Jorge Garcia <jgarcia(a)soe.ucsc.edu> Sent: Tuesday, May 25, 2021 4:43 PM To: ceph-users(a)ceph.io Subject: [ceph-users] cephfs vs rbd vs rgw Check twice before you click! This email originated from outside PNNL. This may be too broad of a topic, or opening a can of worms, but we are running a CEPH environment and I was wondering if there's any guidance about this question: Given that some group would like to store 50-100 TBs of data on CEPH and use it from a linux environment, are there any advantages or disadvantages in terms of performance/ease of use/learning curve to using cephfs vs using a block device thru rbd vs using object storage thru rgw? Here are my general thoughts: cephfs - Until recently, you were not allowed to have multiple filesystems. Not sure about performance. rbd - Can only be mounted on one system at a time, but I guess that filesystem could then be served using NFS. rgw - A different usage model from regular linux file/directory structure. Are there advantages to forcing people to use this interface? I'm tempted to set up 3 separate areas and try them and compare the results, but I'm wondering if somebody has done some similar experiment in the past. Thanks for any help you can provide! Jorge _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

..with the caveat that you can't serve the same files over them, but it is quite true that you can have all three served from the same cluster, stored on the same drives and so on. -- May the most significant bit of your life be positive.