30 Jan
2020
30 Jan
'20
2:43 p.m.
Hello All,
I have a HW RAID based 240 TB data pool with about 200 million files for
users in a scientific institution. Data sizes range from tiny parameter
files for scientific calculations and experiments to huge images of
brain scans. There are group directories, home directories, Windows
roaming profile directories organized in ZFS pools on Solaris operating
systems, exported via NFS and Samba to Linux, macOS, and Windows clients.
I would like to switch to CephFS because of the flexibility and
expandability but I cannot find any recommendations for which storage
backend would be suitable for all the functionality we have.
Since I like the features of ZFS like immediate snapshots of very large
data pools, quotas for each file system within hierarchical data trees
and dynamic expandability by simply adding new disks or disk images
without manual resizing would it be a good idea to create RBD images,
map them onto the file servers and create zpools on the mapped images? I
know that ZFS best works with raw disks but maybe a RBD image is close
enough to a raw disk?
Or would CephFS be the way to go? Can there be multiple CephFS pools for
the group data folders and for the user's home directory folders for
example or do I have to have everything in one single file space?
Maybe someone can share his or her field experience?
Thank you very much.
Best regards
Willi
30 Jan
30 Jan
4:22 p.m.
Hi,
Quoting Willi Schiegel (willi.schiegel(a)technologit.de):
Hello All,
I have a HW RAID based 240 TB data pool with about 200 million files for
users in a scientific institution. Data sizes range from tiny parameter
files for scientific calculations and experiments to huge images of brain
scans. There are group directories, home directories, Windows roaming
profile directories organized in ZFS pools on Solaris operating systems,
exported via NFS and Samba to Linux, macOS, and Windows clients.
I would like to switch to CephFS because of the flexibility and
expandability but I cannot find any recommendations for which storage
backend would be suitable for all the functionality we have.
Since I like the features of ZFS like immediate snapshots of very large data
pools, quotas for each file system within hierarchical data trees and
dynamic expandability by simply adding new disks or disk images without
manual resizing would it be a good idea to create RBD images, map them onto
the file servers and create zpools on the mapped images? I know that ZFS
best works with raw disks but maybe a RBD image is close enough to a raw
disk?
Some of the features also exist within Ceph. Ceph would rebalance your
data accross the cluster immdiately as opposed to zfs (only new data
written to new disks). You can make snapshots of filesystems, you can
set quotas but with different caveats than with zfs [1]. You would have
to setup Ceph with samba (ceph vfs) for your windows / mac clients [2]
which is not (yet) HA (or you would build something yourself with Samba
CTDB, Ceph object locking feature of CTDB and vfs_ceph). If you need to
support nfs you would be able to do so in a HA fashion with nfs-ganesha
(nautilus has fixed most caveats). You would need one ore more MDS
servers. And it would really depend on the workload (and the amount of
clients) if it would work out for you. It might require tuning of the
MDSs. It's one of the more difficult interfaces of Ceph to comprehend.
ZFS with rbd would work (IIRC I have seen a presentation at Cephalacon
of a user using both zfs and rbd). I have certainly done it myself. As
far as zfs is concerned it's just another disk (depends if you would map
it with rbd-nbd, krbd or as virtual disk, but all three should work).
Or would CephFS be the way to go? Can there be
multiple CephFS pools for the
group data folders and for the user's home directory folders for example or
do I have to have everything in one single file space?
You can have multiple data pools and use xattrs to set the appropriate
attributes to use the correct pool. You can use name spaces (ceph, not
linux) to "prefix" all objects put in cephfs. That's there primarily there
to
be able to set permissions per "namespace" instead of on a whole pool.
In this case you can restrict what objects a user is allowed access to.
You can also restrict the permission of creating cephfs snapshots
(from nautilus onwards) on a per user basis.
It would probably require a whole bunch of servers to get the right sizing for
ceph as opposed to your zfs setup for which you proably have one or more
nodes with a bunch of disks. But it would definately scale better and
provide higher availability when done right. Even if you would only use
Ceph for "block" devices. It would also require quite a investment in
time to get used to the quirks of using Ceph in production for large
CephFS workloads.
Gr. Stefan
[1]: https://docs.ceph.com/docs/master/cephfs/quota/
Thank you very much.
Best regards
Willi
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
| BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
1 Feb
1 Feb
11:39 p.m.
Hi Willi,
Since you still need iSCSI/NFS/Samba for Windows clients, I think it better to have
virtual ZFS storage backed by Ceph (RBD). I have experience of running FreeNAS virtually
with some volumes for making ZFS pool. The performance is pretty satisfying, almost 10Gbps
iSCSI throughput on 2x 10GbE compute server with 8 volumes of ZFS pool.
Best regards,
1544
days inactive
1546
days old
2 comments
3 participants
participants (3)
-
mrxlazuardin＠gmail.com -
Stefan Kooman -
Willi Schiegel