ZFS on RBD seems like a waste, and overkill. A redundant storage solution on top of a
redundant storage solution?
You can have multiple file systems within CephFS, the thing to note is that each CephFS
MUST have a SEPARATE active MDS.
For failover, each should have a secondary MDS, and these also need to be separate
(preferably running in standby-replay mode). Each MDS instance can only handle one
responsibility, for one file system. Each file system also uses 2 pools; one for metadata
(think filenames, file properties, and the directory tree), and one for the file data
The containerization present by default in Octopus should make running many MDSs easier.
We run 3 CephFS file systems from our primary cluster. This uses 6 MDSs, and 6 pools. We
assigned the metadata pools to our SSDs (using CRUSH rules) for performance.
You might also work with your users on switching to an Object Storage paradigm (think S3),
as RadosGW has some nice disaster recovery features.
Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International, Inc.
From: Willi Schiegel [mailto:email@example.com]
Sent: Sunday, January 26, 2020 4:24 AM
Subject: [ceph-users] General question CephFS or RBD
I have a HW RAID based 240 TB data pool with about 200 million files for users in a
scientific institution. Data sizes range from tiny parameter files for scientific
calculations and experiments to huge images of brain scans. There are group directories,
home directories, Windows roaming profile directories organized in ZFS pools on Solaris
operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients.
I would like to switch to CephFS because of the flexibility and expandability but I cannot
find any recommendations for which storage backend would be suitable for all the
functionality we have.
Since I like the features of ZFS like immediate snapshots of very large data pools, quotas
for each file system within hierarchical data trees and dynamic expandability by simply
adding new disks or disk images without manual resizing would it be a good idea to create
RBD images, map them onto the file servers and create zpools on the mapped images? I know
that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk?
Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data
folders and for the user's home directory folders for example or do I have to have
everything in one single file space?
Maybe someone can share his or her field experience?
Thank you very much.
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to