Thanks a lot for all your quick responses, I really appreciate that! One of our customers is considering using such a setup for CephFS, so your experience is very valuable here. I guess the most important question would be: did you experience any issues regarding disaster recovery if you faced any? Like the advanced recovery steps from the docs, did they work as well as for a replicated default pool apart from being slower? I vaguely remember that a disaster recovery for a CephFS with additional data pools could be difficult (don't have the details at hand right now), but I guess in this case you would skip that obstacle. So maybe a trade-off worth considering? Thanks again to all responses! Zitat von Loïc Tortay <tortay@cc.in2p3.fr>:
On 28/10/2026 08:37, Eugen Block wrote:
Hi,
just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community.
Hello, One of our CephFS clusters has such a filesystem since about 2021 (cluster installed with Pacific RPMs, now cephadm adopted Squid).
It's for an HTC use case, with lots of files (about 450 millions): --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 10 PiB 5.4 PiB 5.1 PiB 5.1 PiB 48.49 TOTAL 10 PiB 5.4 PiB 5.1 PiB 5.1 PiB 48.49
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_EC_data 4 8192 3.7 PiB 1.42G 4.8 PiB 51.04 3.7 PiB cephfs_metadata 6 256 35 GiB 76.44M 104 GiB 0 1.5 PiB
(it's a 8+2 EC pool)
Gathering POSIX metadata for our accounting is (much) slower than on our main CephFS cluster with a replicated "default pool" (and metadata on Flash) but otherwise it works well enough for the users needs.
Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |