Hi, I totally agree that "--force" isn't a coincidence and there must be some reasoning behind it. That's why I started this thread, very interesting responses so far. And I understand your point, we'll have to discuss this with the customer anyway, it just came up yesterday for the first time. So nothing has been decided yet. Thanks! Zitat von Alexander Patrakov <patrakov@gmail.com>:
Hello Eugen,
The documented disaster recovery steps work. Yet, I am not sure if I would recommend going with the described setup. There must be a reason it requires --force, and I am not sure if Ceph upstream exercises such pools during their QA test suite runs.
What I usually recommend is simple enough:
* The metadata pool should be on SSDs, replicated. * The main data pool should also be on SSDs, replicated. * Before writing any files, add an erasure-coded data pool and set the ceph.dir.layout.pool xattr on the root directory to the name of the EC pool.
Caveats observed so far:
* The main data pool will contain one empty object per file. You might easily end up with millions of such empty objects per PG, and this can slow down recovery or rebalancing. You need something faster than the ubiquitous Micron 7450 Pro. * Someone will always be "clever" enough to set ceph.dir.layout.pool to the main data pool on a subdirectory somewhere deep to get some extra performance from SSDs, without telling others or setting a quota. Result: full SSD OSDs in a few months, disaster recovery.
The setup with an erasure-coded main data pool is immune to the second failure mode. It's up to you whether you prefer to rely on sysadmin training or make the situation structurally impossible. I have not tested whether Ceph tolerates a quota set directly on any of its data pools.
On Wed, Oct 28, 2026 at 7:13 PM Eugen Block <ceph-users@ceph.io> wrote:
Thanks a lot for all your quick responses, I really appreciate that!
One of our customers is considering using such a setup for CephFS, so your experience is very valuable here. I guess the most important question would be: did you experience any issues regarding disaster recovery if you faced any? Like the advanced recovery steps from the docs, did they work as well as for a replicated default pool apart from being slower?
I vaguely remember that a disaster recovery for a CephFS with additional data pools could be difficult (don't have the details at hand right now), but I guess in this case you would skip that obstacle. So maybe a trade-off worth considering?
Thanks again to all responses!
Zitat von Loïc Tortay <tortay@cc.in2p3.fr>:
On 28/10/2026 08:37, Eugen Block wrote:
Hi,
just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community.
Hello, One of our CephFS clusters has such a filesystem since about 2021 (cluster installed with Pacific RPMs, now cephadm adopted Squid).
It's for an HTC use case, with lots of files (about 450 millions): --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 10 PiB 5.4 PiB 5.1 PiB 5.1 PiB 48.49 TOTAL 10 PiB 5.4 PiB 5.1 PiB 5.1 PiB 48.49
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_EC_data 4 8192 3.7 PiB 1.42G 4.8 PiB 51.04 3.7 PiB cephfs_metadata 6 256 35 GiB 76.44M 104 GiB 0 1.5 PiB
(it's a 8+2 EC pool)
Gathering POSIX metadata for our accounting is (much) slower than on our main CephFS cluster with a replicated "default pool" (and metadata on Flash) but otherwise it works well enough for the users needs.
Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov