I have such a file system which is not yet actively used. Its main purpose is storing ~30GiB images. Nothing else. I did a couple of small tests on it (storing large ISO files) and throughput was actually surprisingly good. I'd assume that putting a lot of small files on this FS yield very bad performance though. Your question makes me wonder if I shouldn't just recreate this filesystem the as per best practices anyway. If you want, I can do some quick tests on it since it's not in use anyway. Wannes ________________________________ From: Eugen Block <ceph-users@ceph.io> Sent: Wednesday, October 28, 2026 08:37 To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] CephFS erasure-coded default data pool Hi, just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community. Thanks Eugen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, we operate such a file system too since 2017/2018, in an HTC cluster, actually. As our workloads are commonly write-once-read-many, and reading happens in streaming mode / large chunks (with random read workloads every now and then), this works quite well for us. You can find more details in this older presentation: https://indico.cern.ch/event/765214/contributions/3517132/ The cluster is by now upgraded to 2.1 PiB (gross storage), mixing 4 TB, 12 TB and 16 TB disks, still running fine. In case you want more details, feel free to ask (it's a k = 4, m = 2 setup as described in the linked presentation). Cheers, Oliver Am 28.10.26 um 08:37 schrieb Eugen Block:

Hi,

just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community.

Thanks Eugen

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

Thanks a lot for all your quick responses, I really appreciate that! One of our customers is considering using such a setup for CephFS, so your experience is very valuable here. I guess the most important question would be: did you experience any issues regarding disaster recovery if you faced any? Like the advanced recovery steps from the docs, did they work as well as for a replicated default pool apart from being slower? I vaguely remember that a disaster recovery for a CephFS with additional data pools could be difficult (don't have the details at hand right now), but I guess in this case you would skip that obstacle. So maybe a trade-off worth considering? Thanks again to all responses! Zitat von Loïc Tortay <tortay@cc.in2p3.fr>:

On 28/10/2026 08:37, Eugen Block wrote:

...

Hi,

just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community.

Hello, One of our CephFS clusters has such a filesystem since about 2021 (cluster installed with Pacific RPMs, now cephadm adopted Squid).

It's for an HTC use case, with lots of files (about 450 millions): --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 10 PiB 5.4 PiB 5.1 PiB 5.1 PiB 48.49 TOTAL 10 PiB 5.4 PiB 5.1 PiB 5.1 PiB 48.49

--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_EC_data 4 8192 3.7 PiB 1.42G 4.8 PiB 51.04 3.7 PiB cephfs_metadata 6 256 35 GiB 76.44M 104 GiB 0 1.5 PiB

(it's a 8+2 EC pool)

Gathering POSIX metadata for our accounting is (much) slower than on our main CephFS cluster with a replicated "default pool" (and metadata on Flash) but otherwise it works well enough for the users needs.

Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |

Hi, I totally agree that "--force" isn't a coincidence and there must be some reasoning behind it. That's why I started this thread, very interesting responses so far. And I understand your point, we'll have to discuss this with the customer anyway, it just came up yesterday for the first time. So nothing has been decided yet. Thanks! Zitat von Alexander Patrakov <patrakov@gmail.com>:

Hello Eugen,

The documented disaster recovery steps work. Yet, I am not sure if I would recommend going with the described setup. There must be a reason it requires --force, and I am not sure if Ceph upstream exercises such pools during their QA test suite runs.

What I usually recommend is simple enough:

* The metadata pool should be on SSDs, replicated. * The main data pool should also be on SSDs, replicated. * Before writing any files, add an erasure-coded data pool and set the ceph.dir.layout.pool xattr on the root directory to the name of the EC pool.

Caveats observed so far:

* The main data pool will contain one empty object per file. You might easily end up with millions of such empty objects per PG, and this can slow down recovery or rebalancing. You need something faster than the ubiquitous Micron 7450 Pro. * Someone will always be "clever" enough to set ceph.dir.layout.pool to the main data pool on a subdirectory somewhere deep to get some extra performance from SSDs, without telling others or setting a quota. Result: full SSD OSDs in a few months, disaster recovery.

The setup with an erasure-coded main data pool is immune to the second failure mode. It's up to you whether you prefer to rely on sysadmin training or make the situation structurally impossible. I have not tested whether Ceph tolerates a quota set directly on any of its data pools.

On Wed, Oct 28, 2026 at 7:13 PM Eugen Block <ceph-users@ceph.io> wrote:

...

Thanks a lot for all your quick responses, I really appreciate that!

One of our customers is considering using such a setup for CephFS, so your experience is very valuable here. I guess the most important question would be: did you experience any issues regarding disaster recovery if you faced any? Like the advanced recovery steps from the docs, did they work as well as for a replicated default pool apart from being slower?

I vaguely remember that a disaster recovery for a CephFS with additional data pools could be difficult (don't have the details at hand right now), but I guess in this case you would skip that obstacle. So maybe a trade-off worth considering?

Thanks again to all responses!

Zitat von Loïc Tortay <tortay@cc.in2p3.fr>:

...

On 28/10/2026 08:37, Eugen Block wrote:

...

Hi,

just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community.

Hello, One of our CephFS clusters has such a filesystem since about 2021 (cluster installed with Pacific RPMs, now cephadm adopted Squid).

It's for an HTC use case, with lots of files (about 450 millions): --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 10 PiB 5.4 PiB 5.1 PiB 5.1 PiB 48.49 TOTAL 10 PiB 5.4 PiB 5.1 PiB 5.1 PiB 48.49

--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_EC_data 4 8192 3.7 PiB 1.42G 4.8 PiB 51.04 3.7 PiB cephfs_metadata 6 256 35 GiB 76.44M 104 GiB 0 1.5 PiB

(it's a 8+2 EC pool)

Gathering POSIX metadata for our accounting is (much) slower than on our main CephFS cluster with a replicated "default pool" (and metadata on Flash) but otherwise it works well enough for the users needs.

Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Alexander Patrakov