no snapshots. there are mostly large files yes (few GB)) , but also lots of small ones . large files occupy about 90% of used space. maybe worth to mention for large files, there is about 100-200TB files written/removed daily. We also use a cache space for batch jobs, where inputs are stored in common dirs and hardlinks to them created in job dirs and removed when job is done. This caused some problems a couple of years ago but not any more. Best, Andrej On 9. 6. 2026 18:26, Gregory Farnum wrote:
Do you have any snapshots?
There's also generally some amount of inflation with EC and small files, but I don't think it should reach (7/5.6=)25%, and since you're using 16MB objects I presume most of your files are large. -Greg
On Tue, Jun 9, 2026 at 7:02 AM Andrej Filipčič via ceph-users <ceph-users@ceph.io> wrote:
thanks for this info, but our purge queue is more or less empty "purge_queue": { "pq_executing_ops": 2, "pq_executing_ops_high_water": 13659, "pq_executing": 1, "pq_executing_high_water": 500, "pq_executed_ops": 127095, "pq_executed": 50471, "pq_item_in_journal": 2 },
anyway, I raised the parameters to see if it helps.
Best, Andrej
On 9. 6. 2026 14:57, Loïc Tortay via ceph-users wrote:
On 09/06/2026 14:10, Andrej Filipcic via ceph-users wrote:
Hi,
I have a problem with large amounts of dark data/orphans on cephfs data pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd).
There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. The layout for most files: ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd"
But there is 1.4PiB discrepancy between the pool and the filesystem which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here.
Hello, We had a similar issue last year with a group of users that created and removed files at a very high rate.
Have you read https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-without... ?
We increased the purge rate parameters (very) aggressively to get back to a comfortable situation (i.e. not a pool w/ near full warnings).
Loïc.
-- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 ------------------------------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------