Hi, I have a problem with large amounts of dark data/orphans on cephfs data pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. The layout for most files: ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd" But there is 1.4PiB discrepancy between the pool and the filesystem which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here. ceph was created with 17.2.6 and runs 19.2.3. Do you might have any suggestions how to fix this? Thanks, Andrej spod01 ~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 11 PiB 1.5 PiB 9.7 PiB 9.7 PiB 86.28 nvme 363 TiB 356 TiB 6.8 TiB 6.8 TiB 1.87 ssd 3.5 TiB 3.0 TiB 478 GiB 478 GiB 13.37 TOTAL 12 PiB 1.9 PiB 9.7 PiB 9.7 PiB 83.68 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 850 MiB 213 2.5 GiB 0 178 TiB cephfs_data 2 256 28 GiB 134.71M 86 GiB 2.86 972 GiB cephfs_metadata 3 64 66 GiB 6.66M 199 GiB 6.38 972 GiB cephfs_data_echdd 19 4096 7.0 PiB 594.70M 9.7 PiB 94.90 388 TiB cephfs_data_ecnvme 26 64 1.2 TiB 2.84M 1.7 TiB 0.48 251 TiB ... -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------