Hi, I just noticed an interesting behavior that I didn't find documented anywhere. Normally, CephFS stores backtraces in the both default data pool and the actual data pool a file's layout points to (if different): $ for pool in cephfs2_data cephfs2_data_hr3 cephfs2_data_hec5.4; do echo "# Pool: $pool" rados -p $pool listxattr 100032d5676.00000000 done # Pool: cephfs2_data parent # Pool: cephfs2_data_hr3 error getting xattr set cephfs2_data_hr3/100032d5676.00000000: (2) No such file or directory # Pool: cephfs2_data_hec5.4 layout parent However, I found that some files have backtrace objects in another pool: $ for pool in cephfs2_data cephfs2_data_hr3 cephfs2_data_hec5.4; do echo "# Pool: $pool" rados -p $pool listxattr 100032c464a.00000000 done # Pool: cephfs2_data parent # Pool: cephfs2_data_hr3 parent # Pool: cephfs2_data_hec5.4 layout parent Apparently, this happens when you create a file in a directory that is assigned to a non-default pool, and then change the pool before writing any data into the object. In other words, any pool that a file is *ever* assigned to after being created empty, ends up with a backtrace object. This is visible in the backtrace object itself: rados -p cephfs2_data getxattr 100032c464a.00000000 parent \ | head -c -1 \ | ceph-dencoder type inode_backtrace_t import - decode dump_json { "ino": 1099564861002, "ancestors": [ <snip> ], "pool": 14, "old_pools": [ 11, 12 ] } Here pool 11 is the default pool, pool 14 is the desired pool, and pool 12 is the pool of the directory layout for the directory the file was created in (before being reassigned prior to writing data). I found this rather surprising, as I'm migrating a large amount of data to a new pool by setting file layouts before writing out data to the new temporary file [1] and my temporary directory happened to be assigned to a non-default pool, so now I have a bunch of garbage objects in that pool (which happens to be an HDD pool, so this hurts performance). I would have expected the directory's assigned layout pool to be irrelevant if I override it after opening a new file for writing, but apparently this is not the case. I guess this is so existing clients can find the real pool if they have the old pool cached or something? I'm not entirely sure why this behavior exists. It would be nice if there were some way to clean this up without having to rewrite all the files again (it's terabytes of data). I don't mind if I have to take the FS down for a bit to run a cleanup tool, but doing manual surgery on the backtrace objects and inode omap values seems pretty difficult and error-prone :/ [1] https://gist.github.com/marcan/26cc3ac7241f866dca38916215dd10ff - Hector