We sync the file system without preserving hard links. But we take snapshots after each sync, so I guess deleting files which are still in snapshots can also be in the stray directories? [root@mds02 ~]# ceph daemon mds.mds02 perf dump | grep -i 'stray\|purge'     "finisher-PurgeQueue": {         "num_strays": 990153,         "num_strays_delayed": 32,         "num_strays_enqueuing": 0,         "strays_created": 753278,         "strays_enqueued": 650603,         "strays_reintegrated": 0,         "strays_migrated": 0, num_strays is indeed close to a million On 10/09/2019 12:42, Burkhard Linke wrote: > Hi, > > > do you use hard links in your workload? The 'no space left on device' > message may also refer to too many stray files. Strays are either > files that are to be deleted (e.g. the purge queue), but also files > which are deleted, but hard links are still pointing to the same > content. Since cephfs does not use an indirect layer between inodes > and data, and the data chunks are named after the inode id, removing > the original file will leave stray entries since cephfs is not able to > rename the underlying rados objects. > > > There are 10 hidden directories for stray files, and given a maximum > size of 100.000 entries you can store only up to 1 million entries. I > don't know exactly how entries are distributed among the 10 > directories, so the limit may be reached earlier for a single stray > directory. The performance counters contains some values for stray, so > they are easy to check. The daemonperf output also shows the current > value. > > > The problem of the upper limit of directory entries was solved by > directory fragmentation, so you should check whether fragmentation is > allowed in your filesystem. You can also try to increase the upper > directory entry limit, but this might lead to other problems (too > large rados omap objects....). > > > Regards, > > Burkhard > >