On 2020-09-15 02:09, Nathan Fish wrote:
What about hardlinks, are there any of those? Are there lots of directories or tiny (<4k) files?
The default allocation size of bluestore depends on the disk type. We have this in our config: # 4096 B instead of 16K (SSD) / 64K (HDD) to avoid large overhead for # small (cephfs) files bluestore_min_alloc_size_ssd = 4096 bluestore_min_alloc_size_hdd = 4096 If you have a lot of small files they will consume way more space on CephFS with the default alloc size for realeases older than Octopus and or using hdd drives. The bad news: you cannot change that after the fact. You have to reprovision your OSDs with a smaller alloc size if you want to change that. On the upside: you can use default Ceph recovery mechanism to do that without downtime.
Also, size=2 is not very safe. You want size=3, min_size=2 if you are doing replication.
^^ This, really! Gr. Stefan