Disk consume for CephFS
Hello. I'm using the Nautilus Ceph version for some huge folder with approximately 1.7TB of files.I created the filesystem and started to copy files via rsync. However, I've had to stop the process, because Ceph shows me that the new size of the folder is almost 6TB. I double checked the replicated size and it is 2. I double checked the rsync options and I didn't copy the files followed by symlinks. How would it be possible to explain the extreme difference between the size of the original folder and CephFS?
What about hardlinks, are there any of those? Are there lots of directories or tiny (<4k) files? Also, size=2 is not very safe. You want size=3, min_size=2 if you are doing replication. On Mon, Sep 14, 2020 at 6:15 PM <fotofors@gmail.com> wrote:
Hello.
I'm using the Nautilus Ceph version for some huge folder with approximately 1.7TB of files.I created the filesystem and started to copy files via rsync.
However, I've had to stop the process, because Ceph shows me that the new size of the folder is almost 6TB. I double checked the replicated size and it is 2. I double checked the rsync options and I didn't copy the files followed by symlinks.
How would it be possible to explain the extreme difference between the size of the original folder and CephFS? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2020-09-15 02:09, Nathan Fish wrote:
What about hardlinks, are there any of those? Are there lots of directories or tiny (<4k) files?
The default allocation size of bluestore depends on the disk type. We have this in our config: # 4096 B instead of 16K (SSD) / 64K (HDD) to avoid large overhead for # small (cephfs) files bluestore_min_alloc_size_ssd = 4096 bluestore_min_alloc_size_hdd = 4096 If you have a lot of small files they will consume way more space on CephFS with the default alloc size for realeases older than Octopus and or using hdd drives. The bad news: you cannot change that after the fact. You have to reprovision your OSDs with a smaller alloc size if you want to change that. On the upside: you can use default Ceph recovery mechanism to do that without downtime.
Also, size=2 is not very safe. You want size=3, min_size=2 if you are doing replication.
^^ This, really! Gr. Stefan
Yes, I know this option isn't safe, however, in my current situation, I can't increase it. I probably have some files under 4K, however, when I cleaned zero files I didn't saw any changes in statistics. My current `ceph df details` below: # ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 8.0 TiB 1.6 TiB 6.3 TiB 6.4 TiB 80.32 TOTAL 8.0 TiB 1.6 TiB 6.3 TiB 6.4 TiB 80.32 --- POOLS --- POOL ID STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR device_health_metrics 1 0 B 0 B 0 B 0 0 B 0 B 0 B 0 126 GiB N/A N/A 0 0 B 0 B station_data 9 1.6 TiB 1.6 TiB 0 B 27.90M 6.3 TiB 6.3 TiB 0 B 94.44 190 GiB N/A 2.5 TiB 27.90M 0 B 0 B station_data_metadata 10 15 GiB 178 MiB 15 GiB 82.11k 30 GiB 356 MiB 29 GiB 7.24 190 GiB N/A 8 GiB 82.11k 0 B As you can see, the field STORED is 1.6TB. However, the DATA is 6.3. Does it possible to determine why the files consume so many spaces? Is I am wrong, when I calculate that size=2 means STORED*2, hence the DATA should be 3.2TB?
I suggest trying the rsync --sparse option. Typically, qcow2 files (tend to be large) are sparse files. Without the sparse option, the files expand in their destination. September 14, 2020 6:15 PM, fotofors@gmail.com wrote:
Hello.
I'm using the Nautilus Ceph version for some huge folder with approximately 1.7TB of files.I created the filesystem and started to copy files via rsync.
However, I've had to stop the process, because Ceph shows me that the new size of the folder is almost 6TB. I double checked the replicated size and it is 2. I double checked the rsync options and I didn't copy the files followed by symlinks.
How would it be possible to explain the extreme difference between the size of the original folder and CephFS? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
fotofors＠gmail.com
-
Nathan Fish
-
Stefan Kooman
-
tri＠postix.net