15 Sep
2020
15 Sep
'20
5:15 a.m.
Hello.
I'm using the Nautilus Ceph version for some huge folder with approximately 1.7TB of
files.I created the filesystem and started to copy files via rsync.
However, I've had to stop the process, because Ceph shows me that the new size of the
folder is almost 6TB. I double checked the replicated size and it is 2. I double checked
the rsync options and I didn't copy the files followed by symlinks.
How would it be possible to explain the extreme difference between the size of the
original folder and CephFS?
15 Sep
15 Sep
7:09 a.m.
What about hardlinks, are there any of those? Are there lots of
directories or tiny (<4k) files?
Also, size=2 is not very safe. You want size=3, min_size=2 if you are
doing replication.
On Mon, Sep 14, 2020 at 6:15 PM <fotofors(a)gmail.com> wrote:
>
> Hello.
>
> I'm using the Nautilus Ceph version for some huge folder with approximately 1.7TB
of files.I created the filesystem and started to copy files via rsync.
>
> However, I've had to stop the process, because Ceph shows me that the new size of
the folder is almost 6TB. I double checked the replicated size and it is 2. I double
checked the rsync options and I didn't copy the files followed by symlinks.
>
> How would it be possible to explain the extreme difference between the size of the
original folder and CephFS?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:35 p.m.
On 2020-09-15 02:09, Nathan Fish wrote:
What about hardlinks, are there any of those? Are
there lots of
directories or tiny (<4k) files?
The default allocation size of bluestore depends on the disk type. We
have this in our config:
# 4096 B instead of 16K (SSD) / 64K (HDD) to avoid large overhead for #
small (cephfs) files
bluestore_min_alloc_size_ssd = 4096
bluestore_min_alloc_size_hdd = 4096
If you have a lot of small files they will consume way more space on
CephFS with the default alloc size for realeases older than Octopus and
or using hdd drives.
The bad news: you cannot change that after the fact. You have to
reprovision your OSDs with a smaller alloc size if you want to change
that. On the upside: you can use default Ceph recovery mechanism to do
that without downtime.
Also, size=2 is not very safe. You want size=3,
min_size=2 if you are
doing replication.
^^ This, really!
Gr. Stefan
17 Sep
17 Sep
7:30 p.m.
Yes, I know this option isn't safe, however, in my current situation, I can't
increase it.
I probably have some files under 4K, however, when I cleaned zero files I didn't saw
any changes in statistics. My current `ceph df details` below:
# ceph df detail
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 8.0 TiB 1.6 TiB 6.3 TiB 6.4 TiB 80.32
TOTAL 8.0 TiB 1.6 TiB 6.3 TiB 6.4 TiB 80.32
--- POOLS ---
POOL ID STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP)
%USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR
device_health_metrics 1 0 B 0 B 0 B 0 0 B 0 B 0 B
0 126 GiB N/A N/A 0 0 B 0 B
station_data 9 1.6 TiB 1.6 TiB 0 B 27.90M 6.3 TiB 6.3 TiB 0 B
94.44 190 GiB N/A 2.5 TiB 27.90M 0 B 0 B
station_data_metadata 10 15 GiB 178 MiB 15 GiB 82.11k 30 GiB 356 MiB 29 GiB
7.24 190 GiB N/A 8 GiB 82.11k 0 B
As you can see, the field STORED is 1.6TB. However, the DATA is 6.3.
Does it possible to determine why the files consume so many spaces? Is I am wrong, when I
calculate that size=2 means STORED*2, hence the DATA should be 3.2TB?
15 Sep
15 Sep
7:22 a.m.
I suggest trying the rsync --sparse option. Typically, qcow2 files (tend to be large) are
sparse files. Without the sparse option, the files expand in their destination.
September 14, 2020 6:15 PM, fotofors(a)gmail.com wrote:
> Hello.
>
> I'm using the Nautilus Ceph version for some huge folder with approximately 1.7TB
of files.I
> created the filesystem and started to copy files via rsync.
>
> However, I've had to stop the process, because Ceph shows me that the new size of
the folder is
> almost 6TB. I double checked the replicated size and it is 2. I double checked the
rsync options
> and I didn't copy the files followed by symlinks.
>
> How would it be possible to explain the extreme difference between the size of the
original folder
> and CephFS?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1325
days inactive
1328
days old
4 comments
4 participants
participants (4)
-
fotofors＠gmail.com -
Nathan Fish -
Stefan Kooman -
tri＠postix.net