Hi Jorge, On 4/6/23 07:09, Jorge Garcia wrote:
We have a ceph cluster with a cephfs filesystem that we use mostly for backups. When I do a "ceph -s" or a "ceph df", it reports lots of space:
data: pools: 3 pools, 4104 pgs objects: 1.09 G objects, 944 TiB usage: 1.5 PiB used, 1.0 PiB / 2.5 PiB avail
GLOBAL: SIZE AVAIL RAW USED %RAW USED 2.5 PiB 1.0 PiB 1.5 PiB 59.76 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 2 944 TiB 87.63 133 TiB 880988429 cephfs_metadata 3 128 MiB 0 62 TiB 206535313 .rgw.root 4 0 B 0 62 TiB 0
The whole thing consists of 2 pools: metadata (regular default replication) and data (erasure k:5 m:2). The global raw space reports 2.5 PiB total, with 1.0 PiB still available. But, when the ceph filesystem is mounted, it only reports 1.1 PB total, and the filesystem is almost full:
Filesystem Size Used Avail Use% Mounted on x.x.x.x:yyyy:/ 1.1P 944T 134T 88% /backups
So, where is the rest of my space? Or what am I missing?
What's ceph version you are using ? The late cephs' output will be like: $ ./bin/ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 303 GiB 288 GiB 15 GiB 15 GiB 4.98 TOTAL 303 GiB 288 GiB 15 GiB 15 GiB 4.98 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 577 KiB 2 1.7 MiB 0 95 GiB cephfs.a.meta 2 16 28 MiB 29 84 MiB 0.03 95 GiB cephfs.a.data 3 32 4.0 GiB 1.02k 12 GiB 4.04 95 GiB $ df -h Filesystem Size Used Avail Use% Mounted on 192.168.0.104:40554,192.168.0.104:40556,192.168.0.104:40558:/ 99G 4.0G 95G 5% /mnt/kcephfs From the mountpoint "/mnt/kcephfs" the used disk space is 4GB, which I just created a 4GB size file. And from `ceph df` the STORED is 4GB too while the USED is 12GB, which the pool is replica x3. For you with data (erasure k:5 m:2) the 1.1P should be just for efficient data STORED, does not include the compression, allocation and other overheads. You can get more detail from [1], [2] and [3]. [1] https://tracker.ceph.com/issues/20870 [2] https://tracker.ceph.com/issues/22159 [3] https://github.com/ceph/ceph/commit/db5c5cce5513080ab81c69a705440174645332a8 Thanks - Xiubo
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io