Why is my cephfs almostfull?
We have a ceph cluster with a cephfs filesystem that we use mostly for backups. When I do a "ceph -s" or a "ceph df", it reports lots of space: data: pools: 3 pools, 4104 pgs objects: 1.09 G objects, 944 TiB usage: 1.5 PiB used, 1.0 PiB / 2.5 PiB avail GLOBAL: SIZE AVAIL RAW USED %RAW USED 2.5 PiB 1.0 PiB 1.5 PiB 59.76 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 2 944 TiB 87.63 133 TiB 880988429 cephfs_metadata 3 128 MiB 0 62 TiB 206535313 .rgw.root 4 0 B 0 62 TiB 0 The whole thing consists of 2 pools: metadata (regular default replication) and data (erasure k:5 m:2). The global raw space reports 2.5 PiB total, with 1.0 PiB still available. But, when the ceph filesystem is mounted, it only reports 1.1 PB total, and the filesystem is almost full: Filesystem Size Used Avail Use% Mounted on x.x.x.x:yyyy:/ 1.1P 944T 134T 88% /backups So, where is the rest of my space? Or what am I missing? Thanks!
Hi Jorge, On 4/6/23 07:09, Jorge Garcia wrote:
We have a ceph cluster with a cephfs filesystem that we use mostly for backups. When I do a "ceph -s" or a "ceph df", it reports lots of space:
data: pools: 3 pools, 4104 pgs objects: 1.09 G objects, 944 TiB usage: 1.5 PiB used, 1.0 PiB / 2.5 PiB avail
GLOBAL: SIZE AVAIL RAW USED %RAW USED 2.5 PiB 1.0 PiB 1.5 PiB 59.76 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 2 944 TiB 87.63 133 TiB 880988429 cephfs_metadata 3 128 MiB 0 62 TiB 206535313 .rgw.root 4 0 B 0 62 TiB 0
The whole thing consists of 2 pools: metadata (regular default replication) and data (erasure k:5 m:2). The global raw space reports 2.5 PiB total, with 1.0 PiB still available. But, when the ceph filesystem is mounted, it only reports 1.1 PB total, and the filesystem is almost full:
Filesystem Size Used Avail Use% Mounted on x.x.x.x:yyyy:/ 1.1P 944T 134T 88% /backups
So, where is the rest of my space? Or what am I missing?
What's ceph version you are using ? The late cephs' output will be like: $ ./bin/ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 303 GiB 288 GiB 15 GiB 15 GiB 4.98 TOTAL 303 GiB 288 GiB 15 GiB 15 GiB 4.98 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 577 KiB 2 1.7 MiB 0 95 GiB cephfs.a.meta 2 16 28 MiB 29 84 MiB 0.03 95 GiB cephfs.a.data 3 32 4.0 GiB 1.02k 12 GiB 4.04 95 GiB $ df -h Filesystem Size Used Avail Use% Mounted on 192.168.0.104:40554,192.168.0.104:40556,192.168.0.104:40558:/ 99G 4.0G 95G 5% /mnt/kcephfs From the mountpoint "/mnt/kcephfs" the used disk space is 4GB, which I just created a 4GB size file. And from `ceph df` the STORED is 4GB too while the USED is 12GB, which the pool is replica x3. For you with data (erasure k:5 m:2) the 1.1P should be just for efficient data STORED, does not include the compression, allocation and other overheads. You can get more detail from [1], [2] and [3]. [1] https://tracker.ceph.com/issues/20870 [2] https://tracker.ceph.com/issues/22159 [3] https://github.com/ceph/ceph/commit/db5c5cce5513080ab81c69a705440174645332a8 Thanks - Xiubo
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Jorge, firstly, it would be really helpful if you would not truncate output of ceph status or omit output of commands you refer to, like ceph df. We have seen way too many examples where the clue was in the omitted part. Without any information, my bets in order are (according to many cases of this type on this list): - the pool does not actually use all OSDs - you have an imbalance in your cluster and at least one OSD/failure domain is 85-90% full - you have a huge amount of small files/objects in the data pool and suffer from allocation amplification - you have a quota on the data pool - there is an error in the crush map If you provide a reasonable amount of information, like full output of 'ceph status', 'ceph df detail' and 'ceph osd df tree' (please use https://pastebin.com/), I'm willing to give it a second try. You may also - before replying - investigate a bit on your own to see if there is any potentially relevant information *additional* to the full output of these commands. Anything else that looks odd. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Jorge Garcia <jgarcia@soe.ucsc.edu> Sent: Thursday, April 6, 2023 1:09 AM To: ceph-users Subject: [ceph-users] Why is my cephfs almostfull? We have a ceph cluster with a cephfs filesystem that we use mostly for backups. When I do a "ceph -s" or a "ceph df", it reports lots of space: data: pools: 3 pools, 4104 pgs objects: 1.09 G objects, 944 TiB usage: 1.5 PiB used, 1.0 PiB / 2.5 PiB avail GLOBAL: SIZE AVAIL RAW USED %RAW USED 2.5 PiB 1.0 PiB 1.5 PiB 59.76 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 2 944 TiB 87.63 133 TiB 880988429 cephfs_metadata 3 128 MiB 0 62 TiB 206535313 .rgw.root 4 0 B 0 62 TiB 0 The whole thing consists of 2 pools: metadata (regular default replication) and data (erasure k:5 m:2). The global raw space reports 2.5 PiB total, with 1.0 PiB still available. But, when the ceph filesystem is mounted, it only reports 1.1 PB total, and the filesystem is almost full: Filesystem Size Used Avail Use% Mounted on x.x.x.x:yyyy:/ 1.1P 944T 134T 88% /backups So, where is the rest of my space? Or what am I missing? Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Frank Schilder
-
Jorge Garcia
-
Xiubo Li