Hi, Not sure if it was mentioned but also you could check the following: 1. Snapshots Snapshots can consume a significant amount of space without being immediately obvious. They preserve the state of the filesystem at various points in time. List Snapshots: Use the "*ceph fs subvolume snapshot ls*" command to list all snapshots for a given subvolume. You can also use "*ceph fs ls*" to list the filesystems and then inspect each one for snapshots. 2. Replication Overhead (as Etienne M. has mentioned) Depending on your pool's replication or erasure coding settings, the actual storage usage could be multiple times the size of the stored data. Check Pool Settings: Use "ceph osd pool get <pool_name> size" to check the replication factor. For erasure-coded pools, check with "ceph osd pool get <pool_name> erasure_code_profile" . 3. Use ceph df and ceph osd pool stats These commands give you an overview of storage utilization and I/O statistics, respectively. Disk Usage: "ceph df" can help identify how much space is being used by different pools. Pool Stats: "ceph osd pool stats" might give insights into read/write operations, which could hint at heavy snapshot usage or other anomalies. Orphaned data blocks, which are not linked to any file but still occupy space, can also be a culprit. Just my 2 cents, Bogdan Velica. On Wed, Mar 13, 2024 at 9:21 AM Etienne Menguy <etienne.menguy@ubisoft.com> wrote:

Hi,

Check your replication/EC configuration. How do you get your different sizes/usages?

Étienne ________________________________ From: Thorne Lawler <thorne@ddns.com.au> Sent: Wednesday, 13 March 2024 03:58 To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] CephFS space usage

[Some people who received this message don't often get email from thorne@ddns.com.au. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Hi everyone!

My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB usage for the data pool, but there are only 5.5TB of files in it. There are fewer than 100 files on the filesystem in total, so where is all that space going?

How can I analyze my cephfs to understand what is using that space, and if possible, how can I reclaim that space?

Thank you.

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. / _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Igor, Etienne, Bogdan, The system is a four node cluster. Each node has 12 3.8TB SSDs, and each SSD is an OSD. I have not defined any separate DB / WAL devices - this cluster is mostly at cephadm defaults. Everything is currently configured to have x3 replicas. The system also does various RBD workloads from other pools. There are no subvolumes and no snapshots on the CephFS volume in question. The CephFS volume I am concerned about is called 'shared'. For the purposes of this question I am omitting information about the other pools. [root@san1 ~]# rados df POOL_NAME                     USED  OBJECTS  CLONES    COPIES MISSING_ON_PRIMARY  UNFOUND  DEGRADED       RD_OPS RD      WR_OPS       WR  USED COMPR  UNDER COMPR cephfs.shared.data          41 TiB  3834689       0 11504067                   0        0         0   3219785418 175 TiB  9330001764  229 TiB     7.0 MiB       12 MiB cephfs.shared.meta         757 MiB       85       0 255                   0        0         0   5306018840    26 TiB  9170232158   24 TiB         0 B          0 B total_objects    13169948 total_used       132 TiB total_avail      33 TiB total_space      166 TiB [root@san1 ~]# ceph df detail --- RAW STORAGE --- CLASS     SIZE   AVAIL     USED  RAW USED  %RAW USED ssd    166 TiB  33 TiB  132 TiB   132 TiB      79.82 TOTAL  166 TiB  33 TiB  132 TiB   132 TiB      79.82 --- POOLS --- POOL                       ID  PGS   STORED   (DATA)   (OMAP) OBJECTS     USED   (DATA)   (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR cephfs.shared.meta          3   32  251 MiB  208 MiB   42 MiB       84  752 MiB  625 MiB  127 MiB      0    3.4 TiB            N/A          N/A    N/A         0 B          0 B cephfs.shared.data          4  512   14 TiB   14 TiB      0 B 3.83M   41 TiB   41 TiB      0 B  79.90    3.4 TiB N/A          N/A    N/A     7.0 MiB       12 MiB [root@san1 ~]# ceph osd pool get cephfs.shared.data size size: 3 ...however running 'du' in the root directory of the 'shared' volume says: # du -sh . 5.5T    . So yeah - 14TB is replicated to 41TB, that's fine, but 14TB is a lot more than 5.5TB, so... where is that space going? On 14/03/2024 2:09 am, Igor Fedotov wrote:

Hi Thorn,

could you please share the output of "ceph df detail" command representing the problem?

And please give an overview of your OSD layout - amount of OSDs, shared or dedicated DB/WAL, main and DB volume sizes.

Thanks,

Igor

On 3/13/2024 5:58 AM, Thorne Lawler wrote:

...

Hi everyone!

My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB usage for the data pool, but there are only 5.5TB of files in it. There are fewer than 100 files on the filesystem in total, so where is all that space going?

How can I analyze my cephfs to understand what is using that space, and if possible, how can I reclaim that space?

Thank you.

--

Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

Hey All, It might be easier to check using cephfs dir stats using getfattr, ex. getfattr -n ceph.dir.rentries /path/to/dir Regards, Bailey

-----Original Message----- From: Igor Fedotov <igor.fedotov@croit.io> Sent: March 14, 2024 1:37 PM To: Thorne Lawler <thorne@ddns.com.au>; ceph-users@ceph.io; etienne.menguy@ubisoft.com; vbogdan@gmail.com Subject: [ceph-users] Re: CephFS space usage

Thorn,

you might want to assess amount of files on the mounted fs by runnning "du -h | wc". Does it differ drastically from amount of objects in the pool = ~3.8 M?

And just in case - please run "rados lssnap -p cephfs.shared.data".

Thanks,

Igor

On 3/14/2024 1:42 AM, Thorne Lawler wrote:

...

Igor, Etienne, Bogdan,

The system is a four node cluster. Each node has 12 3.8TB SSDs, and each SSD is an OSD.

I have not defined any separate DB / WAL devices - this cluster is mostly at cephadm defaults.

Everything is currently configured to have x3 replicas.

The system also does various RBD workloads from other pools.

There are no subvolumes and no snapshots on the CephFS volume in

question.

...

The CephFS volume I am concerned about is called 'shared'. For the purposes of this question I am omitting information about the other pools.

[root@san1 ~]# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD

WR_OPS WR

...

USED COMPR UNDER COMPR cephfs.shared.data 41 TiB 3834689 0 11504067 0 0 0 3219785418 175 TiB 9330001764 229 TiB 7.0 MiB 12 MiB cephfs.shared.meta 757 MiB 85 0 255 0 0 0 5306018840 26 TiB 9170232158 24 TiB 0 B 0 B

total_objects 13169948 total_used 132 TiB total_avail 33 TiB total_space 166 TiB

[root@san1 ~]# ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 166 TiB 33 TiB 132 TiB 132 TiB 79.82 TOTAL 166 TiB 33 TiB 132 TiB 132 TiB 79.82

--- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR cephfs.shared.meta 3 32 251 MiB 208 MiB 42 MiB 84 752 MiB 625 MiB 127 MiB 0 3.4 TiB N/A N/A N/A 0 B 0 B cephfs.shared.data 4 512 14 TiB 14 TiB 0 B 3.83M 41 TiB 41 TiB 0 B 79.90 3.4 TiB N/A N/A N/A 7.0 MiB 12 MiB

[root@san1 ~]# ceph osd pool get cephfs.shared.data size size: 3

...however running 'du' in the root directory of the 'shared' volume says:

# du -sh . 5.5T .

So yeah - 14TB is replicated to 41TB, that's fine, but 14TB is a lot more than 5.5TB, so... where is that space going?

On 14/03/2024 2:09 am, Igor Fedotov wrote:

...

Hi Thorn,

could you please share the output of "ceph df detail" command representing the problem?

And please give an overview of your OSD layout - amount of OSDs, shared or dedicated DB/WAL, main and DB volume sizes.

Thanks,

Igor

On 3/13/2024 5:58 AM, Thorne Lawler wrote:

...

Hi everyone!

My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB usage for the data pool, but there are only 5.5TB of files in it. There are fewer than 100 files on the filesystem in total, so where is all that space going?

How can I analyze my cephfs to understand what is using that space, and if possible, how can I reclaim that space?

Thank you.

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Igor, Yes. Just a bit. root@pmx101:/mnt/pve/iso# du -h | wc -l 10 root@pmx101:/mnt/pve/iso# du -h 0       ./snippets 0       ./tmp 257M    ./xcp_nfs_sr/2ba36cf5-291a-17d2-b510-db1a295ce0c2 5.5T    ./xcp_nfs_sr/5aacaebb-4469-96f9-729e-fe45eef06a14 5.5T    ./xcp_nfs_sr 0       ./failover_test 11G     ./template/iso 11G     ./template 0       ./xcpiso 5.5T    . root@pmx101:/mnt/pve/iso# du --inodes 1       ./snippets 1       ./tmp 5       ./xcp_nfs_sr/2ba36cf5-291a-17d2-b510-db1a295ce0c2 53      ./xcp_nfs_sr/5aacaebb-4469-96f9-729e-fe45eef06a14 59      ./xcp_nfs_sr 1       ./failover_test 2       ./template/iso 3       ./template 1       ./xcpiso 67      . root@pmx101:/mnt/pve/iso# rados lssnap -p cephfs.shared.data 0 snaps What/where are all the other objects?!? On 15/03/2024 3:36 am, Igor Fedotov wrote:

Thorn,

you might want to assess amount of files on the mounted fs by runnning "du -h | wc". Does it differ drastically from amount of objects in the pool = ~3.8 M?

And just in case - please run "rados lssnap -p cephfs.shared.data".

Thanks,

Igor

On 3/14/2024 1:42 AM, Thorne Lawler wrote:

...

Igor, Etienne, Bogdan,

The system is a four node cluster. Each node has 12 3.8TB SSDs, and each SSD is an OSD.

I have not defined any separate DB / WAL devices - this cluster is mostly at cephadm defaults.

Everything is currently configured to have x3 replicas.

The system also does various RBD workloads from other pools.

There are no subvolumes and no snapshots on the CephFS volume in question.

The CephFS volume I am concerned about is called 'shared'. For the purposes of this question I am omitting information about the other pools.

[root@san1 ~]# rados df POOL_NAME                     USED  OBJECTS  CLONES COPIES  MISSING_ON_PRIMARY  UNFOUND  DEGRADED RD_OPS        RD      WR_OPS       WR  USED COMPR  UNDER COMPR cephfs.shared.data          41 TiB  3834689       0 11504067                   0        0         0 3219785418   175 TiB  9330001764  229 TiB     7.0 MiB 12 MiB cephfs.shared.meta         757 MiB       85       0 255                   0        0         0   5306018840 26 TiB  9170232158   24 TiB         0 B          0 B

total_objects    13169948 total_used       132 TiB total_avail      33 TiB total_space      166 TiB

[root@san1 ~]# ceph df detail --- RAW STORAGE --- CLASS     SIZE   AVAIL     USED  RAW USED  %RAW USED ssd    166 TiB  33 TiB  132 TiB   132 TiB      79.82 TOTAL  166 TiB  33 TiB  132 TiB   132 TiB      79.82

--- POOLS --- POOL                       ID  PGS   STORED   (DATA) (OMAP)  OBJECTS     USED   (DATA)   (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR cephfs.shared.meta          3   32  251 MiB  208 MiB   42 MiB       84  752 MiB  625 MiB  127 MiB      0    3.4 TiB            N/A          N/A    N/A         0 B 0 B cephfs.shared.data          4  512   14 TiB   14 TiB      0 B    3.83M   41 TiB   41 TiB      0 B  79.90    3.4 TiB            N/A          N/A    N/A     7.0 MiB       12 MiB

[root@san1 ~]# ceph osd pool get cephfs.shared.data size size: 3

...however running 'du' in the root directory of the 'shared' volume says:

# du -sh . 5.5T    .

So yeah - 14TB is replicated to 41TB, that's fine, but 14TB is a lot more than 5.5TB, so... where is that space going?

On 14/03/2024 2:09 am, Igor Fedotov wrote:

...

Hi Thorn,

could you please share the output of "ceph df detail" command representing the problem?

And please give an overview of your OSD layout - amount of OSDs, shared or dedicated DB/WAL, main and DB volume sizes.

Thanks,

Igor

On 3/13/2024 5:58 AM, Thorne Lawler wrote:

...

Hi everyone!

My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB usage for the data pool, but there are only 5.5TB of files in it. There are fewer than 100 files on the filesystem in total, so where is all that space going?

How can I analyze my cephfs to understand what is using that space, and if possible, how can I reclaim that space?

Thank you.

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx

-- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

Hi Thorn, so the problem is apparently bound to huge file sizes. I presume they're split into multiple chunks at ceph side hence producing millions of objects. And possibly something is wrong with this mapping. If this pool has no write load at the moment you might want to run the following experiment: 1) put one more huge file to the filesystem, e.g. of 200GB bytes. And note pool stats (through "ceph df detail --format json-pretty) before and after this operation. 2) then remove the file, wait until object count is stabilized (i.e. removal is completed) and learn the final stats. Are there any leaks? What was stored space (object count) increase in the middle of the above procedure? As it looks like compession is (was?) enabled on the pool-in-question it's worth using non-compressible data for the experiment, e.g. generate them through /dev/urandom. Thanks, Igor On 3/15/2024 2:05 AM, Thorne Lawler wrote:

Igor,

Yes. Just a bit.

root@pmx101:/mnt/pve/iso# du -h | wc -l 10 root@pmx101:/mnt/pve/iso# du -h 0       ./snippets 0       ./tmp 257M    ./xcp_nfs_sr/2ba36cf5-291a-17d2-b510-db1a295ce0c2 5.5T    ./xcp_nfs_sr/5aacaebb-4469-96f9-729e-fe45eef06a14 5.5T    ./xcp_nfs_sr 0       ./failover_test 11G     ./template/iso 11G     ./template 0       ./xcpiso 5.5T    . root@pmx101:/mnt/pve/iso# du --inodes 1       ./snippets 1       ./tmp 5       ./xcp_nfs_sr/2ba36cf5-291a-17d2-b510-db1a295ce0c2 53      ./xcp_nfs_sr/5aacaebb-4469-96f9-729e-fe45eef06a14 59      ./xcp_nfs_sr 1       ./failover_test 2       ./template/iso 3       ./template 1       ./xcpiso 67      . root@pmx101:/mnt/pve/iso# rados lssnap -p cephfs.shared.data 0 snaps

What/where are all the other objects?!?

On 15/03/2024 3:36 am, Igor Fedotov wrote:

...

Thorn,

you might want to assess amount of files on the mounted fs by runnning "du -h | wc". Does it differ drastically from amount of objects in the pool = ~3.8 M?

And just in case - please run "rados lssnap -p cephfs.shared.data".

Thanks,

Igor

On 3/14/2024 1:42 AM, Thorne Lawler wrote:

...

Igor, Etienne, Bogdan,

The system is a four node cluster. Each node has 12 3.8TB SSDs, and each SSD is an OSD.

I have not defined any separate DB / WAL devices - this cluster is mostly at cephadm defaults.

Everything is currently configured to have x3 replicas.

The system also does various RBD workloads from other pools.

There are no subvolumes and no snapshots on the CephFS volume in question.

The CephFS volume I am concerned about is called 'shared'. For the purposes of this question I am omitting information about the other pools.

[root@san1 ~]# rados df POOL_NAME                     USED  OBJECTS  CLONES COPIES  MISSING_ON_PRIMARY  UNFOUND  DEGRADED RD_OPS        RD      WR_OPS       WR  USED COMPR  UNDER COMPR cephfs.shared.data          41 TiB  3834689       0 11504067                   0        0         0 3219785418   175 TiB  9330001764  229 TiB     7.0 MiB       12 MiB cephfs.shared.meta         757 MiB       85       0 255                   0        0         0   5306018840 26 TiB  9170232158   24 TiB         0 B          0 B

total_objects    13169948 total_used       132 TiB total_avail      33 TiB total_space      166 TiB

[root@san1 ~]# ceph df detail --- RAW STORAGE --- CLASS     SIZE   AVAIL     USED  RAW USED  %RAW USED ssd    166 TiB  33 TiB  132 TiB   132 TiB      79.82 TOTAL  166 TiB  33 TiB  132 TiB   132 TiB      79.82

--- POOLS --- POOL                       ID  PGS   STORED   (DATA) (OMAP)  OBJECTS     USED   (DATA)   (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR UNDER COMPR cephfs.shared.meta          3   32  251 MiB  208 MiB   42 MiB       84  752 MiB  625 MiB  127 MiB      0    3.4 TiB            N/A          N/A    N/A         0 B          0 B cephfs.shared.data          4  512   14 TiB   14 TiB 0 B    3.83M   41 TiB   41 TiB      0 B  79.90    3.4 TiB            N/A          N/A    N/A     7.0 MiB 12 MiB

[root@san1 ~]# ceph osd pool get cephfs.shared.data size size: 3

...however running 'du' in the root directory of the 'shared' volume says:

# du -sh . 5.5T    .

So yeah - 14TB is replicated to 41TB, that's fine, but 14TB is a lot more than 5.5TB, so... where is that space going?

On 14/03/2024 2:09 am, Igor Fedotov wrote:

...

Hi Thorn,

could you please share the output of "ceph df detail" command representing the problem?

And please give an overview of your OSD layout - amount of OSDs, shared or dedicated DB/WAL, main and DB volume sizes.

Thanks,

Igor

On 3/13/2024 5:58 AM, Thorne Lawler wrote:

...

Hi everyone!

My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB usage for the data pool, but there are only 5.5TB of files in it. There are fewer than 100 files on the filesystem in total, so where is all that space going?

How can I analyze my cephfs to understand what is using that space, and if possible, how can I reclaim that space?

Thank you.

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx

Can anyone please tell me what "COPIES" means in this context? [ceph: root@san2 /]# rados df -p cephfs.shared.data POOL_NAME             USED  OBJECTS  CLONES    COPIES MISSING_ON_PRIMARY  UNFOUND  DEGRADED      RD_OPS       RD WR_OPS       WR  USED COMPR  UNDER COMPR cephfs.shared.data  41 TiB  3886733       0 11660199                   0        0         0  3249045631  177 TiB  9372801877  231 TiB     7.0 MiB       12 MiB In what sense does my CephFS filesystem include 11660199 'copies'? Copies of what? On 18/03/2024 11:12 am, Thorne Lawler wrote:

Thanks Igor,

I have tried that, and the number of objects and bytes_used took a long time to drop, but they seem to have dropped back to almost the original level:

 * Before creating the file:      o 3885835 objects      o 45349150134272 bytes_used  * After creating the file:      o 3931663 objects      o 45924147249152 bytes_used  * Immediately after deleting the file:      o 3935995 objects      o 45978612027392 bytes_used  * Half an hour after deleting the file:      o 3886013 objects      o 45349569585152 bytes_used

Unfortunately, this is all production infrastructure, so there is always other activity taking place.

What tools are there to visually inspect the object map and see how it relates to the filesystem?

On 15/03/2024 7:18 pm, Igor Fedotov wrote:

...

ceph df detail --format json-pretty --

Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

Hi Thorn, given the amount of files at CephFS volume I presume you don't have severe write load against it. Is that correct? If so we can assume that the numbers you're sharing are mostly refer to your experiment. At peak I can see bytes_used increase = 629,461,893,120 bytes (45978612027392  - 45349150134272). With replica factor = 3 this roughly matches your written data (200GB I presume?). More interestingly is that after file's removal we can see 419,450,880 bytes delta (=45349569585152 - 45349150134272). I could see two options (apart that someone else wrote additional stuff to CephFS during the experiment) to explain this: 1. File removal wasn't completed at the last probe half an hour after file's removal. Did you see stale object counter when making that probe? 2. Some space is leaking. If that's the case this could be a reason for your issue if huge(?) files at CephFS are created/removed periodically. So if we're certain that the leak really occurred (and option 1. above isn't the case) it makes sense to run more experiments with writing/removing a bunch of huge files to the volume to confirm space leakage. On 3/18/2024 3:12 AM, Thorne Lawler wrote:

Thanks Igor,

I have tried that, and the number of objects and bytes_used took a long time to drop, but they seem to have dropped back to almost the original level:

* Before creating the file: o 3885835 objects o 45349150134272 bytes_used * After creating the file: o 3931663 objects o 45924147249152 bytes_used * Immediately after deleting the file: o 3935995 objects o 45978612027392 bytes_used * Half an hour after deleting the file: o 3886013 objects o 45349569585152 bytes_used

Unfortunately, this is all production infrastructure, so there is always other activity taking place.

What tools are there to visually inspect the object map and see how it relates to the filesystem?

Not sure if there is anything like that at CephFS level but you can use rados tool to view objects in cephfs data pool and try to build some mapping between them and CephFS file list. Could be a bit tricky though.

On 15/03/2024 7:18 pm, Igor Fedotov wrote:

...

ceph df detail --format json-pretty --

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx

Alexander, Thank you, but as I said to Igor: The 5.5TB of files on this filesystem are virtual machine disks. They are under constant, heavy write load. There is no way to turn this off. On 19/03/2024 9:36 pm, Alexander E. Patrakov wrote:

Hello Thorne,

Here is one more suggestion on how to debug this. Right now, there is uncertainty on whether there is really a disk space leak or if something simply wrote new data during the test.

If you have at least three OSDs you can reassign, please set their CRUSH device class to something different than before. E.g., "test". Then, create a new pool that targets this device class and add it to CephFS. Then, create an empty directory on CephFS and assign this pool to it using setfattr. Finally, try reproducing the issue using only files in this directory. This way, you will be sure that nobody else is writing any data to the new pool.

On Tue, Mar 19, 2024 at 5:40 PM Igor Fedotov<igor.fedotov@croit.io> wrote:

...

Hi Thorn,

given the amount of files at CephFS volume I presume you don't have severe write load against it. Is that correct?

If so we can assume that the numbers you're sharing are mostly refer to your experiment. At peak I can see bytes_used increase = 629,461,893,120 bytes (45978612027392 - 45349150134272). With replica factor = 3 this roughly matches your written data (200GB I presume?).

More interestingly is that after file's removal we can see 419,450,880 bytes delta (=45349569585152 - 45349150134272). I could see two options (apart that someone else wrote additional stuff to CephFS during the experiment) to explain this:

1. File removal wasn't completed at the last probe half an hour after file's removal. Did you see stale object counter when making that probe?

2. Some space is leaking. If that's the case this could be a reason for your issue if huge(?) files at CephFS are created/removed periodically. So if we're certain that the leak really occurred (and option 1. above isn't the case) it makes sense to run more experiments with writing/removing a bunch of huge files to the volume to confirm space leakage.

On 3/18/2024 3:12 AM, Thorne Lawler wrote:

...

Thanks Igor,

I have tried that, and the number of objects and bytes_used took a long time to drop, but they seem to have dropped back to almost the original level:

* Before creating the file: o 3885835 objects o 45349150134272 bytes_used * After creating the file: o 3931663 objects o 45924147249152 bytes_used * Immediately after deleting the file: o 3935995 objects o 45978612027392 bytes_used * Half an hour after deleting the file: o 3886013 objects o 45349569585152 bytes_used

Unfortunately, this is all production infrastructure, so there is always other activity taking place.

What tools are there to visually inspect the object map and see how it relates to the filesystem?

Not sure if there is anything like that at CephFS level but you can use rados tool to view objects in cephfs data pool and try to build some mapping between them and CephFS file list. Could be a bit tricky though.

...

On 15/03/2024 7:18 pm, Igor Fedotov wrote:

...

ceph df detail --format json-pretty --

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

-- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

Alexander, I'm happy to create a new pool if it will help, but I don't presently see how creating a new pool will help us to identify the source of the 10TB discrepancy in this original cephfs pool. Please help me to understand what you are hoping to find...? On 20/03/2024 6:35 pm, Alexander E. Patrakov wrote:

Thorne,

That's why I asked you to create a separate pool. All writes go to the original pool, and it is possible to see object counts per-pool.

On Wed, Mar 20, 2024 at 6:32 AM Thorne Lawler <thorne@ddns.com.au> wrote:

Alexander,

Thank you, but as I said to Igor: The 5.5TB of files on this filesystem are virtual machine disks. They are under constant, heavy write load. There is no way to turn this off.

On 19/03/2024 9:36 pm, Alexander E. Patrakov wrote:

...

Hello Thorne,

Here is one more suggestion on how to debug this. Right now, there is uncertainty on whether there is really a disk space leak or if something simply wrote new data during the test.

If you have at least three OSDs you can reassign, please set their CRUSH device class to something different than before. E.g., "test". Then, create a new pool that targets this device class and add it to CephFS. Then, create an empty directory on CephFS and assign this pool to it using setfattr. Finally, try reproducing the issue using only files in this directory. This way, you will be sure that nobody else is writing any data to the new pool.

On Tue, Mar 19, 2024 at 5:40 PM Igor Fedotov<igor.fedotov@croit.io> <mailto:igor.fedotov@croit.io> wrote:

...

Hi Thorn,

given the amount of files at CephFS volume I presume you don't have severe write load against it. Is that correct?

If so we can assume that the numbers you're sharing are mostly refer to your experiment. At peak I can see bytes_used increase = 629,461,893,120 bytes (45978612027392 - 45349150134272). With replica factor = 3 this roughly matches your written data (200GB I presume?).

More interestingly is that after file's removal we can see 419,450,880 bytes delta (=45349569585152 - 45349150134272). I could see two options (apart that someone else wrote additional stuff to CephFS during the experiment) to explain this:

1. File removal wasn't completed at the last probe half an hour after file's removal. Did you see stale object counter when making that probe?

2. Some space is leaking. If that's the case this could be a reason for your issue if huge(?) files at CephFS are created/removed periodically. So if we're certain that the leak really occurred (and option 1. above isn't the case) it makes sense to run more experiments with writing/removing a bunch of huge files to the volume to confirm space leakage.

On 3/18/2024 3:12 AM, Thorne Lawler wrote:

...

Thanks Igor,

I have tried that, and the number of objects and bytes_used took a long time to drop, but they seem to have dropped back to almost the original level:

* Before creating the file: o 3885835 objects o 45349150134272 bytes_used * After creating the file: o 3931663 objects o 45924147249152 bytes_used * Immediately after deleting the file: o 3935995 objects o 45978612027392 bytes_used * Half an hour after deleting the file: o 3886013 objects o 45349569585152 bytes_used

Unfortunately, this is all production infrastructure, so there is always other activity taking place.

What tools are there to visually inspect the object map and see how it relates to the filesystem?

Not sure if there is anything like that at CephFS level but you can use rados tool to view objects in cephfs data pool and try to build some mapping between them and CephFS file list. Could be a bit tricky though.

...

On 15/03/2024 7:18 pm, Igor Fedotov wrote:

...

ceph df detail --format json-pretty

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io <http://croit.io>

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Alexander E. Patrakov --

Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

Alexander, Thanks for explaining this. As I suspected, this is a high abstract pursuit of what caused the problem, and while I'm sure this makes sense for Ceph developers, it isn't going to happen in this case. I don't care how it got this way- the tools used to create this pool will never be used in our environment again after I recover this disk space - the entire reason I need to recover the missing space is so I can move enough filesystems around to remove the current structure and the tools that made it. I only need to get that disk space back. Any analysis I do will be solely directed towards achieving that. Thanks. On 21/03/2024 3:10 am, Alexander E. Patrakov wrote:

Hi Thorne,

The idea is quite simple. By retesting the leak with a separate pool, used by nobody except you, in the case if the leak exists and is reproducible (which is not a given), you can definitely pinpoint it without giving any chance to the alternate hypothesis "somebody wrote some data in parallel". And then, even if the leak is small but reproducible, one can say that multiple such events accumulated to 10 TB of garbage in the original pool.

On Wed, Mar 20, 2024 at 7:29 PM Thorne Lawler <thorne@ddns.com.au> wrote:

Alexander,

I'm happy to create a new pool if it will help, but I don't presently see how creating a new pool will help us to identify the source of the 10TB discrepancy in this original cephfs pool.

Please help me to understand what you are hoping to find...?

On 20/03/2024 6:35 pm, Alexander E. Patrakov wrote:

...

Thorne,

That's why I asked you to create a separate pool. All writes go to the original pool, and it is possible to see object counts per-pool.

On Wed, Mar 20, 2024 at 6:32 AM Thorne Lawler <thorne@ddns.com.au> wrote:

Alexander,

Thank you, but as I said to Igor: The 5.5TB of files on this filesystem are virtual machine disks. They are under constant, heavy write load. There is no way to turn this off.

On 19/03/2024 9:36 pm, Alexander E. Patrakov wrote:

...

Hello Thorne,

Here is one more suggestion on how to debug this. Right now, there is uncertainty on whether there is really a disk space leak or if something simply wrote new data during the test.

If you have at least three OSDs you can reassign, please set their CRUSH device class to something different than before. E.g., "test". Then, create a new pool that targets this device class and add it to CephFS. Then, create an empty directory on CephFS and assign this pool to it using setfattr. Finally, try reproducing the issue using only files in this directory. This way, you will be sure that nobody else is writing any data to the new pool.

On Tue, Mar 19, 2024 at 5:40 PM Igor Fedotov<igor.fedotov@croit.io> <mailto:igor.fedotov@croit.io> wrote:

...

Hi Thorn,

given the amount of files at CephFS volume I presume you don't have severe write load against it. Is that correct?

If so we can assume that the numbers you're sharing are mostly refer to your experiment. At peak I can see bytes_used increase = 629,461,893,120 bytes (45978612027392 - 45349150134272). With replica factor = 3 this roughly matches your written data (200GB I presume?).

More interestingly is that after file's removal we can see 419,450,880 bytes delta (=45349569585152 - 45349150134272). I could see two options (apart that someone else wrote additional stuff to CephFS during the experiment) to explain this:

1. File removal wasn't completed at the last probe half an hour after file's removal. Did you see stale object counter when making that probe?

2. Some space is leaking. If that's the case this could be a reason for your issue if huge(?) files at CephFS are created/removed periodically. So if we're certain that the leak really occurred (and option 1. above isn't the case) it makes sense to run more experiments with writing/removing a bunch of huge files to the volume to confirm space leakage.

On 3/18/2024 3:12 AM, Thorne Lawler wrote:

...

Thanks Igor,

I have tried that, and the number of objects and bytes_used took a long time to drop, but they seem to have dropped back to almost the original level:

* Before creating the file: o 3885835 objects o 45349150134272 bytes_used * After creating the file: o 3931663 objects o 45924147249152 bytes_used * Immediately after deleting the file: o 3935995 objects o 45978612027392 bytes_used * Half an hour after deleting the file: o 3886013 objects o 45349569585152 bytes_used

Unfortunately, this is all production infrastructure, so there is always other activity taking place.

What tools are there to visually inspect the object map and see how it relates to the filesystem?

Not sure if there is anything like that at CephFS level but you can use rados tool to view objects in cephfs data pool and try to build some mapping between them and CephFS file list. Could be a bit tricky though.

...

On 15/03/2024 7:18 pm, Igor Fedotov wrote:

...

ceph df detail --format json-pretty

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io <http://croit.io>

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Alexander E. Patrakov

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Alexander E. Patrakov --

Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

Thorne, if that's a bug in Ceph which causes space leakage you might be unable to reclaim the space without total purge of the pool. The problem is that we still uncertain if this is a leakage or something else. Hence the need for more thorough research. Thanks, Igor On 3/20/2024 9:13 PM, Thorne Lawler wrote:

Alexander,

Thanks for explaining this. As I suspected, this is a high abstract pursuit of what caused the problem, and while I'm sure this makes sense for Ceph developers, it isn't going to happen in this case.

I don't care how it got this way- the tools used to create this pool will never be used in our environment again after I recover this disk space - the entire reason I need to recover the missing space is so I can move enough filesystems around to remove the current structure and the tools that made it.

I only need to get that disk space back. Any analysis I do will be solely directed towards achieving that.

Thanks.

On 21/03/2024 3:10 am, Alexander E. Patrakov wrote:

...

Hi Thorne,

The idea is quite simple. By retesting the leak with a separate pool, used by nobody except you, in the case if the leak exists and is reproducible (which is not a given), you can definitely pinpoint it without giving any chance to the alternate hypothesis "somebody wrote some data in parallel". And then, even if the leak is small but reproducible, one can say that multiple such events accumulated to 10 TB of garbage in the original pool.

On Wed, Mar 20, 2024 at 7:29 PM Thorne Lawler <thorne@ddns.com.au> wrote:

Alexander,

I'm happy to create a new pool if it will help, but I don't presently see how creating a new pool will help us to identify the source of the 10TB discrepancy in this original cephfs pool.

Please help me to understand what you are hoping to find...?

On 20/03/2024 6:35 pm, Alexander E. Patrakov wrote:

...

Thorne,

That's why I asked you to create a separate pool. All writes go to the original pool, and it is possible to see object counts per-pool.

On Wed, Mar 20, 2024 at 6:32 AM Thorne Lawler <thorne@ddns.com.au> wrote:

Alexander,

Thank you, but as I said to Igor: The 5.5TB of files on this filesystem are virtual machine disks. They are under constant, heavy write load. There is no way to turn this off.

On 19/03/2024 9:36 pm, Alexander E. Patrakov wrote:

...

Hello Thorne,

Here is one more suggestion on how to debug this. Right now, there is uncertainty on whether there is really a disk space leak or if something simply wrote new data during the test.

If you have at least three OSDs you can reassign, please set their CRUSH device class to something different than before. E.g., "test". Then, create a new pool that targets this device class and add it to CephFS. Then, create an empty directory on CephFS and assign this pool to it using setfattr. Finally, try reproducing the issue using only files in this directory. This way, you will be sure that nobody else is writing any data to the new pool.

On Tue, Mar 19, 2024 at 5:40 PM Igor Fedotov<igor.fedotov@croit.io> <mailto:igor.fedotov@croit.io> wrote:

...

Hi Thorn,

given the amount of files at CephFS volume I presume you don't have severe write load against it. Is that correct?

If so we can assume that the numbers you're sharing are mostly refer to your experiment. At peak I can see bytes_used increase = 629,461,893,120 bytes (45978612027392 - 45349150134272). With replica factor = 3 this roughly matches your written data (200GB I presume?).

More interestingly is that after file's removal we can see 419,450,880 bytes delta (=45349569585152 - 45349150134272). I could see two options (apart that someone else wrote additional stuff to CephFS during the experiment) to explain this:

1. File removal wasn't completed at the last probe half an hour after file's removal. Did you see stale object counter when making that probe?

2. Some space is leaking. If that's the case this could be a reason for your issue if huge(?) files at CephFS are created/removed periodically. So if we're certain that the leak really occurred (and option 1. above isn't the case) it makes sense to run more experiments with writing/removing a bunch of huge files to the volume to confirm space leakage.

On 3/18/2024 3:12 AM, Thorne Lawler wrote:

...

Thanks Igor,

I have tried that, and the number of objects and bytes_used took a long time to drop, but they seem to have dropped back to almost the original level:

* Before creating the file: o 3885835 objects o 45349150134272 bytes_used * After creating the file: o 3931663 objects o 45924147249152 bytes_used * Immediately after deleting the file: o 3935995 objects o 45978612027392 bytes_used * Half an hour after deleting the file: o 3886013 objects o 45349569585152 bytes_used

Unfortunately, this is all production infrastructure, so there is always other activity taking place.

What tools are there to visually inspect the object map and see how it relates to the filesystem?

Not sure if there is anything like that at CephFS level but you can use rados tool to view objects in cephfs data pool and try to build some mapping between them and CephFS file list. Could be a bit tricky though.

...

On 15/03/2024 7:18 pm, Igor Fedotov wrote: > ceph df detail --format json-pretty --

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io <http://croit.io>

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Alexander E. Patrakov

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Alexander E. Patrakov --

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx

Those files are VM disk images, and they're under constant heavy use, so yes- there/is/ constant severe write load against this disk.

Why are you using CephFS for an RBD application?

Grep through the ls output for ‘rados bench’ leftovers, it’s easy to leave them behind.

On Mar 20, 2024, at 5:28 PM, Igor Fedotov <igor.fedotov@croit.io> wrote:

Hi Thorne,

unfortunately I'm unaware of any tools high level enough to easily map files to rados objects without deep undestanding how this works. You might want to try "rados ls" command to get the list of all the objects in the cephfs data pool. And then learn how that mapping is performed and parse your listing.

Thanks,

Igor

...

On 3/20/2024 1:30 AM, Thorne Lawler wrote:

Igor,

Those files are VM disk images, and they're under constant heavy use, so yes- there/is/ constant severe write load against this disk.

Apart from writing more test files into the filesystems, there must be Ceph diagnostic tools to describe what those objects are being used for, surely?

We're talking about an extra 10TB of space. How hard can it be to determine which file those objects are associated with?

...

On 19/03/2024 8:39 pm, Igor Fedotov wrote:

Hi Thorn,

given the amount of files at CephFS volume I presume you don't have severe write load against it. Is that correct?

If so we can assume that the numbers you're sharing are mostly refer to your experiment. At peak I can see bytes_used increase = 629,461,893,120 bytes (45978612027392 - 45349150134272). With replica factor = 3 this roughly matches your written data (200GB I presume?).

More interestingly is that after file's removal we can see 419,450,880 bytes delta (=45349569585152 - 45349150134272). I could see two options (apart that someone else wrote additional stuff to CephFS during the experiment) to explain this:

1. File removal wasn't completed at the last probe half an hour after file's removal. Did you see stale object counter when making that probe?

2. Some space is leaking. If that's the case this could be a reason for your issue if huge(?) files at CephFS are created/removed periodically. So if we're certain that the leak really occurred (and option 1. above isn't the case) it makes sense to run more experiments with writing/removing a bunch of huge files to the volume to confirm space leakage.

On 3/18/2024 3:12 AM, Thorne Lawler wrote:

...

Thanks Igor,

I have tried that, and the number of objects and bytes_used took a long time to drop, but they seem to have dropped back to almost the original level:

* Before creating the file: o 3885835 objects o 45349150134272 bytes_used * After creating the file: o 3931663 objects o 45924147249152 bytes_used * Immediately after deleting the file: o 3935995 objects o 45978612027392 bytes_used * Half an hour after deleting the file: o 3886013 objects o 45349569585152 bytes_used

Unfortunately, this is all production infrastructure, so there is always other activity taking place.

What tools are there to visually inspect the object map and see how it relates to the filesystem?

Not sure if there is anything like that at CephFS level but you can use rados tool to view objects in cephfs data pool and try to build some mapping between them and CephFS file list. Could be a bit tricky though.

...

On 15/03/2024 7:18 pm, Igor Fedotov wrote:

...

ceph df detail --format json-pretty --

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx --

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

-- Igor Fedotov Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io