Hi list, we created a cluster for using cephfs with a kubernetes cluster. Since a few weeks now the cluster keeps filling up at an alarming rate (100 GB per day). This is while the most relevant pg is deep scrubbing and was interupted a few times. We use about 150G (du using the mounted filesystem) on the cephfs filesystem and try not to use snapshots (.snap directories "exist" but are empty). We do not understand why the pgs get bigger and bigger while cephfs stays about the same size (overwrites on files certainly happen). I suspect some snapshots mechanism. Any ideas how to debug this to stop it? Maybe we should try to speed up the deep scrubbing somehow? ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) cluster: id: ece0290c-cd32-11ec-a0e2-005056a9dd02 health: HEALTH_WARN 1 MDSs report slow metadata IOs 3 nearfull osd(s) 13 pool(s) nearfull services: mon: 3 daemons, quorum acdh-gluster-hdd3,acdh-gluster-hdd1,acdh-gluster-hdd2 (age 3d) mgr: acdh-gluster-hdd3.kzsplh(active, since 5d), standbys: acdh-gluster-hdd2.kiotbg, acdh-gluster-hdd1.ywgyfx mds: 1/1 daemons up, 2 standby osd: 3 osds: 3 up (since 4d), 3 in (since 7w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 292 pgs objects: 167.25M objects, 1.2 TiB usage: 3.3 TiB used, 1.2 TiB / 4.5 TiB avail pgs: 290 active+clean 1 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 58 MiB/s rd, 3.6 MiB/s wr, 51 op/s rd, 148 op/s wr rancher-ceph-fs - 227 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mds.acdh-gluster-hdd1.pqydya Reqs: 68 /s 793k 792k 102k 210k POOL TYPE USED AVAIL rancherFsPoolMetadata metadata 160G 329G rancherFsPoolDefaultData data 2268k 329G rancherFsPoolMainData data 2584G 658G STANDBY MDS ceph-mds.acdh-gluster-hdd2.zfleqe ceph-mds.acdh-gluster-hdd3.etaobl MDS version: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) (rancherFsPoolMainData is a 2+1 erasure encoded pool) --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 4.5 TiB 1.2 TiB 3.3 TiB 3.3 TiB 73.46 TOTAL 4.5 TiB 1.2 TiB 3.3 TiB 3.3 TiB 73.46 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 331 GiB rancher-rbd-erasure 2 32 8.4 GiB 2.16k 13 GiB 1.25 661 GiB rancher-rbd-meta 3 32 55 B 11 36 KiB 0 331 GiB rancherFsPoolMetadata 4 32 53 GiB 5.18M 160 GiB 13.88 331 GiB rancherFsPoolDefaultData 5 1 29 KiB 80.00M 2.2 MiB 0 331 GiB rancherFsPoolMainData 6 1 1.7 TiB 82.08M 2.5 TiB 72.23 661 GiB .rgw.root 7 32 1.3 KiB 4 48 KiB 0 331 GiB default.rgw.log 8 32 3.6 KiB 209 408 KiB 0 331 GiB default.rgw.control 9 32 0 B 8 0 B 0 331 GiB default.rgw.meta 10 32 3.8 KiB 11 124 KiB 0 331 GiB default.rgw.buckets.index 11 32 2.4 MiB 33 7.2 MiB 0 331 GiB default.rgw.buckets.non-ec 12 32 0 B 0 0 B 0 331 GiB default.rgw.buckets.data 14 1 55 GiB 16.57k 83 GiB 7.70 661 GiB HEALTH_WARN 1 MDSs report slow metadata IOs; 3 nearfull osd(s); 13 pool(s) nearfull [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.ceph-mds.acdh-gluster-hdd1.pqydya(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 306 secs [WRN] OSD_NEARFULL: 3 nearfull osd(s) osd.0 is near full osd.2 is near full osd.3 is near full [WRN] POOL_NEARFULL: 13 pool(s) nearfull pool 'device_health_metrics' is nearfull pool 'rancher-rbd-erasure' is nearfull pool 'rancher-rbd-meta' is nearfull pool 'rancherFsPoolMetadata' is nearfull pool 'rancherFsPoolDefaultData' is nearfull pool 'rancherFsPoolMainData' is nearfull pool '.rgw.root' is nearfull pool 'default.rgw.log' is nearfull pool 'default.rgw.control' is nearfull pool 'default.rgw.meta' is nearfull pool 'default.rgw.buckets.index' is nearfull pool 'default.rgw.buckets.non-ec' is nearfull pool 'default.rgw.buckets.data' is nearfull (near full is set to 0.66) -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam@oeaw.ac.at | www.oeaw.ac.at/acdh