Hi list, we created a cluster for using cephfs with a kubernetes cluster. Since a few weeks now the cluster keeps filling up at an alarming rate (100 GB per day). This is while the most relevant pg is deep scrubbing and was interupted a few times. We use about 150G (du using the mounted filesystem) on the cephfs filesystem and try not to use snapshots (.snap directories "exist" but are empty). We do not understand why the pgs get bigger and bigger while cephfs stays about the same size (overwrites on files certainly happen). I suspect some snapshots mechanism. Any ideas how to debug this to stop it? Maybe we should try to speed up the deep scrubbing somehow? ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)   cluster:     id:     ece0290c-cd32-11ec-a0e2-005056a9dd02     health: HEALTH_WARN             1 MDSs report slow metadata IOs             3 nearfull osd(s)             13 pool(s) nearfull   services:     mon: 3 daemons, quorum acdh-gluster-hdd3,acdh-gluster-hdd1,acdh-gluster-hdd2 (age 3d)     mgr: acdh-gluster-hdd3.kzsplh(active, since 5d), standbys: acdh-gluster-hdd2.kiotbg, acdh-gluster-hdd1.ywgyfx     mds: 1/1 daemons up, 2 standby     osd: 3 osds: 3 up (since 4d), 3 in (since 7w)     rgw: 3 daemons active (3 hosts, 1 zones)   data:     volumes: 1/1 healthy     pools:   13 pools, 292 pgs     objects: 167.25M objects, 1.2 TiB     usage:   3.3 TiB used, 1.2 TiB / 4.5 TiB avail     pgs:     290 active+clean              1   active+clean+scrubbing+deep              1   active+clean+scrubbing   io:     client:   58 MiB/s rd, 3.6 MiB/s wr, 51 op/s rd, 148 op/s wr rancher-ceph-fs - 227 clients =============== RANK  STATE                  MDS                    ACTIVITY DNS    INOS   DIRS   CAPS  0    active  ceph-mds.acdh-gluster-hdd1.pqydya  Reqs:   68 /s 793k   792k   102k   210k           POOL              TYPE     USED  AVAIL  rancherFsPoolMetadata    metadata   160G   329G rancherFsPoolDefaultData    data    2268k   329G  rancherFsPoolMainData      data    2584G   658G            STANDBY MDS ceph-mds.acdh-gluster-hdd2.zfleqe ceph-mds.acdh-gluster-hdd3.etaobl MDS version: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) (rancherFsPoolMainData is a 2+1 erasure encoded pool) --- RAW STORAGE --- CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED hdd    4.5 TiB  1.2 TiB  3.3 TiB   3.3 TiB      73.46 TOTAL  4.5 TiB  1.2 TiB  3.3 TiB   3.3 TiB      73.46 --- POOLS --- POOL                        ID  PGS   STORED  OBJECTS     USED %USED  MAX AVAIL device_health_metrics        1    1      0 B        0      0 B      0    331 GiB rancher-rbd-erasure          2   32  8.4 GiB    2.16k   13 GiB 1.25    661 GiB rancher-rbd-meta             3   32     55 B       11   36 KiB      0    331 GiB rancherFsPoolMetadata        4   32   53 GiB    5.18M  160 GiB 13.88    331 GiB rancherFsPoolDefaultData     5    1   29 KiB   80.00M  2.2 MiB      0    331 GiB rancherFsPoolMainData        6    1  1.7 TiB   82.08M  2.5 TiB 72.23    661 GiB .rgw.root                    7   32  1.3 KiB        4   48 KiB      0    331 GiB default.rgw.log              8   32  3.6 KiB      209  408 KiB      0    331 GiB default.rgw.control          9   32      0 B        8      0 B      0    331 GiB default.rgw.meta            10   32  3.8 KiB       11  124 KiB      0    331 GiB default.rgw.buckets.index   11   32  2.4 MiB       33  7.2 MiB      0    331 GiB default.rgw.buckets.non-ec  12   32      0 B        0      0 B      0    331 GiB default.rgw.buckets.data    14    1   55 GiB   16.57k   83 GiB 7.70    661 GiB HEALTH_WARN 1 MDSs report slow metadata IOs; 3 nearfull osd(s); 13 pool(s) nearfull [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs     mds.ceph-mds.acdh-gluster-hdd1.pqydya(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 306 secs [WRN] OSD_NEARFULL: 3 nearfull osd(s)     osd.0 is near full     osd.2 is near full     osd.3 is near full [WRN] POOL_NEARFULL: 13 pool(s) nearfull     pool 'device_health_metrics' is nearfull     pool 'rancher-rbd-erasure' is nearfull     pool 'rancher-rbd-meta' is nearfull     pool 'rancherFsPoolMetadata' is nearfull     pool 'rancherFsPoolDefaultData' is nearfull     pool 'rancherFsPoolMainData' is nearfull     pool '.rgw.root' is nearfull     pool 'default.rgw.log' is nearfull     pool 'default.rgw.control' is nearfull     pool 'default.rgw.meta' is nearfull     pool 'default.rgw.buckets.index' is nearfull     pool 'default.rgw.buckets.non-ec' is nearfull     pool 'default.rgw.buckets.data' is nearfull (near full is set to 0.66) -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam@oeaw.ac.at | www.oeaw.ac.at/acdh