Hello, We have a cluster used mostly for CephFS, to which we recently added new disk servers (cephadm managed, running v19.2.6). It's a 45% capacity increase, from 52 to 76 OSD nodes. All new nodes have HDD+external WAL/DB on NVMe drives and some dedicated NVMe drives for MD (w/ colocated WAL/DB). The OSDs were added using the method described here: https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ We also increased the numbe of PGs in the main data (EC) pool (at the same time), before releasing the balancer following https://docs.clyso.com/docs/kb/rados/pg-splitting-with-upmap-remapped/ It went OK for a few days, then we received an alert for the metadata pool being full. Indeed it became full since "something" used all the space on the SSDs that hold the metadata and the "default" pools. These pools are 3-way replicated. The SSDs used for these pools have a colocated WAL/DB. Graphics from our monitoring showing the reduction in MD space available and actual MD usage: https://hidoc.in2p3.fr/s/VsVaULFB9v The reduction of available space (downward slope on the first graph) in the metadata pool started with the rebalance of the data, until zero bytes were left free in the pool. For instance one of the (old) MD OSD: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up What consummes 200+ GiB on each MD OSD when the total MD pool usage is less than 200 GiB ? Is it the colocated WAL/DB ? There is no "ceph.dir.layout.pool" attribute targeting the "cephfs_ssd" or "cephfs_default" pool (no client has the "p" permission). More details: At some point the smaller MD OSD were really full (> 99% occupation), they all crashed and refuse to start (they are down). Of course the FS is down too. ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 17 ssd 0.21829 1.00000 224 GiB 223 GiB 216 GiB 1.1 GiB 6.4 GiB 424 MiB 99.81 1.97 0 down We managed to add extra OSDs for MD to try to restart, but we are progressing very slowly. "ceph df" (even in detailled mode) does not show what is using that SSD space (this is the state after several days of recovery effort, the FS is not online yet, recovery is still ongoing): [ceph: root@$ADMNODE2 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 18 PiB 8.7 PiB 9.4 PiB 9.4 PiB 51.93 ssd 682 TiB 555 TiB 127 TiB 127 TiB 18.64 TOTAL 19 PiB 9.3 PiB 9.5 PiB 9.5 PiB 50.75 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 2 8 1.0 GiB 265 3.1 GiB 0 37 TiB cephfs_EC_data 3 16384 7.1 PiB 3.15G 8.9 PiB 63.49 4.1 PiB cephfs_metadata 4 1024 193 GiB 91.25M 578 GiB 0.51 37 TiB cephfs_default 5 128 258 MiB 1.34G 773 MiB 0 37 TiB cephfs_ssd 6 128 0 B 0 0 B 0 37 TiB .rgw.root 7 32 1.9 KiB 4 76 KiB 0 1.7 PiB default.rgw.log 8 32 6.8 KiB 210 760 KiB 0 1.7 PiB default.rgw.control 9 32 0 B 8 0 B 0 1.7 PiB default.rgw.meta 10 32 39 KiB 53 1013 KiB 0 1.7 PiB default.rgw.buckets.index 11 32 3.2 GiB 209 9.6 GiB 0 1.7 PiB default.rgw.buckets.data 12 32 5.9 TiB 12.42M 18 TiB 0.34 1.7 PiB default.rgw.buckets.non-ec 13 32 16 MiB 3.20k 48 MiB 0 1.7 PiB cephfs_nvme 14 4096 2.7 TiB 1.84M 3.3 TiB 1.23 214 TiB [ceph: root@$ADMNODE2 /]# => The issue is: what still consumes 120+ TiB on SSDs ? The sum of "USED" for all pools besides "cephfs_EC_data" is way lower than 127 TiB ("EC_data" and "nvme" are EC pools). Before the extension we had about 24 TiB of SSDs available for the "cephfs_metadata", "cephfs_default" and "cephfs_ssd" pools (the latter is no longer used and was replaced with the "cephfs_nvme" pool). We do not use snapshots at all on this cluster. There are no purge queue to empty on the various MDS. The "full-ratio" settings are at the default values. There are no CephFS (non admin) clients with the ability to change the destination pool or create snapshots, and I think "ceph df" would show if the SSD pools were abused by a "clever user". (the "cephfs_default" usage is due to our recovery effort, it was 0 bytes before) Loïc. -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |