Good day We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions. However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left. We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why? Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again. We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs. Below are some of our servers and the DB used / SLOW used. Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi@idia.ac.za <mfundo@idia.ac.za> Rondebosch, Cape Town, 7600, South Africa