Hi! We have S3/CephFS cluster with dedicated SSDs for bucket indexes, CephFS metadata and for a few small rgw metadata pools. We have special crush rules for these pools. All of our pools (pool naming is old because it's very old cluster) that are placed on SSDs: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 32 1.3 KiB 5 108 KiB 0 690 GiB .rgw.control 2 32 0 B 8 0 B 0 690 GiB .rgw 3 32 2.0 MiB 8.02k 328 MiB 0.02 690 GiB .rgw.gc 4 64 1.6 GiB 64 4.9 GiB 0.24 690 GiB .users.uid 5 32 1.8 MiB 2.63k 67 MiB 0 690 GiB .users 6 32 275 KiB 7.02k 379 MiB 0.02 690 GiB .usage 7 32 30 MiB 64 30 MiB 0 690 GiB .intent-log 8 64 0 B 0 0 B 0 690 GiB .log 9 64 6.6 GiB 34.29k 21 GiB 1.01 690 GiB .rgw.buckets.index 12 4096 1.9 TiB 248.96k 1.9 TiB 48.02 690 GiB .users.email 13 32 64 KiB 1.61k 72 MiB 0 690 GiB fs1_meta 14 64 467 MiB 35.39k 829 MiB 0.04 690 GiB But on all our SSDs we can see high DATA usage, for example: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 57 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 96 GiB 1.4 GiB 40 GiB 90.96 1.37 138 up osd.57 19 ssd 0.00005 1.00000 447 GiB 402 GiB 309 GiB 92 GiB 1.4 GiB 44 GiB 90.09 1.35 145 up osd.19 10 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 97 GiB 1.1 GiB 40 GiB 91.01 1.37 137 up osd.10 12 ssd 0.00005 1.00000 447 GiB 401 GiB 309 GiB 91 GiB 1.3 GiB 46 GiB 89.77 1.35 134 up osd.12 All SSDs are bluestore. Ceph 16.3.11 and many SSDs were deployed on 16.x. As I understand, the majority of these pools contain only "OMAP" which is stored in RocksDB and tracked in stats as *OMAP*. And I don't understand why we have such high *DATA* and we can see that DATA usage grows by 1-2 GB/day for the last 90 days (we don't have more monitoring data)! I have checked crush rules and all SSDs for PGs from other pools and can't see any other PGs from "pure" data pools. Rules are correct. How can I see what is stored in bluestore DATA on some OSD? I can export full RocksDB, but not DATA stored in bluestore.. What kind of DATA can be stored in these OSDs with these pools? We did a lot of offline compactions, 2-5 times for some SSDs in the last 90 days. But i'm not sure if it's related