[ceph-users] OSD utilization vs PG shard sum

Hi! I'm creating a custom balancer (mail will follow soonish), and for it to work, I'm calculating OSD usages manually, with the goal to simulate PG movements. But the calculations don't match up, I'm missing some size component. So far I have: osd_used_bytes = sum(pg_shardsizes) But that does not add up: used = ceph osd df -> osd['kb_used'] occupied = pg shard sum 404 used=1.080T occupied=1.137T => 58.303G 405 used=1.031T occupied=1.089T => 59.255G 406 used=4.459T occupied=4.563T => 105.685G 407 used=4.414T occupied=4.433T => 19.751G 408 used=4.428T occupied=4.449T => 21.906G 409 used=4.440T occupied=4.417T => -23.441G 410 used=4.416T occupied=4.397T => -19.868G 411 used=4.446T occupied=4.488T => 42.905G 412 used=4.414T occupied=4.386T => -28.452G 413 used=4.439T occupied=4.461T => 23.326G Especially weird are the negative deltas, they mean the sum of pg shardsizes is more than the osd-reported size. Could this be compression? If yes, how can I get per-PG compression stats? Missing from the calculation is the bluefs_db_size (is that available via json? I could only find the daemon-socket perf counters and prometheus). It's around 2G for each OSD, and should not contribute much to the delta. In short: How do I reliably calculate the real OSD utilization when summing up sizes of currently mapped PGs? -- Jonas