Hi!
I'm creating a custom balancer (mail will follow soonish), and for it to work, I'm
calculating OSD usages manually, with
the goal to simulate PG movements.
But the calculations don't match up, I'm missing some size component.
So far I have:
osd_used_bytes = sum(pg_shardsizes)
But that does not add up:
used = ceph osd df -> osd['kb_used']
occupied = pg shard sum
404 used=1.080T occupied=1.137T => 58.303G
405 used=1.031T occupied=1.089T => 59.255G
406 used=4.459T occupied=4.563T => 105.685G
407 used=4.414T occupied=4.433T => 19.751G
408 used=4.428T occupied=4.449T => 21.906G
409 used=4.440T occupied=4.417T => -23.441G
410 used=4.416T occupied=4.397T => -19.868G
411 used=4.446T occupied=4.488T => 42.905G
412 used=4.414T occupied=4.386T => -28.452G
413 used=4.439T occupied=4.461T => 23.326G
Especially weird are the negative deltas, they mean the sum of pg shardsizes is more than
the osd-reported size.
Could this be compression? If yes, how can I get per-PG compression stats?
Missing from the calculation is the bluefs_db_size (is that available via json? I could
only find the daemon-socket perf
counters and prometheus). It's around 2G for each OSD, and should not contribute much
to the delta.
In short: How do I reliably calculate the real OSD utilization when summing up sizes of
currently mapped PGs?
-- Jonas
Show replies by date