Hi!
Thank you! The output of both commands are below. I still dont understand why there are 21T used data (because 5.5T*3 = 16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the osd output tells we have 25T free space.
As I know MAX AVAIL is calculated with respect to mon_osd_full_ratio, max OSD %USE, WEIGHT, total weight of set of OSDs in pool and replication factor of pool. Max used hdd OSD is 34. So (0.95 - 0.65) * 931 / 3 * 1.3 ~= 4.5T
(0.95 - 0.65) * 931 / 3 * 1.3 * 37 ~= 4.5T
1.3 here state for avg osd size / 931 (you have 13*1.8T osd and other 24*931G) The logic behind is to estimate amount of data you can write into pool before max used OSD became full.
Thanks for helping understanding these numbers! probably we then recreate the bluestore osds one by one with reduced min alloc size.