25 Feb
2021
25 Feb
'21
7:29 p.m.
On Thu, Feb 25, 2021 at 10:55:05AM +0000, Simon Sutter wrote:
The output of ceph df detail is:
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41
TOTAL 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41
--- POOLS ---
POOL ID PGS STORED OBJECTS %USED MAX AVAIL
device_health_metrics 1 1 51 MiB 48 0 30 TiB
rep_data_fs 2 32 14 KiB 3.41k 0 30 TiB
rep_meta_fs 3 32 227 MiB 1.72k 0 30 TiB
ec_bkp1 4 32 4.2 TiB 1.10M 6.11 67 TiB
So ec_bkp1 uses 4.2TiB an there are 67TiB free usable Storage.
This means total EC usable storage would be 71.2TiB.
But calculating with the 109TiB RAW storage, shouldn't it be 81.75?
The "MAX AVAIL" is not "free", i.e. it is not a difference between
total and used. It is estimated by the cluster as "how much data user
will be able to store additionally until one of osds is filled up".
Consider an example, when by some reason all osds are 10% full and one
50% full. The cluster will "assume" that if you store additional data
it will be distributed the same way, i.e. that one osd will continue
to store more than others, so when all other osds will 20% full that
one will be 100% full and the cluster is not usable.
I.e. basically "MAX AVAIL" in this example is (N - 1) * 10% + 1 * 50%,
instead of (N - 1) * 90% + 1 * 50%, which whould you expect for "free".
To make "MAX AVAIL" match exactly "free", you have to have a
perfectly
balanced cluster. Look at `ceph osd df` output to see how well data is
balanced in your case.
--
Mykola Golub