10 Apr
2020
10 Apr
'20
11:41 p.m.
Hopefully someone can sanity check me here, but I'm getting the feeling that the MAX
AVAIL in ceph df isn't reporting the correct value in 14.2.8 (mon/mgr/mds are .8, most
OSDs are .7)
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 530 TiB 163 TiB 366 TiB 367 TiB 69.19
ssd 107 TiB 37 TiB 70 TiB 70 TiB 65.33
TOTAL 637 TiB 201 TiB 436 TiB 437 TiB 68.54
POOLS:
POOL ID STORED OBJECTS USED %USED MAX
AVAIL
fs-metadata 16 44 GiB 4.16M 44 GiB 0.25 5.6
TiB
cephfs-hdd-3x 17 46 TiB 109.54M 144 TiB 61.81 30
TiB
objects-hybrid 20 46 TiB 537.08M 187 TiB 91.71 5.6
TiB
objects-hdd 24 224 GiB 50.81k 676 GiB 0.74 30
TiB
rbd-hybrid 29 3.8 TiB 1.19M 11 TiB 40.38 5.6
TiB
device_health_metrics 33 270 MiB 327 270 MiB 0 30
TiB
rbd-ssd 34 4.2 TiB 1.19M 12 TiB 41.55 5.6
TiB
cephfs-hdd-ec73 37 42 TiB 30.35M 72 TiB 44.78 62
TiB
I have a few pools that don't look like they are calculating the available storage for
that pool correctly.
Specifically, any of my hybrid pools (20,29) or all-SSD pools (16,34).
For my hybrid pools, I have a crush rule of take 1 of host in the ssd root, take -1
chassis in the hdd root.
For my ssd pools, I have a crush rule of take 0 of host in the ssd root.
Now I have 60 ssd osds 1.92T each, and sadly distribution is imperfect (leaving those
issues out of this), and I have plenty of underfull and overfull osds, which I am trying
to manually reweighs to get my most full's down to free up space:
$ ceph osd df class ssd | sort -k17
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR
PGS STATUS
MIN/MAX VAR: 0.80/1.21 STDDEV: 5.56
TOTAL 107 TiB 70 TiB 68 TiB 163 GiB 431 GiB 37 TiB 65.33
28 ssd 1.77879 1.00000 1.8 TiB 951 GiB 916 GiB 7.1 GiB 5.8 GiB 871 GiB 52.20 0.80
68 up
33 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1010 GiB 6.0 MiB 5.9 GiB 777 GiB 57.33 0.88
74 up
47 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1011 GiB 6.7 GiB 6.4 GiB 776 GiB 57.38 0.88
75 up
[SNIP]
57 ssd 1.77879 0.98000 1.8 TiB 1.4 TiB 1.3 TiB 6.2 GiB 8.6 GiB 417 GiB 77.08 1.18
102 up
107 ssd 1.80429 1.00000 1.8 TiB 1.4 TiB 1.3 TiB 7.0 GiB 8.7 GiB 422 GiB 77.15 1.18
102 up
50 ssd 1.77879 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 5.5 MiB 8.6 GiB 381 GiB 79.10 1.21
105 up
60 ssd 1.77879 0.92000 1.8 TiB 1.4 TiB 1.4 TiB 6.2 MiB 9.0 GiB 379 GiB 79.17 1.21
105 up
That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be ~22.8TiB free,
if all OSD's grew evenly, which they won't, which is still far short of 37TiB raw
free, as expected.
However, what doesn't track is the 5.6TiB available at the pools level, even for a 3x
replicated pool (5.6*3=16.8TiB, which is about 34% less than my napkin math, which would
be 22.8/3=7.6TiB.
But what tracks even less is the hybrid pools, which use 1/3 of what the 3x-replicated
data consumes.
Meaning if my napkin math is right, should show ~22.8TiB free.
Am I grossly mis-understanding how this is calculated?
Maybe this is fixed in Octopus?
Just trying to get a grasp on what I'm seeing not matching expectations.
Thanks,
Reed