Hopefully someone can sanity check me here, but I'm getting the feeling that the MAX AVAIL in ceph df isn't reporting the correct value in 14.2.8 (mon/mgr/mds are .8, most OSDs are .7)
RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 530 TiB 163 TiB 366 TiB 367 TiB 69.19 ssd 107 TiB 37 TiB 70 TiB 70 TiB 65.33 TOTAL 637 TiB 201 TiB 436 TiB 437 TiB 68.54
POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL fs-metadata 16 44 GiB 4.16M 44 GiB 0.25 5.6 TiB cephfs-hdd-3x 17 46 TiB 109.54M 144 TiB 61.81 30 TiB objects-hybrid 20 46 TiB 537.08M 187 TiB 91.71 5.6 TiB objects-hdd 24 224 GiB 50.81k 676 GiB 0.74 30 TiB rbd-hybrid 29 3.8 TiB 1.19M 11 TiB 40.38 5.6 TiB device_health_metrics 33 270 MiB 327 270 MiB 0 30 TiB rbd-ssd 34 4.2 TiB 1.19M 12 TiB 41.55 5.6 TiB cephfs-hdd-ec73 37 42 TiB 30.35M 72 TiB 44.78 62 TiB
I have a few pools that don't look like they are calculating the available storage for that pool correctly. Specifically, any of my hybrid pools (20,29) or all-SSD pools (16,34). For my hybrid pools, I have a crush rule of take 1 of host in the ssd root, take -1 chassis in the hdd root. For my ssd pools, I have a crush rule of take 0 of host in the ssd root. Now I have 60 ssd osds 1.92T each, and sadly distribution is imperfect (leaving those issues out of this), and I have plenty of underfull and overfull osds, which I am trying to manually reweighs to get my most full's down to free up space:
$ ceph osd df class ssd | sort -k17 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS MIN/MAX VAR: 0.80/1.21 STDDEV: 5.56 TOTAL 107 TiB 70 TiB 68 TiB 163 GiB 431 GiB 37 TiB 65.33 28 ssd 1.77879 1.00000 1.8 TiB 951 GiB 916 GiB 7.1 GiB 5.8 GiB 871 GiB 52.20 0.80 68 up 33 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1010 GiB 6.0 MiB 5.9 GiB 777 GiB 57.33 0.88 74 up 47 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1011 GiB 6.7 GiB 6.4 GiB 776 GiB 57.38 0.88 75 up [SNIP] 57 ssd 1.77879 0.98000 1.8 TiB 1.4 TiB 1.3 TiB 6.2 GiB 8.6 GiB 417 GiB 77.08 1.18 102 up 107 ssd 1.80429 1.00000 1.8 TiB 1.4 TiB 1.3 TiB 7.0 GiB 8.7 GiB 422 GiB 77.15 1.18 102 up 50 ssd 1.77879 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 5.5 MiB 8.6 GiB 381 GiB 79.10 1.21 105 up 60 ssd 1.77879 0.92000 1.8 TiB 1.4 TiB 1.4 TiB 6.2 MiB 9.0 GiB 379 GiB 79.17 1.21 105 up
That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be ~22.8TiB free, if all OSD's grew evenly, which they won't, which is still far short of 37TiB raw free, as expected. However, what doesn't track is the 5.6TiB available at the pools level, even for a 3x replicated pool (5.6*3=16.8TiB, which is about 34% less than my napkin math, which would be 22.8/3=7.6TiB. But what tracks even less is the hybrid pools, which use 1/3 of what the 3x-replicated data consumes. Meaning if my napkin math is right, should show ~22.8TiB free. Am I grossly mis-understanding how this is calculated? Maybe this is fixed in Octopus? Just trying to get a grasp on what I'm seeing not matching expectations. Thanks, Reed
On Sat, Apr 11, 2020 at 12:43 AM Reed Dier <reed.dier@focusvq.com> wrote:
That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be ~22.8TiB free, if all OSD's grew evenly, which they won't
Yes, that's the problem. They won't grow evenly. The fullest one will grow faster than the others. Also, your full-ratio is probably 95% not 100%. So it'll be full as soon as OSD 70 takes another ~360 GB of data. But the others won't take 360 GB of data but less because of the bad balancing. For example, OSD 28 will only get around 233 GB of data by the time OSD 70 has 360 GB. Paul
, which is still far short of 37TiB raw free, as expected. However, what doesn't track is the 5.6TiB available at the pools level, even for a 3x replicated pool (5.6*3=16.8TiB, which is about 34% less than my napkin math, which would be 22.8/3=7.6TiB. But what tracks even less is the hybrid pools, which use 1/3 of what the 3x-replicated data consumes. Meaning if my napkin math is right, should show ~22.8TiB free.
Am I grossly mis-understanding how this is calculated? Maybe this is fixed in Octopus?
Just trying to get a grasp on what I'm seeing not matching expectations.
Thanks,
Reed _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That definitely makes sense. However should a hybrid pool not have 3x available of that of all SSD pool? There is plenty of rust behind it that won't impede it being able to satisfy all 3 replicants. Example lets say I write 5.6TiB (current max avail): to a hybrid pool, that's 5.6TiB to ssd osds, and 11.2TiB to hdd osds. to an all ssd pool, that's 16.8TiB written to ssd osd. And those are vastly different amounts to the ssd osds, obviously, and feel like the max avail is misleading, at least for these hybrid pools, which are admittedly less common for ceph I imagine. But its worth noting I'm just using crush roots to point my crush rules, and not actually using the device class, although it is set properly. And I imagine that if someone had an oddly specific crush rulesets to direct pg distribution, they too could see weird (possibly misleading) results like this. Reed
On Apr 10, 2020, at 5:55 PM, Paul Emmerich <paul.emmerich@croit.io> wrote:
On Sat, Apr 11, 2020 at 12:43 AM Reed Dier <reed.dier@focusvq.com> wrote:
That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be ~22.8TiB free, if all OSD's grew evenly, which they won't
Yes, that's the problem. They won't grow evenly. The fullest one will grow faster than the others. Also, your full-ratio is probably 95% not 100%. So it'll be full as soon as OSD 70 takes another ~360 GB of data. But the others won't take 360 GB of data but less because of the bad balancing. For example, OSD 28 will only get around 233 GB of data by the time OSD 70 has 360 GB.
Paul
, which is still far short of 37TiB raw free, as expected. However, what doesn't track is the 5.6TiB available at the pools level, even for a 3x replicated pool (5.6*3=16.8TiB, which is about 34% less than my napkin math, which would be 22.8/3=7.6TiB. But what tracks even less is the hybrid pools, which use 1/3 of what the 3x-replicated data consumes. Meaning if my napkin math is right, should show ~22.8TiB free.
Am I grossly mis-understanding how this is calculated? Maybe this is fixed in Octopus?
Just trying to get a grasp on what I'm seeing not matching expectations.
Thanks,
Reed _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Paul Emmerich
-
Reed Dier