28 Jul
2020
28 Jul
'20
5:32 p.m.
I'm having a hard time understanding the EC usable space vs. raw.
https://ceph.io/geen-categorie/ceph-erasure-coding-overhead-in-a-nutshell/
indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space,
but that's not lining up with what i'd expect just from k data chunks + m
parity chunks.
So, for example, k=4, m=2. you'd expect every 4 byte object written would
consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server
cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw
storage.
What am I missing here?
28 Jul
28 Jul
5:40 p.m.
It would be 4/(4+2) = 4/6 =2/3 or k/(k+m)?
-----Original Message-----
From: David Orman <ormandj(a)corenode.com>
Sent: Tuesday, July 28, 2020 9:32 PM
To: ceph-users <ceph-users(a)ceph.io>
Subject: [ceph-users] Usable space vs. Overhead
[CAUTION: External Mail]
I'm having a hard time understanding the EC usable space vs. raw.
https://urldefense.com/v3/__https://ceph.io/geen-categorie/ceph-erasure-cod…
indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space, but
that's not lining up with what i'd expect just from k data chunks + m parity
chunks.
So, for example, k=4, m=2. you'd expect every 4 byte object written would consume 6
bytes, so 50% overhead. however, the prior formula in a 7 server cluster, using 4+2
encoding, would indicate 66.67% usable capacity vs. raw storage.
What am I missing here?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
10:16 p.m.
That's what the formula on the ceph link arrives at, a 2/3 or 66.66%
overhead. But if a 4 byte object is split into 4x1 byte chunks data (4
bytes total) + 2x 1 byte chunks parity (2 bytes total), you arrive at 6
bytes, which is 50% more than 4 bytes. So 50% overhead, vs. 33.33% overhead
as the other formula arrives at. I'm curious what I'm missing.
On Tue, Jul 28, 2020 at 3:40 PM Alan Johnson <alanj(a)supermicro.com> wrote:
It would be 4/(4+2) = 4/6 =2/3 or k/(k+m)?
-----Original Message-----
From: David Orman <ormandj(a)corenode.com>
Sent: Tuesday, July 28, 2020 9:32 PM
To: ceph-users <ceph-users(a)ceph.io>
Subject: [ceph-users] Usable space vs. Overhead
[CAUTION: External Mail]
I'm having a hard time understanding the EC usable space vs. raw.
https://urldefense.com/v3/__https://ceph.io/geen-categorie/ceph-erasure-cod…
indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space,
but that's not lining up with what i'd expect just from k data chunks + m
parity chunks.
So, for example, k=4, m=2. you'd expect every 4 byte object written would
consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server
cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw
storage.
What am I missing here?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
29 Jul
29 Jul
12:02 a.m.
A k=3, m=3 scheme would be 3:3 = 50% , you get to use 4 bytes out of 6 bytes = 4:6 = 2:3 =
66.6%?
From: David Orman <ormandj(a)corenode.com>
Sent: Wednesday, July 29, 2020 2:17 AM
To: Alan Johnson (System) <alanj(a)supermicro.com>
Cc: ceph-users <ceph-users(a)ceph.io>
Subject: Re: [ceph-users] Usable space vs. Overhead
[CAUTION: External Mail]
That's what the formula on the ceph link arrives at, a 2/3 or 66.66% overhead. But if
a 4 byte object is split into 4x1 byte chunks data (4 bytes total) + 2x 1 byte chunks
parity (2 bytes total), you arrive at 6 bytes, which is 50% more than 4 bytes. So 50%
overhead, vs. 33.33% overhead as the other formula arrives at. I'm curious what
I'm missing.
On Tue, Jul 28, 2020 at 3:40 PM Alan Johnson
<alanj@supermicro.com<mailto:alanj@supermicro.com>> wrote:
It would be 4/(4+2) = 4/6 =2/3 or k/(k+m)?
-----Original Message-----
From: David Orman <ormandj@corenode.com<mailto:ormandj@corenode.com>>
Sent: Tuesday, July 28, 2020 9:32 PM
To: ceph-users <ceph-users@ceph.io<mailto:ceph-users@ceph.io>>
Subject: [ceph-users] Usable space vs. Overhead
[CAUTION: External Mail]
I'm having a hard time understanding the EC usable space vs. raw.
https://urldefense.com/v3/__https://ceph.io/geen-categorie/ceph-erasure-cod…
indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space, but
that's not lining up with what i'd expect just from k data chunks + m parity
chunks.
So, for example, k=4, m=2. you'd expect every 4 byte object written would consume 6
bytes, so 50% overhead. however, the prior formula in a 7 server cluster, using 4+2
encoding, would indicate 66.67% usable capacity vs. raw storage.
What am I missing here?
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To
unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
4:26 a.m.
Den ons 29 juli 2020 kl 03:17 skrev David Orman <ormandj(a)corenode.com>om>:
That's what the formula on the ceph link arrives
at, a 2/3 or 66.66%
overhead. But if a 4 byte object is split into 4x1 byte chunks data (4
bytes total) + 2x 1 byte chunks parity (2 bytes total), you arrive at 6
bytes, which is 50% more than 4 bytes. So 50% overhead, vs. 33.33% overhead
as the other formula arrives at. I'm curious what I'm missing.
Are you sure you are not just mixing up overhead with usable %?
50% overhead means you write 4 bytes, get 2 bytes "extra" for a total of 6.
In this case 4 out of 6 is 66.67% usable space, i.e. two thirds.
So if the formula says you will get 66% usable it means you get two-thirds
usable out of your drives with EC4+2, and it can also be said that the data
is 100%, and the overhead is 50% of that, but you need to know which of
the figures you want to calculate.
Either "how large is the growth of the data I put in"
OR "How much of the stored data is my original bytes and how much
in percent is the checksums".
For 4+2, the growth is 50%, since you add two (50% of four) to 4 original
bytes,
and for a six-drive setting, two drives go to checksums so you only get 66%
usable
if you fill that cluster up. The space allocated to checksums (33%)
is "50% of 66%" so the overhead is still 50% no matter how you calculate i
--
May the most significant bit of your life be positive.
11:33 a.m.
Hi,
Thank you, everyone, for the help. I absolutely was mixing up the two,
which is why I was asking for guidance. The example made it clear. The
question I was trying to answer was: what would the capacity of the cluster
be, for actual data, based on the raw disk space + server/drive count +
erasure coding profile. It sounds like the 'usable' calculation (66% in
this case) is the accurate number, assuming I were to fill the cluster to
100%, which I realize is not ideal with Ceph.
Respectfully,
David Orman
On Wed, Jul 29, 2020 at 2:27 AM Janne Johansson <icepic.dz(a)gmail.com> wrote:
Den ons 29 juli 2020 kl 03:17 skrev David Orman <ormandj(a)corenode.com>om>:
That's what the formula on the ceph link
arrives at, a 2/3 or 66.66%
overhead. But if a 4 byte object is split into 4x1 byte chunks data (4
bytes total) + 2x 1 byte chunks parity (2 bytes total), you arrive at 6
bytes, which is 50% more than 4 bytes. So 50% overhead, vs. 33.33%
overhead
as the other formula arrives at. I'm curious what I'm missing.
Are you sure you are not just mixing up overhead with usable %?
50% overhead means you write 4 bytes, get 2 bytes "extra" for a total of 6.
In this case 4 out of 6 is 66.67% usable space, i.e. two thirds.
So if the formula says you will get 66% usable it means you get two-thirds
usable out of your drives with EC4+2, and it can also be said that the data
is 100%, and the overhead is 50% of that, but you need to know which of
the figures you want to calculate.
Either "how large is the growth of the data I put in"
OR "How much of the stored data is my original bytes and how much
in percent is the checksums".
For 4+2, the growth is 50%, since you add two (50% of four) to 4 original
bytes,
and for a six-drive setting, two drives go to checksums so you only get
66% usable
if you fill that cluster up. The space allocated to checksums (33%)
is "50% of 66%" so the overhead is still 50% no matter how you calculate i
--
May the most significant bit of your life be positive.
12:03 p.m.
Den ons 29 juli 2020 kl 16:34 skrev David Orman <ormandj(a)corenode.com>om>:
Thank you, everyone, for the help. I absolutely was
mixing up the two,
which is why I was asking for guidance. The example made it clear. The
question I was trying to answer was: what would the capacity of the cluster
be, for actual data, based on the raw disk space + server/drive count +
erasure coding profile. It sounds like the 'usable' calculation (66% in
this case) is the accurate number, assuming I were to fill the cluster to
100%, which I realize is not ideal with Ceph.
It is bad on almost all kinds of storage systems to fill it up like that,
any storage that has any concept of data that can move (so excluding tapes
or CDroms more or less) will want to have some extra space, and ceph will
start to warn/act/refuse when you pass 85,90,95% filled, so aim for
something where you will be starting to buy more nodes/disks when your
first OSD is over 70 or so, otherwise you will be doing a lot of manual
work like rebalancing and reweighing in order to not go above 85 until your
new drives can be added to the system.
If you have few nodes, one host outage will represent a large part of the
available storage, so one can make all kinds of calculations on overhead
and things like "with EC4+2 I can lose two drives and still recover", but
if you only have 6 hosts and one goes dead (for any reason), your total has
fallen with 16.7% so if you were at some 70% full with 6 hosts, you are
going to be all but totally filled up with only 5 which will cause issues
(like OSDs refusing IO to not move to 100% full), even if you only lost one
of each EC4+2-group from that host.
--
May the most significant bit of your life be positive.
4:19 a.m.
Aren't you just looking at the same thing from two different perspective?
In one case you say: I have 100% of useful data, and I need to add 50% of parity for a
total of 150% raw data.
In the other, you say: Out of 100% of raw data, 2/3 is useful data, 1/3 is parity, which
gives you your 33.3% overhead.
But it's the exact same thing, it just depends on whether you consider your overhead
as a percentage of total (raw) data, or as a percentage of useful data.
--
Ben
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, July 28, 2020 10:32 PM, David Orman <ormandj(a)corenode.com> wrote:
I'm having a hard time understanding the EC usable
space vs. raw.
https://ceph.io/geen-categorie/ceph-erasure-coding-overhead-in-a-nutshell/
indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space,
but that's not lining up with what i'd expect just from k data chunks + m
parity chunks.
So, for example, k=4, m=2. you'd expect every 4 byte object written would
consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server
cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw
storage.
What am I missing here?
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1372
days inactive
1373
days old
7 comments
4 participants
participants (4)
-
Alan Johnson -
Benoît Knecht -
David Orman -
Janne Johansson