That's what the formula on the ceph link arrives at, a 2/3 or 66.66% overhead. But if a 4 byte object is split into 4x1 byte chunks data (4 bytes total) + 2x 1 byte chunks parity (2 bytes total), you arrive at 6 bytes, which is 50% more than 4 bytes. So 50% overhead, vs. 33.33% overhead as the other formula arrives at. I'm curious what I'm missing. On Tue, Jul 28, 2020 at 3:40 PM Alan Johnson <alanj@supermicro.com> wrote:

It would be 4/(4+2) = 4/6 =2/3 or k/(k+m)?

-----Original Message----- From: David Orman <ormandj@corenode.com> Sent: Tuesday, July 28, 2020 9:32 PM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Usable space vs. Overhead

[CAUTION: External Mail]

I'm having a hard time understanding the EC usable space vs. raw.

https://urldefense.com/v3/__https://ceph.io/geen-categorie/ceph-erasure-codi... indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space, but that's not lining up with what i'd expect just from k data chunks + m parity chunks.

So, for example, k=4, m=2. you'd expect every 4 byte object written would consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw storage.

What am I missing here? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Den ons 29 juli 2020 kl 03:17 skrev David Orman <ormandj@corenode.com>:

That's what the formula on the ceph link arrives at, a 2/3 or 66.66% overhead. But if a 4 byte object is split into 4x1 byte chunks data (4 bytes total) + 2x 1 byte chunks parity (2 bytes total), you arrive at 6 bytes, which is 50% more than 4 bytes. So 50% overhead, vs. 33.33% overhead as the other formula arrives at. I'm curious what I'm missing.

Are you sure you are not just mixing up overhead with usable %? 50% overhead means you write 4 bytes, get 2 bytes "extra" for a total of 6. In this case 4 out of 6 is 66.67% usable space, i.e. two thirds. So if the formula says you will get 66% usable it means you get two-thirds usable out of your drives with EC4+2, and it can also be said that the data is 100%, and the overhead is 50% of that, but you need to know which of the figures you want to calculate. Either "how large is the growth of the data I put in" OR "How much of the stored data is my original bytes and how much in percent is the checksums". For 4+2, the growth is 50%, since you add two (50% of four) to 4 original bytes, and for a six-drive setting, two drives go to checksums so you only get 66% usable if you fill that cluster up. The space allocated to checksums (33%) is "50% of 66%" so the overhead is still 50% no matter how you calculate i -- May the most significant bit of your life be positive.

Den ons 29 juli 2020 kl 16:34 skrev David Orman <ormandj@corenode.com>:

Thank you, everyone, for the help. I absolutely was mixing up the two, which is why I was asking for guidance. The example made it clear. The question I was trying to answer was: what would the capacity of the cluster be, for actual data, based on the raw disk space + server/drive count + erasure coding profile. It sounds like the 'usable' calculation (66% in this case) is the accurate number, assuming I were to fill the cluster to 100%, which I realize is not ideal with Ceph.

It is bad on almost all kinds of storage systems to fill it up like that, any storage that has any concept of data that can move (so excluding tapes or CDroms more or less) will want to have some extra space, and ceph will start to warn/act/refuse when you pass 85,90,95% filled, so aim for something where you will be starting to buy more nodes/disks when your first OSD is over 70 or so, otherwise you will be doing a lot of manual work like rebalancing and reweighing in order to not go above 85 until your new drives can be added to the system. If you have few nodes, one host outage will represent a large part of the available storage, so one can make all kinds of calculations on overhead and things like "with EC4+2 I can lose two drives and still recover", but if you only have 6 hosts and one goes dead (for any reason), your total has fallen with 16.7% so if you were at some 70% full with 6 hosts, you are going to be all but totally filled up with only 5 which will cause issues (like OSDs refusing IO to not move to 100% full), even if you only lost one of each EC4+2-group from that host. -- May the most significant bit of your life be positive.