Erasure coding best practice
Hi, Couple of time I saw some use-case in the mail list where people using erasure code on a "weird" way. Normally we use more data chunks than coding chunks, like k=4 m=2 or k=8 m=3. What I mean "weird" I saw someone using k=3 and m=8 (I might remember wrong). I'm trying to understand what is the benefit of the higher coding chunks? You can use smaller object size? Let's say 4:2 the minimum object size should be 24K at least or with 8:3 it would be 44K because nothing will be stored on smaller space. In case of k=3 m=8, smallest object can be 12K, but you can lose 8 nodes (pgs) and data still there? Ty
Hello Szabo, Some of these "weird" erasure coding setups come from old-style stretch clusters where the cluster was designed to withstand the loss of one datacenter out of two. For example, a 2+4 EC setup could be used together with a rule that selects three hosts from one datacenter and three from the other. On paper, this offers a space advantage over the official stretch cluster setup, which requires four replicas. Please note that this practice itself is questionable, as it is possible to satisfy the requirement of having min_size up-to-date shards total without having enough shards in each datacenter, which is what matters for data safety. In other words, there are scenarios involving partial recovery followed by another incident where such old-style stretch clusters might lose data. Don't do that. On Tue, Dec 17, 2024 at 1:03 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi,
Couple of time I saw some use-case in the mail list where people using erasure code on a "weird" way. Normally we use more data chunks than coding chunks, like k=4 m=2 or k=8 m=3. What I mean "weird" I saw someone using k=3 and m=8 (I might remember wrong).
I'm trying to understand what is the benefit of the higher coding chunks? You can use smaller object size? Let's say 4:2 the minimum object size should be 24K at least or with 8:3 it would be 44K because nothing will be stored on smaller space. In case of k=3 m=8, smallest object can be 12K, but you can lose 8 nodes (pgs) and data still there?
Ty _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
To be honest with 3:8 we could protect the cluster more from osd flapping. Let's say you have less chance to have 8 down pgs on 8 separate nodes then with 8:3 only 3pgs on 3 nodes. Of course this comes with the cost on storage used. Is there any disadvantage performance wise on this? Istvan ________________________________ From: Alexander Patrakov <patrakov@gmail.com> Sent: Tuesday, December 17, 2024 4:48:25 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Erasure coding best practice Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Hello Szabo, Some of these "weird" erasure coding setups come from old-style stretch clusters where the cluster was designed to withstand the loss of one datacenter out of two. For example, a 2+4 EC setup could be used together with a rule that selects three hosts from one datacenter and three from the other. On paper, this offers a space advantage over the official stretch cluster setup, which requires four replicas. Please note that this practice itself is questionable, as it is possible to satisfy the requirement of having min_size up-to-date shards total without having enough shards in each datacenter, which is what matters for data safety. In other words, there are scenarios involving partial recovery followed by another incident where such old-style stretch clusters might lose data. Don't do that. On Tue, Dec 17, 2024 at 1:03 PM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi,
Couple of time I saw some use-case in the mail list where people using erasure code on a "weird" way. Normally we use more data chunks than coding chunks, like k=4 m=2 or k=8 m=3. What I mean "weird" I saw someone using k=3 and m=8 (I might remember wrong).
I'm trying to understand what is the benefit of the higher coding chunks? You can use smaller object size? Let's say 4:2 the minimum object size should be 24K at least or with 8:3 it would be 44K because nothing will be stored on smaller space. In case of k=3 m=8, smallest object can be 12K, but you can lose 8 nodes (pgs) and data still there?
Ty _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
To be honest with 3:8 we could protect the cluster more from osd flapping. Let's say you have less chance to have 8 down pgs on 8 separate nodes then with 8:3 only 3pgs on 3 nodes. Of course this comes with the cost on storage used. Is there any disadvantage performance wise on this?
A few years back someone asserted that EC values with small prime factors are advantageous, so 23,11 would be doubleplus ungood. In general EC comes with a write tradeoff: each write (today) touches k+m hosts/OSDs, which means increased network amp and write IOPs amp. The latter can be painful for rotational OSDs. This additionally means that recovery/backfill are slower, as more OSDs are tied up with each op and thus fewer PGs can recover in parallel.
I'm trying to understand what is the benefit of the higher coding chunks? You can use smaller object size? Let's say 4:2 the minimum object size should be 24K at least or with 8:3 it would be 44K because nothing will be stored on smaller space. In case of k=3 m=8, smallest object can be 12K, but you can lose 8 nodes (pgs) and data still there?
BlueStore will allocate no less than min_alloc_size on a given OSD; the default is 4KB for most media. I think the above implies that parity shards don’t take space, but in fact an m shard uses space just like a k shard, so for space efficiency what matters is the sum of k+m, not their specific values. https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToPQjN8bDI/edit?gid=358760253#gid=358760253 Bluestore Space Amplification Cheat Sheet docs.google.com
Ty _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
To be honest with 3:8 we could protect the cluster more from osd flapping. Let's say you have less chance to have 8 down pgs on 8 separate nodes then with 8:3 only 3pgs on 3 nodes. Of course this comes with the cost on storage used. Is there any disadvantage performance wise on this?
A few years back someone asserted that EC values with small prime factors are advantageous, so 23,11 would be doubleplus ungood.
I thought it was that K should preferably be a power of two, M as many as your security demands require. Also pools should have power-of-two PGs, and bucket shards would be primes. I could be wrong though. -- May the most significant bit of your life be positive.
Just repeating what I read. I suspect that the effect is minimal. Back when I did ZFS a lot there was conventional wisdom of a given party group not having more than 9 drives, to keep rebuild and writes semi-manageable.
A few years back someone asserted that EC values with small prime factors are advantageous, so 23,11 would be doubleplus ungood.
I thought it was that K should preferably be a power of two, M as many as your security demands require. Also pools should have power-of-two PGs, and bucket shards would be primes.
I could be wrong though.
-- May the most significant bit of your life be positive.
I would like to know more about those corner cases and why it’s not recommended to use this approach. Because our customers and we ourselves have been using such profiles for years, including multiple occasions when one of two DCs failed with k7m11. They were quite happy with the resiliency Ceph provided. Zitat von Anthony D'Atri <aad@dreamsnake.net>:
Just repeating what I read. I suspect that the effect is minimal.
Back when I did ZFS a lot there was conventional wisdom of a given party group not having more than 9 drives, to keep rebuild and writes semi-manageable.
A few years back someone asserted that EC values with small prime factors are advantageous, so 23,11 would be doubleplus ungood.
I thought it was that K should preferably be a power of two, M as many as your security demands require. Also pools should have power-of-two PGs, and bucket shards would be primes.
I could be wrong though.
-- May the most significant bit of your life be positive.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Alexander Patrakov
-
Anthony D'Atri
-
Eugen Block
-
Janne Johansson
-
Szabo, Istvan (Agoda)