Erasure coding scheme 2+4 = good idea?
Hi all, I'd like your thoughts and comments on this idea: Setup: - two fault domains = DCs - connected with 100 GBit, < 1 ms - 80 NVME SSDs on each side Goal: One fault domain can be lost, and then there's still have some redundancy. Option 1: Replicated pool with size = 4. This gives me two copies on each side, thus meets the goal. But the efficiency is only 25%. Option 2: Erasure coded pool with 2 + 4 scheme. This gives me 3 chunks on each side. If I lose one side, I still have 3 chunks left, where I only need 2. Thus the goal is also met. Efficiency is 33%. Even though I did a lot of googling, I couldn't find anything about a similar setup. In all profiles, there is k <= m. What do you think about 2+4, is it a good idea or a bad one, or do I miss something and it doesn't work at all? In particular: is it possible to recover two data chunks out of 2 coding chunks? As I read the documentation, this should be no problem, just want to confirm. -- Andre Tann
Hi Andre, Your setup and thoughts make good sense,but are somewhat unusual. Having only two instances of your failure domain limits you quite a lot, and goes against common best practices, which says at least 3 and preferably four. It is understandable that you are limited to 2, when your fault domain is as large as a datacenter, but it does limit your options somewhat. The reason why you can't find information on erasure coding where k>m is probably because of the low storage efficiency this provides, and people tend to choose EC to improve storage efficiency. Both setups that you suggest yourself ( 4 x replication and 2+4 EC should meet your demands. Given how close they are in storage efficiency ( 25% vs 33%), I would lean towards 4x replication. It is a lot simpler to quickly understand the degree of redundancy available, both during initial planning, and when facing an actual potentially stressful error scenario with degraded availability, compared to Erasure Coding which requires a bit more mental arithmetic to figure out. how badly screwed you are in a given situation. So assuming, that the ideal situation of spreading your OSDs across 3 or 4 data centers is not an option( this would allow you to safely go with the default 3x replication), I would clearly lean towards 4x replication with 2 copies in each DC, but a 2+4 EC solution also seems like a viable choice offering a slightly better storage efficiency, at the expense of increased complexity. Best Regards, Simon Kepp, Founder, CTO Kepp Technologies On Thu, Oct 10, 2024 at 1:17 AM Andre Tann <atann@alphasrv.net> wrote:
Hi all,
I'd like your thoughts and comments on this idea:
Setup: - two fault domains = DCs - connected with 100 GBit, < 1 ms - 80 NVME SSDs on each side
Goal: One fault domain can be lost, and then there's still have some redundancy.
Option 1: Replicated pool with size = 4. This gives me two copies on each side, thus meets the goal. But the efficiency is only 25%.
Option 2: Erasure coded pool with 2 + 4 scheme. This gives me 3 chunks on each side. If I lose one side, I still have 3 chunks left, where I only need 2. Thus the goal is also met. Efficiency is 33%.
Even though I did a lot of googling, I couldn't find anything about a similar setup. In all profiles, there is k <= m.
What do you think about 2+4, is it a good idea or a bad one, or do I miss something and it doesn't work at all?
In particular: is it possible to recover two data chunks out of 2 coding chunks? As I read the documentation, this should be no problem, just want to confirm.
-- Andre Tann _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
Is it possible to recover two data chunks out of 2 coding chunks?
Yes.
What do you think about 2+4, is it a good idea or a bad one?
Some differences between a replica pool and an erasure code pool to consider: 1. If an OSD fails there will be a lot more network traffic between the sites to backfill/recover an erasure coded pool than a replica pool. 2. Replica pools can be configured to support local reads where clients send read I/O requests to an OSD at the same site. For erasure coded pools all read I/O must be sent via the primary OSD which half the time will be on the remote site. 3. In general, there will be more inter-site traffic with an erasure code pool than a replica pool. If the round trip latency between sites is really low and you have plenty of network bandwidth this might not be an issue, otherwise this will impact performance. Cheers, Bill. bill_scales@uk.ibm.com<mailto:bill_scales@uk.ibm.com> IBM Distinguished Engineer, IBM Storage From: Andre Tann <atann@alphasrv.net> Date: Thursday, 10 October 2024 at 00:16 To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [EXTERNAL] [ceph-users] Erasure coding scheme 2+4 = good idea? Hi all, I'd like your thoughts and comments on this idea: Setup: - two fault domains = DCs - connected with 100 GBit, < 1 ms - 80 NVME SSDs on each side Goal: One fault domain can be lost, and then there's still have some redundancy. Option 1: Replicated pool with size = 4. This gives me two copies on each side, thus meets the goal. But the efficiency is only 25%. Option 2: Erasure coded pool with 2 + 4 scheme. This gives me 3 chunks on each side. If I lose one side, I still have 3 chunks left, where I only need 2. Thus the goal is also met. Efficiency is 33%. Even though I did a lot of googling, I couldn't find anything about a similar setup. In all profiles, there is k <= m. What do you think about 2+4, is it a good idea or a bad one, or do I miss something and it doesn't work at all? In particular: is it possible to recover two data chunks out of 2 coding chunks? As I read the documentation, this should be no problem, just want to confirm. -- Andre Tann _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
I would like to recommend to look at stretch-mode. There have been discussions on this list about the reliability of 2-DC set-ups and just using crush rules to distribute shards doesn't cut it. There are corner cases that are only handled correctly by stretch mode if the system needs to be up with a DC down. For example, it will make sure that with min_size=2 an ACK is only sent to a client if each DC has a shard. An ordinary crush rule will not do that. Stretch mode only works for replicated pools. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Hi Frank, Am 10.10.24 um 11:13 schrieb Frank Schilder:
I would like to recommend to look at stretch-mode. There have been discussions on this list about the reliability of 2-DC set-ups and just using crush rules to distribute shards doesn't cut it. There are corner cases that are only handled correctly by stretch mode if the system needs to be up with a DC down. For example, it will make sure that with min_size=2 an ACK is only sent to a client if each DC has a shard. An ordinary crush rule will not do that.
Stretch mode only works for replicated pools. Thanks for pointing out this one, I think Bills idea in point 2 goes into the same direction. To avoid unneccessary network traffic, we definitely need to consider stretch mode.
-- Andre Tann
Hi Bill Am 10.10.24 um 10:57 schrieb Bill Scales:
2. Replica pools can be configured to support local reads where clients send read I/O requests to an OSD at the same site. For erasure coded pools all read I/O must be sent via the primary OSD which half the time will be on the remote site.
This is a good hint, never heard of this feature. Do you happen to have a further link? On first googling, I didn't see any results. Ah, you mean stretch clusters? -- Andre Tann
participants (4)
-
Andre Tann
-
Bill Scales
-
Frank Schilder
-
Simon Kepp