EC profiles where m>k (EC 8+12)
Hi Ceph users! I've been proposed an interesting EC setup I hadn't thought about before. Scenario is : we have two server rooms and want to store ~4PiB with the ability to loose 1 server room without loss of data or RW availability. For the context, performance is not needed (cold storage mostly, used as a big filesystem). The idea is to use EC 8+12 over 24 servers (12 on each server room), so if we loose 1 room we still have half of the EC parts (10/20) and are able to loose 2 more servers before reaching the point where we loose data. I find this pretty elegant when working on a two-sites context, as efficiency is 40% (better than 33% three times replication) and the redundancy is good. What do you think of this setup ? Did you ever used EC profiles with M > K ? Thanks for sharing your thoughts! Cheers, Fabien
Hi, we have multiple customers with such profiles, for example one with k7 m11 for a two-site cluster (in total 20 nodes). The customer is pretty happy with the resiliency because they actually had multiple outages of one DC and everything was still working fine. Although there's also the stretch mode (which I haven't tested properly yet) I can encourage you to use such a profile. Just be advised to properly test your crush rule. ;-) Regards, Eugen Zitat von Fabien Sirjean <fsirjean@eddie.fdn.fr>:
Hi Ceph users!
I've been proposed an interesting EC setup I hadn't thought about before.
Scenario is : we have two server rooms and want to store ~4PiB with the ability to loose 1 server room without loss of data or RW availability.
For the context, performance is not needed (cold storage mostly, used as a big filesystem).
The idea is to use EC 8+12 over 24 servers (12 on each server room), so if we loose 1 room we still have half of the EC parts (10/20) and are able to loose 2 more servers before reaching the point where we loose data.
I find this pretty elegant when working on a two-sites context, as efficiency is 40% (better than 33% three times replication) and the redundancy is good.
What do you think of this setup ? Did you ever used EC profiles with M > K ?
Thanks for sharing your thoughts!
Cheers,
Fabien _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, thanks for your reply! Stretch mode is obviously useful with small pools, but with its size of 4 this is a 25% efficiency and we can't afford it (buying 16 PiB raw for 4 PiB net, it's quite hard to justify to budget holders...). Good to hear that you used such EC setup in prod, thanks for sharing! Cheers, F. On 3/24/23 13:11, Eugen Block wrote:
Hi,
we have multiple customers with such profiles, for example one with k7 m11 for a two-site cluster (in total 20 nodes). The customer is pretty happy with the resiliency because they actually had multiple outages of one DC and everything was still working fine. Although there's also the stretch mode (which I haven't tested properly yet) I can encourage you to use such a profile. Just be advised to properly test your crush rule. ;-)
Regards, Eugen
Zitat von Fabien Sirjean <fsirjean@eddie.fdn.fr>:
Hi Ceph users!
I've been proposed an interesting EC setup I hadn't thought about before.
Scenario is : we have two server rooms and want to store ~4PiB with the ability to loose 1 server room without loss of data or RW availability.
For the context, performance is not needed (cold storage mostly, used as a big filesystem).
The idea is to use EC 8+12 over 24 servers (12 on each server room), so if we loose 1 room we still have half of the EC parts (10/20) and are able to loose 2 more servers before reaching the point where we loose data.
I find this pretty elegant when working on a two-sites context, as efficiency is 40% (better than 33% three times replication) and the redundancy is good.
What do you think of this setup ? Did you ever used EC profiles with M > K ?
Thanks for sharing your thoughts!
Cheers,
Fabien _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The question I have regarding this setup is, how can you guarantee that the 12 m chunks will be located evenly across the two rooms. What would happen if by chance all 12 chunks were in room B? Usually you use failure domains to make sure of the distribution of chunks across domains, but you can't do that here as you are using host as failure domain, but need room to be somehow included in that. ________________________________ From: Fabien Sirjean <fsirjean@eddie.fdn.fr> Sent: 24 March 2023 12:00 To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] EC profiles where m>k (EC 8+12) CAUTION: This email originates from outside THG Hi Ceph users! I've been proposed an interesting EC setup I hadn't thought about before. Scenario is : we have two server rooms and want to store ~4PiB with the ability to loose 1 server room without loss of data or RW availability. For the context, performance is not needed (cold storage mostly, used as a big filesystem). The idea is to use EC 8+12 over 24 servers (12 on each server room), so if we loose 1 room we still have half of the EC parts (10/20) and are able to loose 2 more servers before reaching the point where we loose data. I find this pretty elegant when working on a two-sites context, as efficiency is 40% (better than 33% three times replication) and the redundancy is good. What do you think of this setup ? Did you ever used EC profiles with M > K ? Thanks for sharing your thoughts! Cheers, Fabien _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Danny Webb Principal OpenStack Engineer Danny.Webb@thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>
A custom CRUSH rule can have two steps to enforce that.
On Mar 24, 2023, at 11:04, Danny Webb <Danny.Webb@thehutgroup.com> wrote:
The question I have regarding this setup is, how can you guarantee that the 12 m chunks will be located evenly across the two rooms. What would happen if by chance all 12 chunks were in room B? Usually you use failure domains to make sure of the distribution of chunks across domains, but you can't do that here as you are using host as failure domain, but need room to be somehow included in that. ________________________________ From: Fabien Sirjean <fsirjean@eddie.fdn.fr> Sent: 24 March 2023 12:00 To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] EC profiles where m>k (EC 8+12)
CAUTION: This email originates from outside THG
Hi Ceph users!
I've been proposed an interesting EC setup I hadn't thought about before.
Scenario is : we have two server rooms and want to store ~4PiB with the ability to loose 1 server room without loss of data or RW availability.
For the context, performance is not needed (cold storage mostly, used as a big filesystem).
The idea is to use EC 8+12 over 24 servers (12 on each server room), so if we loose 1 room we still have half of the EC parts (10/20) and are able to loose 2 more servers before reaching the point where we loose data.
I find this pretty elegant when working on a two-sites context, as efficiency is 40% (better than 33% three times replication) and the redundancy is good.
What do you think of this setup ? Did you ever used EC profiles with M > K ?
Thanks for sharing your thoughts!
Cheers,
Fabien _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Danny Webb Principal OpenStack Engineer Danny.Webb@thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Fabien, we have also used it several times for 2 DC setups. However, we always try to use as few chunks as possible, as it is very inefficient when storing small files (min alloc size) and it can also lead to quite some problems with backfill and recovery in large ceph clusters. Joachim ___________________________________ Clyso GmbH - Ceph Foundation Member Am 24.03.23 um 13:00 schrieb Fabien Sirjean:
Hi Ceph users!
I've been proposed an interesting EC setup I hadn't thought about before.
Scenario is : we have two server rooms and want to store ~4PiB with the ability to loose 1 server room without loss of data or RW availability.
For the context, performance is not needed (cold storage mostly, used as a big filesystem).
The idea is to use EC 8+12 over 24 servers (12 on each server room), so if we loose 1 room we still have half of the EC parts (10/20) and are able to loose 2 more servers before reaching the point where we loose data.
I find this pretty elegant when working on a two-sites context, as efficiency is 40% (better than 33% three times replication) and the redundancy is good.
What do you think of this setup ? Did you ever used EC profiles with M
K ?
Thanks for sharing your thoughts!
Cheers,
Fabien _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Clyso GmbH - Ceph Foundation Member
-
Danny Webb
-
Eugen Block
-
Fabien Sirjean