24 Mar
2023
24 Mar
'23
5:30 p.m.
Hi Ceph users!
I've been proposed an interesting EC setup I hadn't thought about before.
Scenario is : we have two server rooms and want to store ~4PiB with the
ability to loose 1 server room without loss of data or RW availability.
For the context, performance is not needed (cold storage mostly, used as
a big filesystem).
The idea is to use EC 8+12 over 24 servers (12 on each server room), so
if we loose 1 room we still have half of the EC parts (10/20) and are
able to loose 2 more servers before reaching the point where we loose data.
I find this pretty elegant when working on a two-sites context, as
efficiency is 40% (better than 33% three times replication) and the
redundancy is good.
What do you think of this setup ? Did you ever used EC profiles with M > K ?
Thanks for sharing your thoughts!
Cheers,
Fabien
24 Mar
24 Mar
5:41 p.m.
Hi,
we have multiple customers with such profiles, for example one with k7
m11 for a two-site cluster (in total 20 nodes). The customer is pretty
happy with the resiliency because they actually had multiple outages
of one DC and everything was still working fine. Although there's also
the stretch mode (which I haven't tested properly yet) I can encourage
you to use such a profile. Just be advised to properly test your crush
rule. ;-)
Regards,
Eugen
Zitat von Fabien Sirjean <fsirjean(a)eddie.fdn.fr>fr>:
Hi Ceph users!
I've been proposed an interesting EC setup I hadn't thought about before.
Scenario is : we have two server rooms and want to store ~4PiB with
the ability to loose 1 server room without loss of data or RW
availability.
For the context, performance is not needed (cold storage mostly,
used as a big filesystem).
The idea is to use EC 8+12 over 24 servers (12 on each server room),
so if we loose 1 room we still have half of the EC parts (10/20) and
are able to loose 2 more servers before reaching the point where we
loose data.
I find this pretty elegant when working on a two-sites context, as
efficiency is 40% (better than 33% three times replication) and the
redundancy is good.
What do you think of this setup ? Did you ever used EC profiles with M > K ?
Thanks for sharing your thoughts!
Cheers,
Fabien
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:04 p.m.
Hi, thanks for your reply!
Stretch mode is obviously useful with small pools, but with its size of
4 this is a 25% efficiency and we can't afford it (buying 16 PiB raw for
4 PiB net, it's quite hard to justify to budget holders...).
Good to hear that you used such EC setup in prod, thanks for sharing!
Cheers,
F.
On 3/24/23 13:11, Eugen Block wrote:
> Hi,
>
> we have multiple customers with such profiles, for example one with k7
> m11 for a two-site cluster (in total 20 nodes). The customer is pretty
> happy with the resiliency because they actually had multiple outages
> of one DC and everything was still working fine. Although there's also
> the stretch mode (which I haven't tested properly yet) I can encourage
> you to use such a profile. Just be advised to properly test your crush
> rule. ;-)
>
> Regards,
> Eugen
>
> Zitat von Fabien Sirjean <fsirjean(a)eddie.fdn.fr>fr>:
>
>> Hi Ceph users!
>>
>> I've been proposed an interesting EC setup I hadn't thought about
>> before.
>>
>> Scenario is : we have two server rooms and want to store ~4PiB with
>> the ability to loose 1 server room without loss of data or RW
>> availability.
>>
>> For the context, performance is not needed (cold storage mostly, used
>> as a big filesystem).
>>
>> The idea is to use EC 8+12 over 24 servers (12 on each server room),
>> so if we loose 1 room we still have half of the EC parts (10/20) and
>> are able to loose 2 more servers before reaching the point where we
>> loose data.
>>
>> I find this pretty elegant when working on a two-sites context, as
>> efficiency is 40% (better than 33% three times replication) and the
>> redundancy is good.
>>
>> What do you think of this setup ? Did you ever used EC profiles with
>> M > K ?
>>
>> Thanks for sharing your thoughts!
>>
>> Cheers,
>>
>> Fabien
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:34 p.m.
The question I have regarding this setup is, how can you guarantee that the 12 m chunks
will be located evenly across the two rooms. What would happen if by chance all 12 chunks
were in room B? Usually you use failure domains to make sure of the distribution of
chunks across domains, but you can't do that here as you are using host as failure
domain, but need room to be somehow included in that.
________________________________
From: Fabien Sirjean <fsirjean(a)eddie.fdn.fr>
Sent: 24 March 2023 12:00
To: ceph-users <ceph-users(a)ceph.io>
Subject: [ceph-users] EC profiles where m>k (EC 8+12)
CAUTION: This email originates from outside THG
Hi Ceph users!
I've been proposed an interesting EC setup I hadn't thought about before.
Scenario is : we have two server rooms and want to store ~4PiB with the
ability to loose 1 server room without loss of data or RW availability.
For the context, performance is not needed (cold storage mostly, used as
a big filesystem).
The idea is to use EC 8+12 over 24 servers (12 on each server room), so
if we loose 1 room we still have half of the EC parts (10/20) and are
able to loose 2 more servers before reaching the point where we loose data.
I find this pretty elegant when working on a two-sites context, as
efficiency is 40% (better than 33% three times replication) and the
redundancy is good.
What do you think of this setup ? Did you ever used EC profiles with M > K ?
Thanks for sharing your thoughts!
Cheers,
Fabien
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Danny Webb
Principal OpenStack Engineer
Danny.Webb(a)thehutgroup.com
[THG Ingenuity Logo]<https://www.thg.com>
[https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc…
[https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>
![https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc…](https://i.imgur.com/wbpVRW6.png]%3Chttps://www.linkedin.com/company/thgplc/?originalSubdomain=uk%3E){kind=link}
9:24 p.m.
A custom CRUSH rule can have two steps to enforce that.
On Mar 24, 2023, at 11:04, Danny Webb
<Danny.Webb(a)thehutgroup.com> wrote:
The question I have regarding this setup is, how can you guarantee that the 12 m chunks
will be located evenly across the two rooms. What would happen if by chance all 12 chunks
were in room B? Usually you use failure domains to make sure of the distribution of
chunks across domains, but you can't do that here as you are using host as failure
domain, but need room to be somehow included in that.
________________________________
From: Fabien Sirjean <fsirjean(a)eddie.fdn.fr>
Sent: 24 March 2023 12:00
To: ceph-users <ceph-users(a)ceph.io>
Subject: [ceph-users] EC profiles where m>k (EC 8+12)
CAUTION: This email originates from outside THG
Hi Ceph users!
I've been proposed an interesting EC setup I hadn't thought about before.
Scenario is : we have two server rooms and want to store ~4PiB with the
ability to loose 1 server room without loss of data or RW availability.
For the context, performance is not needed (cold storage mostly, used as
a big filesystem).
The idea is to use EC 8+12 over 24 servers (12 on each server room), so
if we loose 1 room we still have half of the EC parts (10/20) and are
able to loose 2 more servers before reaching the point where we loose data.
I find this pretty elegant when working on a two-sites context, as
efficiency is 40% (better than 33% three times replication) and the
redundancy is good.
What do you think of this setup ? Did you ever used EC profiles with M > K ?
Thanks for sharing your thoughts!
Cheers,
Fabien
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Danny Webb
Principal OpenStack Engineer
Danny.Webb(a)thehutgroup.com
[THG Ingenuity Logo]<https://www.thg.com>
[https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc…
[https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
27 Mar
27 Mar
5:50 p.m.
Hi Fabien,
we have also used it several times for 2 DC setups.
However, we always try to use as few chunks as possible, as it is very
inefficient when storing small files (min alloc size) and it can also
lead to quite some problems with backfill and recovery in large ceph
clusters.
Joachim
___________________________________
Clyso GmbH - Ceph Foundation Member
Am 24.03.23 um 13:00 schrieb Fabien Sirjean:
> Hi Ceph users!
>
> I've been proposed an interesting EC setup I hadn't thought about before.
>
> Scenario is : we have two server rooms and want to store ~4PiB with
> the ability to loose 1 server room without loss of data or RW
> availability.
>
> For the context, performance is not needed (cold storage mostly, used
> as a big filesystem).
>
> The idea is to use EC 8+12 over 24 servers (12 on each server room),
> so if we loose 1 room we still have half of the EC parts (10/20) and
> are able to loose 2 more servers before reaching the point where we
> loose data.
>
> I find this pretty elegant when working on a two-sites context, as
> efficiency is 40% (better than 33% three times replication) and the
> redundancy is good.
>
> What do you think of this setup ? Did you ever used EC profiles with M
> > K ?
>
> Thanks for sharing your thoughts!
>
> Cheers,
>
> Fabien
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
416
days inactive
419
days old
5 comments
5 participants
participants (5)
-
Anthony D'Atri -
Clyso GmbH - Ceph Foundation Member -
Danny Webb -
Eugen Block -
Fabien Sirjean