28 May
2020
28 May
'20
12:18 p.m.
Hi, in our production cluster, we have the following setup
- 10 nodes
- 3 drives / server (so far), mix of SSD and HDD (different pools) + NVMe
- dual 10G in LACP, linked to two different switches (Cisco vPC)
- OSDs, MONs and MGRs are colocated
- A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2
- 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2)
- switches have dual PSUs and are connected to both rails
- CEPH nodes - single power supply
- Odd nodes (1,3,5...) are connected to PDU1
- Even nodes (2,4,6...) are connected to PDU2
... I can provide a drawing if it helps :)
Now, the default crush map ensures that multiple copies of the same object
won't find their way on the same host, which is fine. But I'm thinking
that in case of power failure [1] of either ATS or PDU, we'd be losing half
the nodes in the cluster at the same time. How would I go about tuning our
map so it took into account that, for a 3 copy replicated pool, we don't
have those stored on hosts, say, 5,7,9 ?
And, what about when using EC pools ? We currently have 5+2 SSD pools -
how would we avoid losing availability in case of a power loss where 50%
of the server are offline ?
I've gone over https://docs.ceph.com/docs/master/rados/operations/crush-map/
but don't believe I'm at the stage where I dare make changes without
incurring a huge data migration (probably can't be avoided).
Any input appreciated.
Cheers,
Phil
[1] both power feeds lost at the same time is really hard to protect against :)
28 May
28 May
12:42 p.m.
Immediate thought: Forget about crush maps, osds, etc. If you lose half
the nodes (when one power rail fails) your MONs will lose quorum. I
don't see how you can win with that configuration...
On 28/05/2020 13:18, Phil Regnauld wrote:
Hi, in our production cluster, we have the following
setup
- 10 nodes
- 3 drives / server (so far), mix of SSD and HDD (different pools) + NVMe
- dual 10G in LACP, linked to two different switches (Cisco vPC)
- OSDs, MONs and MGRs are colocated
- A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2
- 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2)
- switches have dual PSUs and are connected to both rails
- CEPH nodes - single power supply
- Odd nodes (1,3,5...) are connected to PDU1
- Even nodes (2,4,6...) are connected to PDU2
... I can provide a drawing if it helps :)
Now, the default crush map ensures that multiple copies of the same object
won't find their way on the same host, which is fine. But I'm thinking
that in case of power failure [1] of either ATS or PDU, we'd be losing half
the nodes in the cluster at the same time. How would I go about tuning our
map so it took into account that, for a 3 copy replicated pool, we don't
have those stored on hosts, say, 5,7,9 ?
And, what about when using EC pools ? We currently have 5+2 SSD pools -
how would we avoid losing availability in case of a power loss where 50%
of the server are offline ?
I've gone over https://docs.ceph.com/docs/master/rados/operations/crush-map/
but don't believe I'm at the stage where I dare make changes without
incurring a huge data migration (probably can't be avoided).
Any input appreciated.
Cheers,
Phil
[1] both power feeds lost at the same time is really hard to protect against :)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:47 p.m.
I would second that, there's no winning in this case for your
requirements and single PSU nodes. If there were 3 feeds, then yes; you
could make an extra layer in your crushmap much like you would
incorporate a rack topology in the crushmap.
On 5/28/20 2:42 PM, Chris Palmer wrote:
> Immediate thought: Forget about crush maps, osds, etc. If you lose
> half the nodes (when one power rail fails) your MONs will lose quorum.
> I don't see how you can win with that configuration...
>
> On 28/05/2020 13:18, Phil Regnauld wrote:
>> Hi, in our production cluster, we have the following setup
>>
>> - 10 nodes
>> - 3 drives / server (so far), mix of SSD and HDD (different
>> pools) + NVMe
>> - dual 10G in LACP, linked to two different switches (Cisco vPC)
>> - OSDs, MONs and MGRs are colocated
>>
>> - A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2
>> - 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2)
>>
>> - switches have dual PSUs and are connected to both rails
>> - CEPH nodes - single power supply
>>
>> - Odd nodes (1,3,5...) are connected to PDU1
>> - Even nodes (2,4,6...) are connected to PDU2
>>
>> ... I can provide a drawing if it helps :)
>>
>> Now, the default crush map ensures that multiple copies of the same
>> object
>> won't find their way on the same host, which is fine. But I'm thinking
>> that in case of power failure [1] of either ATS or PDU, we'd be
>> losing half
>> the nodes in the cluster at the same time. How would I go about
>> tuning our
>> map so it took into account that, for a 3 copy replicated pool, we don't
>> have those stored on hosts, say, 5,7,9 ?
>>
>> And, what about when using EC pools ? We currently have 5+2 SSD pools -
>> how would we avoid losing availability in case of a power loss where 50%
>> of the server are offline ?
>>
>> I've gone over
>> https://docs.ceph.com/docs/master/rados/operations/crush-map/
>> but don't believe I'm at the stage where I dare make changes without
>> incurring a huge data migration (probably can't be avoided).
>>
>> Any input appreciated.
>>
>> Cheers,
>> Phil
>>
>> [1] both power feeds lost at the same time is really hard to protect
>> against :)
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
29 May
29 May
7:58 a.m.
Hans van den Bogert (hansbogert) writes:
I would second that, there's no winning in this
case for your requirements
and single PSU nodes. If there were 3 feeds, then yes; you could make an
extra layer in your crushmap much like you would incorporate a rack topology
in the crushmap.
I'm not fully up on coffee for today, so I haven't yet worked out why
3 feeds would help ? To have a 'tie breaker' of sorts, with hosts spread
across 3 rails ?
10:11 a.m.
Hello!
Fri, May 29, 2020 at 09:58:58AM +0200, pr wrote:
Hans van den Bogert (hansbogert) writes:
You can break up your setup on 3 pieces, so outage of one of them will not lead
to outage of more than 1/3 of your cluster. So your cluster will survive. All of
your switches, connected to 2 PDUs with separate PSU, will survive in case of
outage of one PDU/PSU/ATS.
I would second that, there's no winning in
this case for your requirements
and single PSU nodes. If there were 3 feeds, then yes; you could make an
extra layer in your crushmap much like you would incorporate a rack topology
in the crushmap.
I'm not fully up on coffee for today, so I haven't yet worked out why
3 feeds would help ? To have a 'tie breaker' of sorts, with hosts spread
across 3 rails ? 
4:48 p.m.
Phil;
I like to refer to basic principles, and design assumptions / choices when considering
things like this. I also like to refer to more broadly understood technologies. Finally;
I'm still relatively new to Ceph, so here it goes...
TLDR: Ceph is (likes to be) double-redundent (like RAID-6), while dual power (n+1) is
single-redundant.
Like RAID, Ceph (or more precisely a Ceph pool) can be in, and moves through, the
following states:
Normal --> Partially Failed (degraded) --> Recovering --> Normal.
When talking about these systems, we often gloss over Recovery, acting as if it takes no
time. Recovery does take time though, and if anything ELSE happens while recovery is
ongoing, what can the software do?
Think RAID-5; what happens if a drive fails in a RAID-5 array, and during recovery an
unreadable block is found on another drive? That's single redundancy. If you use
RAID-6, the array goes to the second redundancy level, and the recovery continues.
As a result of the long recovery times expected of modern large hard-drives, Ceph pushes
for double-redundancy (3x replication, 5-2 EC). Further, it decreases availability the
more redundancy is degraded (i.e. when the first layer of redundancy is compromised,
writes are still allowed. When the second is lost, writes are disallowed, but reads are
allowed. Only when all three layers are compromised are reads disallowed).
Dual power feeds (n+1) is only single-redundant, thus the entire system can't achieve
better than single-redundancy. Depending on the reliability of the power, and your
service guarantees, this may be acceptable.
If you add ATSs, then you need to look at the failure rate (MTBF, or similar) to determine
if your service guarantees are impacted.
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Phil Regnauld [mailto:pr@x0.dk]
Sent: Friday, May 29, 2020 12:59 AM
To: Hans van den Bogert
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: CEPH failure domain - power considerations
Hans van den Bogert (hansbogert) writes:
I would second that, there's no winning in this
case for your
requirements and single PSU nodes. If there were 3 feeds, then yes;
you could make an extra layer in your crushmap much like you would
incorporate a rack topology in the crushmap.
I'm not fully up on coffee for today, so I haven't yet worked out why
3 feeds would help ? To have a 'tie breaker' of sorts, with hosts spread
across 3 rails ?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
5:44 p.m.
Phil, this would be an excellent contribution to the blog or the introductory
documentation. I’ve been using Ceph for over a year this brought together a lot of
concepts that I hadn’t related so succinctly in the past.
One of the things that I hadn’t really conceptualized well was “why size of 3?” I knew
that PGs went to read-only without a quorum of OSDs to write to, but this is a much
simpler way to think about it.
Something I have been experimenting with that might also be interesting to the discussion
is “when to use redundancy at all”. Kafka is a good example of "eventually
consistent" software that is designed for complete node failure and extremely high
performance. If Kafka is backed by a replicated pool, I’ve come to believe this is
suboptimal compared to having three Kafka instances, each without replication in Ceph.
The logical question is “why use Ceph at all then?” To me, this is about centralized
management process. If I am building with Ceph in most places, using it everywhere creates
operational consistency. (Modifying CRUSH maps is the path to enabling unreplicated
storage that is pinned to a specific machine that also contains the Kafka workload.)
At any rate, eventually consistent software packages can provide additional options for
top level failure domain requirements.
Brian
On May 29, 2020, at 10:48 AM,
<DHilsbos(a)performair.com> <DHilsbos(a)performair.com> wrote:
Phil;
I like to refer to basic principles, and design assumptions / choices when considering
things like this. I also like to refer to more broadly understood technologies. Finally;
I'm still relatively new to Ceph, so here it goes...
TLDR: Ceph is (likes to be) double-redundent (like RAID-6), while dual power (n+1) is
single-redundant.
Like RAID, Ceph (or more precisely a Ceph pool) can be in, and moves through, the
following states:
Normal --> Partially Failed (degraded) --> Recovering --> Normal.
When talking about these systems, we often gloss over Recovery, acting as if it takes no
time. Recovery does take time though, and if anything ELSE happens while recovery is
ongoing, what can the software do?
Think RAID-5; what happens if a drive fails in a RAID-5 array, and during recovery an
unreadable block is found on another drive? That's single redundancy. If you use
RAID-6, the array goes to the second redundancy level, and the recovery continues.
As a result of the long recovery times expected of modern large hard-drives, Ceph pushes
for double-redundancy (3x replication, 5-2 EC). Further, it decreases availability the
more redundancy is degraded (i.e. when the first layer of redundancy is compromised,
writes are still allowed. When the second is lost, writes are disallowed, but reads are
allowed. Only when all three layers are compromised are reads disallowed).
Dual power feeds (n+1) is only single-redundant, thus the entire system can't achieve
better than single-redundancy. Depending on the reliability of the power, and your
service guarantees, this may be acceptable.
If you add ATSs, then you need to look at the failure rate (MTBF, or similar) to
determine if your service guarantees are impacted.
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Phil Regnauld [mailto:pr@x0.dk]
Sent: Friday, May 29, 2020 12:59 AM
To: Hans van den Bogert
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: CEPH failure domain - power considerations
Hans van den Bogert (hansbogert) writes:
I would second that, there's no winning in
this case for your
requirements and single PSU nodes. If there were 3 feeds, then yes;
you could make an extra layer in your crushmap much like you would
incorporate a rack topology in the crushmap.
I'm not fully up on coffee for today, so I haven't yet worked out why
3 feeds would help ? To have a 'tie breaker' of sorts, with hosts spread
across 3 rails ?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:57 a.m.
Chris Palmer (chris.palmer) writes:
Immediate thought: Forget about crush maps, osds, etc.
If you lose half the
nodes (when one power rail fails) your MONs will lose quorum. I don't see
how you can win with that configuration...
That's a good point, I'll have to think that one through :) I've
experienced an APC rail failure not that long ago (on an older
one, though), so I know These Things Happen™, and trying to work
out the consequences. Lab test time :)
28 May
28 May
1:24 p.m.
Hi,
On 5/28/20 2:18 PM, Phil Regnauld wrote:
Hi, in our production cluster, we have the following
setup
*snipsnap*
Buy some power transfer switches. You can connect those to the two PDUs,
and in case of a power failure on one PDUs they will still be able to
use the second PDU.
We only use them for "small" machines, so I dunno whether yuo can get
them for higher loads.
Regards,
Burkhard
--
Dr. rer. nat. Burkhard Linke
Bioinformatics and Systems Biology
Justus-Liebig-University Giessen
35392 Giessen, Germany
Phone: (+49) (0)641 9935810
1:30 p.m.
Hi,
ATS ( Automatic Transfer Switch ) Works well.
We use in other services for single PSU server, they transfer the power from source B to
UPS in nano secs , preventing all services from going down.
You can get for 8A / 16A or 32 A , always monitorizable by SNMP , webinterface.
-----Mensaje original-----
De: Burkhard Linke <Burkhard.Linke(a)computational.bio.uni-giessen.de>
Enviado el: jueves, 28 de mayo de 2020 15:25
Para: ceph-users(a)ceph.io
Asunto: [ceph-users] Re: CEPH failure domain - power considerations
Hi,
On 5/28/20 2:18 PM, Phil Regnauld wrote:
Hi, in our production cluster, we have the following
setup
*snipsnap*
Buy some power transfer switches. You can connect those to the two PDUs, and in case of a
power failure on one PDUs they will still be able to use the second PDU.
We only use them for "small" machines, so I dunno whether yuo can get them for
higher loads.
Regards,
Burkhard
--
Dr. rer. nat. Burkhard Linke
Bioinformatics and Systems Biology
Justus-Liebig-University Giessen
35392 Giessen, Germany
Phone: (+49) (0)641 9935810
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
29 May
29 May
7:59 a.m.
Burkhard Linke (Burkhard.Linke) writes:
Buy some power transfer switches. You can connect those to the two PDUs, and
in case of a power failure on one PDUs they will still be able to use the
second PDU.
ATS = power switches (in my original mail).
We only use them for "small" machines, so I
dunno whether yuo can get them
for higher loads.
I have 2 x 32A 220v.
1435
days inactive
1436
days old
10 comments
8 participants
participants (8)
-
Brian Topping -
Burkhard Linke -
Chris Palmer -
DHilsbos＠performair.com -
EDH - Manuel Rios -
Hans van den Bogert -
Max Krasilnikov -
Phil Regnauld