CEPH failure domain - power considerations
Hi, in our production cluster, we have the following setup - 10 nodes - 3 drives / server (so far), mix of SSD and HDD (different pools) + NVMe - dual 10G in LACP, linked to two different switches (Cisco vPC) - OSDs, MONs and MGRs are colocated - A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2 - 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2) - switches have dual PSUs and are connected to both rails - CEPH nodes - single power supply - Odd nodes (1,3,5...) are connected to PDU1 - Even nodes (2,4,6...) are connected to PDU2 ... I can provide a drawing if it helps :) Now, the default crush map ensures that multiple copies of the same object won't find their way on the same host, which is fine. But I'm thinking that in case of power failure [1] of either ATS or PDU, we'd be losing half the nodes in the cluster at the same time. How would I go about tuning our map so it took into account that, for a 3 copy replicated pool, we don't have those stored on hosts, say, 5,7,9 ? And, what about when using EC pools ? We currently have 5+2 SSD pools - how would we avoid losing availability in case of a power loss where 50% of the server are offline ? I've gone over https://docs.ceph.com/docs/master/rados/operations/crush-map/ but don't believe I'm at the stage where I dare make changes without incurring a huge data migration (probably can't be avoided). Any input appreciated. Cheers, Phil [1] both power feeds lost at the same time is really hard to protect against :)
Immediate thought: Forget about crush maps, osds, etc. If you lose half the nodes (when one power rail fails) your MONs will lose quorum. I don't see how you can win with that configuration... On 28/05/2020 13:18, Phil Regnauld wrote:
Hi, in our production cluster, we have the following setup
- 10 nodes - 3 drives / server (so far), mix of SSD and HDD (different pools) + NVMe - dual 10G in LACP, linked to two different switches (Cisco vPC) - OSDs, MONs and MGRs are colocated
- A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2 - 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2)
- switches have dual PSUs and are connected to both rails - CEPH nodes - single power supply
- Odd nodes (1,3,5...) are connected to PDU1 - Even nodes (2,4,6...) are connected to PDU2
... I can provide a drawing if it helps :)
Now, the default crush map ensures that multiple copies of the same object won't find their way on the same host, which is fine. But I'm thinking that in case of power failure [1] of either ATS or PDU, we'd be losing half the nodes in the cluster at the same time. How would I go about tuning our map so it took into account that, for a 3 copy replicated pool, we don't have those stored on hosts, say, 5,7,9 ?
And, what about when using EC pools ? We currently have 5+2 SSD pools - how would we avoid losing availability in case of a power loss where 50% of the server are offline ?
I've gone over https://docs.ceph.com/docs/master/rados/operations/crush-map/ but don't believe I'm at the stage where I dare make changes without incurring a huge data migration (probably can't be avoided).
Any input appreciated.
Cheers, Phil
[1] both power feeds lost at the same time is really hard to protect against :) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I would second that, there's no winning in this case for your requirements and single PSU nodes. If there were 3 feeds, then yes; you could make an extra layer in your crushmap much like you would incorporate a rack topology in the crushmap. On 5/28/20 2:42 PM, Chris Palmer wrote:
Immediate thought: Forget about crush maps, osds, etc. If you lose half the nodes (when one power rail fails) your MONs will lose quorum. I don't see how you can win with that configuration...
On 28/05/2020 13:18, Phil Regnauld wrote:
Hi, in our production cluster, we have the following setup
- 10 nodes - 3 drives / server (so far), mix of SSD and HDD (different pools) + NVMe - dual 10G in LACP, linked to two different switches (Cisco vPC) - OSDs, MONs and MGRs are colocated
- A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2 - 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2)
- switches have dual PSUs and are connected to both rails - CEPH nodes - single power supply
- Odd nodes (1,3,5...) are connected to PDU1 - Even nodes (2,4,6...) are connected to PDU2
... I can provide a drawing if it helps :)
Now, the default crush map ensures that multiple copies of the same object won't find their way on the same host, which is fine. But I'm thinking that in case of power failure [1] of either ATS or PDU, we'd be losing half the nodes in the cluster at the same time. How would I go about tuning our map so it took into account that, for a 3 copy replicated pool, we don't have those stored on hosts, say, 5,7,9 ?
And, what about when using EC pools ? We currently have 5+2 SSD pools - how would we avoid losing availability in case of a power loss where 50% of the server are offline ?
I've gone over https://docs.ceph.com/docs/master/rados/operations/crush-map/ but don't believe I'm at the stage where I dare make changes without incurring a huge data migration (probably can't be avoided).
Any input appreciated.
Cheers, Phil
[1] both power feeds lost at the same time is really hard to protect against :) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hans van den Bogert (hansbogert) writes:
I would second that, there's no winning in this case for your requirements and single PSU nodes. If there were 3 feeds, then yes; you could make an extra layer in your crushmap much like you would incorporate a rack topology in the crushmap.
I'm not fully up on coffee for today, so I haven't yet worked out why 3 feeds would help ? To have a 'tie breaker' of sorts, with hosts spread across 3 rails ?
Hello! Fri, May 29, 2020 at 09:58:58AM +0200, pr wrote:
Hans van den Bogert (hansbogert) writes:
I would second that, there's no winning in this case for your requirements and single PSU nodes. If there were 3 feeds, then yes; you could make an extra layer in your crushmap much like you would incorporate a rack topology in the crushmap.
I'm not fully up on coffee for today, so I haven't yet worked out why 3 feeds would help ? To have a 'tie breaker' of sorts, with hosts spread across 3 rails ?
You can break up your setup on 3 pieces, so outage of one of them will not lead to outage of more than 1/3 of your cluster. So your cluster will survive. All of your switches, connected to 2 PDUs with separate PSU, will survive in case of outage of one PDU/PSU/ATS.
Phil; I like to refer to basic principles, and design assumptions / choices when considering things like this. I also like to refer to more broadly understood technologies. Finally; I'm still relatively new to Ceph, so here it goes... TLDR: Ceph is (likes to be) double-redundent (like RAID-6), while dual power (n+1) is single-redundant. Like RAID, Ceph (or more precisely a Ceph pool) can be in, and moves through, the following states: Normal --> Partially Failed (degraded) --> Recovering --> Normal. When talking about these systems, we often gloss over Recovery, acting as if it takes no time. Recovery does take time though, and if anything ELSE happens while recovery is ongoing, what can the software do? Think RAID-5; what happens if a drive fails in a RAID-5 array, and during recovery an unreadable block is found on another drive? That's single redundancy. If you use RAID-6, the array goes to the second redundancy level, and the recovery continues. As a result of the long recovery times expected of modern large hard-drives, Ceph pushes for double-redundancy (3x replication, 5-2 EC). Further, it decreases availability the more redundancy is degraded (i.e. when the first layer of redundancy is compromised, writes are still allowed. When the second is lost, writes are disallowed, but reads are allowed. Only when all three layers are compromised are reads disallowed). Dual power feeds (n+1) is only single-redundant, thus the entire system can't achieve better than single-redundancy. Depending on the reliability of the power, and your service guarantees, this may be acceptable. If you add ATSs, then you need to look at the failure rate (MTBF, or similar) to determine if your service guarantees are impacted. Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Phil Regnauld [mailto:pr@x0.dk] Sent: Friday, May 29, 2020 12:59 AM To: Hans van den Bogert Cc: ceph-users@ceph.io Subject: [ceph-users] Re: CEPH failure domain - power considerations Hans van den Bogert (hansbogert) writes:
I would second that, there's no winning in this case for your requirements and single PSU nodes. If there were 3 feeds, then yes; you could make an extra layer in your crushmap much like you would incorporate a rack topology in the crushmap.
I'm not fully up on coffee for today, so I haven't yet worked out why 3 feeds would help ? To have a 'tie breaker' of sorts, with hosts spread across 3 rails ? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Phil, this would be an excellent contribution to the blog or the introductory documentation. I’ve been using Ceph for over a year this brought together a lot of concepts that I hadn’t related so succinctly in the past. One of the things that I hadn’t really conceptualized well was “why size of 3?” I knew that PGs went to read-only without a quorum of OSDs to write to, but this is a much simpler way to think about it. Something I have been experimenting with that might also be interesting to the discussion is “when to use redundancy at all”. Kafka is a good example of "eventually consistent" software that is designed for complete node failure and extremely high performance. If Kafka is backed by a replicated pool, I’ve come to believe this is suboptimal compared to having three Kafka instances, each without replication in Ceph. The logical question is “why use Ceph at all then?” To me, this is about centralized management process. If I am building with Ceph in most places, using it everywhere creates operational consistency. (Modifying CRUSH maps is the path to enabling unreplicated storage that is pinned to a specific machine that also contains the Kafka workload.) At any rate, eventually consistent software packages can provide additional options for top level failure domain requirements. Brian
On May 29, 2020, at 10:48 AM, <DHilsbos@performair.com> <DHilsbos@performair.com> wrote:
Phil;
I like to refer to basic principles, and design assumptions / choices when considering things like this. I also like to refer to more broadly understood technologies. Finally; I'm still relatively new to Ceph, so here it goes...
TLDR: Ceph is (likes to be) double-redundent (like RAID-6), while dual power (n+1) is single-redundant.
Like RAID, Ceph (or more precisely a Ceph pool) can be in, and moves through, the following states:
Normal --> Partially Failed (degraded) --> Recovering --> Normal.
When talking about these systems, we often gloss over Recovery, acting as if it takes no time. Recovery does take time though, and if anything ELSE happens while recovery is ongoing, what can the software do?
Think RAID-5; what happens if a drive fails in a RAID-5 array, and during recovery an unreadable block is found on another drive? That's single redundancy. If you use RAID-6, the array goes to the second redundancy level, and the recovery continues.
As a result of the long recovery times expected of modern large hard-drives, Ceph pushes for double-redundancy (3x replication, 5-2 EC). Further, it decreases availability the more redundancy is degraded (i.e. when the first layer of redundancy is compromised, writes are still allowed. When the second is lost, writes are disallowed, but reads are allowed. Only when all three layers are compromised are reads disallowed).
Dual power feeds (n+1) is only single-redundant, thus the entire system can't achieve better than single-redundancy. Depending on the reliability of the power, and your service guarantees, this may be acceptable.
If you add ATSs, then you need to look at the failure rate (MTBF, or similar) to determine if your service guarantees are impacted.
Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Phil Regnauld [mailto:pr@x0.dk] Sent: Friday, May 29, 2020 12:59 AM To: Hans van den Bogert Cc: ceph-users@ceph.io Subject: [ceph-users] Re: CEPH failure domain - power considerations
Hans van den Bogert (hansbogert) writes:
I would second that, there's no winning in this case for your requirements and single PSU nodes. If there were 3 feeds, then yes; you could make an extra layer in your crushmap much like you would incorporate a rack topology in the crushmap.
I'm not fully up on coffee for today, so I haven't yet worked out why 3 feeds would help ? To have a 'tie breaker' of sorts, with hosts spread across 3 rails ? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Chris Palmer (chris.palmer) writes:
Immediate thought: Forget about crush maps, osds, etc. If you lose half the nodes (when one power rail fails) your MONs will lose quorum. I don't see how you can win with that configuration...
That's a good point, I'll have to think that one through :) I've experienced an APC rail failure not that long ago (on an older one, though), so I know These Things Happen™, and trying to work out the consequences. Lab test time :)
Hi, On 5/28/20 2:18 PM, Phil Regnauld wrote:
Hi, in our production cluster, we have the following setup
*snipsnap* Buy some power transfer switches. You can connect those to the two PDUs, and in case of a power failure on one PDUs they will still be able to use the second PDU. We only use them for "small" machines, so I dunno whether yuo can get them for higher loads. Regards, Burkhard -- Dr. rer. nat. Burkhard Linke Bioinformatics and Systems Biology Justus-Liebig-University Giessen 35392 Giessen, Germany Phone: (+49) (0)641 9935810
Hi, ATS ( Automatic Transfer Switch ) Works well. We use in other services for single PSU server, they transfer the power from source B to UPS in nano secs , preventing all services from going down. You can get for 8A / 16A or 32 A , always monitorizable by SNMP , webinterface. -----Mensaje original----- De: Burkhard Linke <Burkhard.Linke@computational.bio.uni-giessen.de> Enviado el: jueves, 28 de mayo de 2020 15:25 Para: ceph-users@ceph.io Asunto: [ceph-users] Re: CEPH failure domain - power considerations Hi, On 5/28/20 2:18 PM, Phil Regnauld wrote:
Hi, in our production cluster, we have the following setup
*snipsnap* Buy some power transfer switches. You can connect those to the two PDUs, and in case of a power failure on one PDUs they will still be able to use the second PDU. We only use them for "small" machines, so I dunno whether yuo can get them for higher loads. Regards, Burkhard -- Dr. rer. nat. Burkhard Linke Bioinformatics and Systems Biology Justus-Liebig-University Giessen 35392 Giessen, Germany Phone: (+49) (0)641 9935810 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Burkhard Linke (Burkhard.Linke) writes:
Buy some power transfer switches. You can connect those to the two PDUs, and in case of a power failure on one PDUs they will still be able to use the second PDU.
ATS = power switches (in my original mail).
We only use them for "small" machines, so I dunno whether yuo can get them for higher loads.
I have 2 x 32A 220v.
participants (8)
-
Brian Topping
-
Burkhard Linke
-
Chris Palmer
-
DHilsbos＠performair.com
-
EDH - Manuel Rios
-
Hans van den Bogert
-
Max Krasilnikov
-
Phil Regnauld