Network traffic with failure domain datacenter
Hi We are looking at a cluster split between two DCs with the DCs as failure domains. Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters.
Unless I am mistaken, the first/primary of each PG is the one "doing" the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases. I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible. -- May the most significant bit of your life be positive.
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters.
Unless I am mistaken, the first/primary of each PG is the one "doing" the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases.
Are we talking about recovery, or backfill?
I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
The LRC plugin might be able to localize *recovery*, but backfill from adding OSDs, changing CRUSH rules, etc. would always have to hit both.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 07-05-2025 18:16, Anthony D'Atri wrote:
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters.
Unless I am mistaken, the first/primary of each PG is the one "doing" the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases.
Are we talking about recovery, or backfill?
I was asking about both
I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
The LRC plugin might be able to localize *recovery*, but backfill from adding OSDs, changing CRUSH rules, etc. would always have to hit both.
Thanks both, I'll take a look at the LRC plugin. Mvh. Torkil
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
Hi Torkil, I would be interested in your results. It's been a few years since I saw the LRC plugin in production. Does anyone else in the ceph community use it? Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mi., 7. Mai 2025 um 21:00 Uhr schrieb Torkil Svensgaard <torkil@drcmr.dk
:
On 07-05-2025 18:16, Anthony D'Atri wrote:
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com>
wrote:
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place
should
largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters.
Unless I am mistaken, the first/primary of each PG is the one "doing" the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases.
Are we talking about recovery, or backfill?
I was asking about both
I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
The LRC plugin might be able to localize *recovery*, but backfill from adding OSDs, changing CRUSH rules, etc. would always have to hit both.
Thanks both, I'll take a look at the LRC plugin.
Mvh.
Torkil
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, two years ago I created this tracker: https://tracker.ceph.com/issues/61861 There was also a thread here on this list: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/RLTF4NTN5KG... We did consider it during a proof of concept once, but I had the impression that the plugin did not work as expected, and so did OP from the mentioned thread. So if you asked me, I wouldn't recommend it. Regards, Eugen Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Torkil, I would be interested in your results. It's been a few years since I saw the LRC plugin in production. Does anyone else in the ceph community use it? Joachim
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677
Am Mi., 7. Mai 2025 um 21:00 Uhr schrieb Torkil Svensgaard <torkil@drcmr.dk
:
On 07-05-2025 18:16, Anthony D'Atri wrote:
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com>
wrote:
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place
should
largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters.
Unless I am mistaken, the first/primary of each PG is the one "doing" the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases.
Are we talking about recovery, or backfill?
I was asking about both
I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
The LRC plugin might be able to localize *recovery*, but backfill from adding OSDs, changing CRUSH rules, etc. would always have to hit both.
Thanks both, I'll take a look at the LRC plugin.
Mvh.
Torkil
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I agree, I have never seen someone using it.
On May 8, 2025, at 2:58 AM, Eugen Block <eblock@nde.ag> wrote:
Hi,
two years ago I created this tracker:
https://tracker.ceph.com/issues/61861
There was also a thread here on this list:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/RLTF4NTN5KG...
We did consider it during a proof of concept once, but I had the impression that the plugin did not work as expected, and so did OP from the mentioned thread. So if you asked me, I wouldn't recommend it.
Regards, Eugen
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Torkil, I would be interested in your results. It's been a few years since I saw the LRC plugin in production. Does anyone else in the ceph community use it? Joachim
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677
Am Mi., 7. Mai 2025 um 21:00 Uhr schrieb Torkil Svensgaard <torkil@drcmr.dk
:
On 07-05-2025 18:16, Anthony D'Atri wrote:
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters.
Unless I am mistaken, the first/primary of each PG is the one "doing" the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases.
Are we talking about recovery, or backfill?
I was asking about both
I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
The LRC plugin might be able to localize *recovery*, but backfill from adding OSDs, changing CRUSH rules, etc. would always have to hit both.
Thanks both, I'll take a look at the LRC plugin.
Mvh.
Torkil
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
To be pedantic … backfill usually means copying data in toto, so like normal write replication it necessarily has to traverse the WAN. Recovery of just a lost shard/replica in theory with the LRC plugin, but as noted that doesn’t seem like a good choice. With the default EC plugin, there *may* be some read locality preference but it’s not something I would bank on. Stretch clusters are great when you need zero RPO when you really need a single cluster and can manage client endpoint use accordingly. But with tradeoffs, in many cases two clusters with async replication can be a better solution, depends on needs and what you’re solving for.
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters.
Unless I am mistaken, the first/primary of each PG is the one "doing" the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases. I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
-- May the most significant bit of your life be positive.
It’s good to be 8-bit-clean, if you aren’t , then Kermit can compensate.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There is also the issue that if you have a 4+8 EC pool, you ideally need at least 4+8 of whatever your failure domain is, in this case DCs. This is more than most people have. Is this k=4, m=8? What is the benefit of this compared to an ordinary replicated pool with 3 copies? Even if you set the failure domain to, say rack, there is no guarantee that there is no PG with more than 8 parts in a single DC without some crushmap trickery. If this is k=8, m=4, then only 4 failures can be handled and there is no way to split 12 parts so that both DCs contain 4 or less at the same time. You really need 3 DCs and a fast, highly available network in between. /Peter Den 2025-05-08 kl. 17:45, skrev Anthony D'Atri:
To be pedantic … backfill usually means copying data in toto, so like normal write replication it necessarily has to traverse the WAN.
Recovery of just a lost shard/replica in theory with the LRC plugin, but as noted that doesn’t seem like a good choice. With the default EC plugin, there *may* be some read locality preference but it’s not something I would bank on.
Stretch clusters are great when you need zero RPO when you really need a single cluster and can manage client endpoint use accordingly. But with tradeoffs, in many cases two clusters with async replication can be a better solution, depends on needs and what you’re solving for.
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters. Unless I am mistaken, the first/primary of each PG is the one "doing"
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>: the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases. I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
-- May the most significant bit of your life be positive. It’s good to be 8-bit-clean, if you aren’t , then Kermit can compensate.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On May 8, 2025, at 3:14 PM, Peter Linder <peter.linder@fiberdirekt.se> wrote:
There is also the issue that if you have a 4+8 EC pool, you ideally need at least 4+8 of whatever your failure domain is, in this case DCs.
Well, in terms of a formal stretch cluster, EC isn’t actually supported today, though if those DCs are VERY close to each other with small RTT one could fake it with something like 4+8, but without the mon quorum and min_size advantages of stretch mode. With two DCS, formal stretch mode needs R4 pools.
This is more than most people have.
Is this k=4, m=8? What is the benefit of this compared to an ordinary replicated pool with 3 copies?
Even if you set the failure domain to, say rack, there is no guarantee that there is no PG with more than 8 parts in a single DC without some crushmap trickery.
If this is k=8, m=4, then only 4 failures can be handled and there is no way to split 12 parts so that both DCs contain 4 or less at the same time.
You really need 3 DCs and a fast, highly available network in between.
With stretch typical 2 DCs and a tiebreaker mon elsewhere, which can have higher latency and even be a cloud VM.
/Peter
Den 2025-05-08 kl. 17:45, skrev Anthony D'Atri:
To be pedantic … backfill usually means copying data in toto, so like normal write replication it necessarily has to traverse the WAN.
Recovery of just a lost shard/replica in theory with the LRC plugin, but as noted that doesn’t seem like a good choice. With the default EC plugin, there *may* be some read locality preference but it’s not something I would bank on.
Stretch clusters are great when you need zero RPO when you really need a single cluster and can manage client endpoint use accordingly. But with tradeoffs, in many cases two clusters with async replication can be a better solution, depends on needs and what you’re solving for.
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters. Unless I am mistaken, the first/primary of each PG is the one "doing"
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>: the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases. I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
-- May the most significant bit of your life be positive. It’s good to be 8-bit-clean, if you aren’t , then Kermit can compensate.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 08-05-2025 21:14, Peter Linder wrote:
There is also the issue that if you have a 4+8 EC pool, you ideally need at least 4+8 of whatever your failure domain is, in this case DCs. This is more than most people have.
Is this k=4, m=8? What is the benefit of this compared to an ordinary replicated pool with 3 copies?
My bad, I think I've misunderstood the definition of a failure domain, it would actually be host. We are going to have 2 DCs each with 7+ hosts, and a tiebreaker MON in a third DC. The should allow us to lose one DC and an additional host and still be online.
Even if you set the failure domain to, say rack, there is no guarantee that there is no PG with more than 8 parts in a single DC without some crushmap trickery.
We would use crush to ensure the placement we want, something like this: rule EC_4_8 { id ZYX type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class nvmebulk step choose indep 0 type datacenter step chooseleaf indep 6 type host step emit }
If this is k=8, m=4, then only 4 failures can be handled and there is no way to split 12 parts so that both DCs contain 4 or less at the same time.
You really need 3 DCs and a fast, highly available network in between.
/Peter
Den 2025-05-08 kl. 17:45, skrev Anthony D'Atri:
To be pedantic … backfill usually means copying data in toto, so like normal write replication it necessarily has to traverse the WAN.
Recovery of just a lost shard/replica in theory with the LRC plugin, but as noted that doesn’t seem like a good choice. With the default EC plugin, there *may* be some read locality preference but it’s not something I would bank on.
We looked at the LRC plugin and we don't think it would be worth the risk going with that since it seems somewhat abandoned and not really used by anyone.
Stretch clusters are great when you need zero RPO when you really need a single cluster and can manage client endpoint use accordingly. But with tradeoffs, in many cases two clusters with async replication can be a better solution, depends on needs and what you’re solving for.
We did consider two clusters + replication but then we would need more hardware to get the same usable space, and money is scarce. The WAN would probably be 2x10G and at a distance of less than 10km. The pools would mainly be bulk storage so I think that should work ok. Thanks all. Mvh. Torkil
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters. Unless I am mistaken, the first/primary of each PG is the one "doing"
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>: the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases. I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
-- May the most significant bit of your life be positive. It’s good to be 8-bit-clean, if you aren’t , then Kermit can compensate.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
Maybe a case for a new-style MSR rule?
On May 9, 2025, at 5:13 PM, Torkil Svensgaard <torkil@drcmr.dk> wrote:
On 08-05-2025 21:14, Peter Linder wrote: There is also the issue that if you have a 4+8 EC pool, you ideally need at least 4+8 of whatever your failure domain is, in this case DCs. This is more than most people have. Is this k=4, m=8? What is the benefit of this compared to an ordinary replicated pool with 3 copies?
My bad, I think I've misunderstood the definition of a failure domain, it would actually be host.
We are going to have 2 DCs each with 7+ hosts, and a tiebreaker MON in a third DC. The should allow us to lose one DC and an additional host and still be online.
Even if you set the failure domain to, say rack, there is no guarantee that there is no PG with more than 8 parts in a single DC without some crushmap trickery.
We would use crush to ensure the placement we want, something like this:
rule EC_4_8 { id ZYX type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class nvmebulk step choose indep 0 type datacenter step chooseleaf indep 6 type host step emit }
If this is k=8, m=4, then only 4 failures can be handled and there is no way to split 12 parts so that both DCs contain 4 or less at the same time. You really need 3 DCs and a fast, highly available network in between. /Peter
Den 2025-05-08 kl. 17:45, skrev Anthony D'Atri: To be pedantic … backfill usually means copying data in toto, so like normal write replication it necessarily has to traverse the WAN.
Recovery of just a lost shard/replica in theory with the LRC plugin, but as noted that doesn’t seem like a good choice. With the default EC plugin, there *may* be some read locality preference but it’s not something I would bank on.
We looked at the LRC plugin and we don't think it would be worth the risk going with that since it seems somewhat abandoned and not really used by anyone.
Stretch clusters are great when you need zero RPO when you really need a single cluster and can manage client endpoint use accordingly. But with tradeoffs, in many cases two clusters with async replication can be a better solution, depends on needs and what you’re solving for.
We did consider two clusters + replication but then we would need more hardware to get the same usable space, and money is scarce.
The WAN would probably be 2x10G and at a distance of less than 10km. The pools would mainly be bulk storage so I think that should work ok.
Thanks all.
Mvh.
Torkil
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters. Unless I am mistaken, the first/primary of each PG is the one "doing"
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>: the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases. I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
-- May the most significant bit of your life be positive. It’s good to be 8-bit-clean, if you aren’t , then Kermit can compensate.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 10-05-2025 00:25, Anthony D'Atri wrote:
Maybe a case for a new-style MSR rule?
Interesting, I wasn't aware of that new feature. We are on squid but we do have some older clients we need to to something about first. Thanks. Mvh. Torkil
On May 9, 2025, at 5:13 PM, Torkil Svensgaard <torkil@drcmr.dk> wrote:
On 08-05-2025 21:14, Peter Linder wrote: There is also the issue that if you have a 4+8 EC pool, you ideally need at least 4+8 of whatever your failure domain is, in this case DCs. This is more than most people have. Is this k=4, m=8? What is the benefit of this compared to an ordinary replicated pool with 3 copies?
My bad, I think I've misunderstood the definition of a failure domain, it would actually be host.
We are going to have 2 DCs each with 7+ hosts, and a tiebreaker MON in a third DC. The should allow us to lose one DC and an additional host and still be online.
Even if you set the failure domain to, say rack, there is no guarantee that there is no PG with more than 8 parts in a single DC without some crushmap trickery.
We would use crush to ensure the placement we want, something like this:
rule EC_4_8 { id ZYX type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class nvmebulk step choose indep 0 type datacenter step chooseleaf indep 6 type host step emit }
If this is k=8, m=4, then only 4 failures can be handled and there is no way to split 12 parts so that both DCs contain 4 or less at the same time. You really need 3 DCs and a fast, highly available network in between. /Peter
Den 2025-05-08 kl. 17:45, skrev Anthony D'Atri: To be pedantic … backfill usually means copying data in toto, so like normal write replication it necessarily has to traverse the WAN.
Recovery of just a lost shard/replica in theory with the LRC plugin, but as noted that doesn’t seem like a good choice. With the default EC plugin, there *may* be some read locality preference but it’s not something I would bank on.
We looked at the LRC plugin and we don't think it would be worth the risk going with that since it seems somewhat abandoned and not really used by anyone.
Stretch clusters are great when you need zero RPO when you really need a single cluster and can manage client endpoint use accordingly. But with tradeoffs, in many cases two clusters with async replication can be a better solution, depends on needs and what you’re solving for.
We did consider two clusters + replication but then we would need more hardware to get the same usable space, and money is scarce.
The WAN would probably be 2x10G and at a distance of less than 10km. The pools would mainly be bulk storage so I think that should work ok.
Thanks all.
Mvh.
Torkil
On May 7, 2025, at 5:06 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made? Pools would be EC 4+8, if that matters. Unless I am mistaken, the first/primary of each PG is the one "doing"
Den ons 7 maj 2025 kl 10:59 skrev Torkil Svensgaard <torkil@drcmr.dk>: the backfills, so if the primaries are evenly distributed between the sites, the source of all backfills would be in the remote DC in 50% of the cases. I do not think the backfills are going to calculate how it can use only "local" pieces to rebuild a missing/degraded PG piece without going over the DC-DC link even if it is theoretically possible.
-- May the most significant bit of your life be positive. It’s good to be 8-bit-clean, if you aren’t , then Kermit can compensate.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
Since the idea is for DR, you would *want* each DC to have current data at all times, no?
On May 7, 2025, at 4:40 AM, Torkil Svensgaard <torkil@drcmr.dk> wrote:
Hi
We are looking at a cluster split between two DCs with the DCs as failure domains.
Am I right in assuming that any recovery or backfill taking place should largely happen inside each DC and not between them? Or can no such assumptions be made?
Pools would be EC 4+8, if that matters.
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Anthony D'Atri
-
Anthony D'Atri
-
Eugen Block
-
Janne Johansson
-
Joachim Kraftmayer
-
Peter Linder
-
Torkil Svensgaard