Stretched pool or not ?
Hi, i am working on the plan for a 3 datacenter setup using ceph (in proxmox nodes). Each datacenter has 3 physical nodes to start with and 100Gbit switches. I will also have 2 x 100 Gbit/s connectivity between the datacenters (each datacenter to each other). the physical nodes have 2 x 100Gbit/s for the public network and 2 x 100Gbit/s for the cluster network. About this setup i have 2 questions. 1. is it even necessary to evaluate a stretched cluster since the WAN connections are as fast as the local ones (including the latency, since it is only 25km) ? 2. If using a stretched pool across all 3 datacenters, what happens if one datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ? If i am on the wrong path, maybe someone has a link for me, where is can find information on this setup ? Cheers Soeren
Please make list posts in plain text.
i am working on the plan for a 3 datacenter setup using ceph (in proxmox nodes).
Each datacenter has 3 physical nodes to start with and 100Gbit switches. I will also have 2 x 100 Gbit/s connectivity between the datacenters (each datacenter to each other). the physical nodes have 2 x 100Gbit/s for the public network and 2 x 100Gbit/s for the cluster network.
You almost certainly don’t need a cluster / replication network unless these are exceptionally large nodes.
About this setup i have 2 questions.
is it even necessary to evaluate a stretched cluster since the WAN connections are as fast as the local ones (including the latency, since it is only 25km) ?
There’s more to latency than just distance. What is the measured latency? A:B, B:C, C:A?
If using a stretched pool across all 3 datacenters, what happens if one datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ?
That sentence refers to a non-stretch cluster. Tell us why you’re spreading across three DCs, what you’re trying to accomplish, and what your performance requirements are. AIUI a stretch 3-site cluster requires all pools to be replicated, size=6. Explicit stretch mode treats the mon quorum in a different way. With two OSD sites you deploy a tiebreaker at a third site, which is possibly just a cloud VM. With three OSD sites, I might speculate that one would deploy 7 mons, 2 At each OSD site + tiebreaker. Operations on a stretch cluster can be slow. Sometimes separate clusters with asynchronous replication make more sense.
If i am on the wrong path, maybe someone has a link for me, where is can find information on this setup ?
Cheers Soeren
Hi Soeren. First, I would like to clarify something. There are two options: *stretched cluster* and *stretch mode.* Sometimes this cannot be relied upon. If you have a “stretched-cluster” deployment in which much of your cluster is behind a single network component, you might need to use stretch mode to ensure data integrity. source: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#id1 The focus in this sentence is on ‘single network component’. I hope you don't have a single point of failure in your setup. Which option is best for your requirements? Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mo., 28. Apr. 2025 um 02:50 Uhr schrieb Anthony D'Atri < anthony.datri@gmail.com>:
Please make list posts in plain text.
i am working on the plan for a 3 datacenter setup using ceph (in proxmox nodes).
Each datacenter has 3 physical nodes to start with and 100Gbit switches. I will also have 2 x 100 Gbit/s connectivity between the datacenters (each datacenter to each other). the physical nodes have 2 x 100Gbit/s for the public network and 2 x 100Gbit/s for the cluster network.
You almost certainly don’t need a cluster / replication network unless these are exceptionally large nodes.
About this setup i have 2 questions.
is it even necessary to evaluate a stretched cluster since the WAN
connections are as fast as the local ones (including the latency, since it is only 25km) ?
There’s more to latency than just distance. What is the measured latency? A:B, B:C, C:A?
If using a stretched pool across all 3 datacenters, what happens if one
datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ?
That sentence refers to a non-stretch cluster.
Tell us why you’re spreading across three DCs, what you’re trying to accomplish, and what your performance requirements are.
AIUI a stretch 3-site cluster requires all pools to be replicated, size=6.
Explicit stretch mode treats the mon quorum in a different way. With two OSD sites you deploy a tiebreaker at a third site, which is possibly just a cloud VM. With three OSD sites, I might speculate that one would deploy 7 mons, 2 At each OSD site + tiebreaker.
Operations on a stretch cluster can be slow. Sometimes separate clusters with asynchronous replication make more sense.
If i am on the wrong path, maybe someone has a link for me, where is can
find information on this setup ?
Cheers Soeren
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
HI Joachim and Anthony, first: thanks for taking the time to answer. (now in plain text, sorry, i did not think about that) the sentence i am referring to is for "stretched pools" without a tiebreaker which is "stretch mode" if i understood the documentation correctly. I read this in the "Limitations" section exactly on the page your link refers to as well. The reason behind having 3 datacenters is because we are having alot of k8s clusters which also need to have quorum, if i distribute the etcd nodes across 3 datacenters, the outage of one datacenter will keep the k8s cluster operational. Thats why i was explicitly referring to stretched pools, not stretch mode (still hope i understand everything right). We do not have a single point of failure in the setup, all connections and devices are redundant. The latency between the datacenters is most likely very low (we can not measure since i am in planning stages. The connections between the datacenters are on dark fibres connected through modules directly in the Top of the Rack switches, compared to the local connectivity it will be almost the same. We have an existing similar setup between 2 datacenters where the WAN connection add below 1ms latency. On "exceptionally large nodes", those are all identical servers, 3 per datacenter with 16 x 3.84 TB nvme disks, 128 AMD Epyc cores (on 2 sockets) and 1.5 TB memory, i would not count them as "exceptionally large". I will read up a little more on asych replication. Cheers Soeren ________________________________________ From: Joachim Kraftmayer <joachim.kraftmayer@clyso.com> Sent: Monday, April 28, 2025 8:23 AM To: Anthony D'Atri <anthony.datri@gmail.com> Cc: Soeren Malchow <soeren.malchow@convotis.com>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Stretched pool or not ? Hi Soeren. First, I would like to clarify something. There are two options: stretched cluster and stretch mode. Sometimes this cannot be relied upon. If you have a “stretched-cluster” deployment in which much of your cluster is behind a single network component, you might need to use stretch mode to ensure data integrity. source: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#id1 The focus in this sentence is on ‘single network component’. I hope you don't have a single point of failure in your setup. Which option is best for your requirements? Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mo., 28. Apr. 2025 um 02:50 Uhr schrieb Anthony D'Atri <anthony.datri@gmail.com>: Please make list posts in plain text.
i am working on the plan for a 3 datacenter setup using ceph (in proxmox nodes).
Each datacenter has 3 physical nodes to start with and 100Gbit switches. I will also have 2 x 100 Gbit/s connectivity between the datacenters (each datacenter to each other). the physical nodes have 2 x 100Gbit/s for the public network and 2 x 100Gbit/s for the cluster network.
You almost certainly don’t need a cluster / replication network unless these are exceptionally large nodes.
About this setup i have 2 questions.
is it even necessary to evaluate a stretched cluster since the WAN connections are as fast as the local ones (including the latency, since it is only 25km) ?
There’s more to latency than just distance. What is the measured latency? A:B, B:C, C:A?
If using a stretched pool across all 3 datacenters, what happens if one datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ?
That sentence refers to a non-stretch cluster. Tell us why you’re spreading across three DCs, what you’re trying to accomplish, and what your performance requirements are. AIUI a stretch 3-site cluster requires all pools to be replicated, size=6. Explicit stretch mode treats the mon quorum in a different way. With two OSD sites you deploy a tiebreaker at a third site, which is possibly just a cloud VM. With three OSD sites, I might speculate that one would deploy 7 mons, 2 At each OSD site + tiebreaker. Operations on a stretch cluster can be slow. Sometimes separate clusters with asynchronous replication make more sense.
If i am on the wrong path, maybe someone has a link for me, where is can find information on this setup ?
Cheers Soeren
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Soeren, understood. So stretched pools need also a stretched ceph cluster. So a simple setup would be with replication size 3 for replicated pools and 3 or more ceph monitors, ... The failure domain is the datacenter. That means if you lose one DC the ceph cluster is still online. back to your original question:
If using a stretched pool across all 3 datacenters, what happens if one datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ?
If you have changed your failover domain to ‘Datacenter’, at least two DCs must be available to handle I/O operations. Do you want to use Ceph for the etcd database? Hope it helps, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mo., 28. Apr. 2025 um 12:15 Uhr schrieb Soeren Malchow < soeren.malchow@convotis.com>:
HI Joachim and Anthony,
first: thanks for taking the time to answer. (now in plain text, sorry, i did not think about that)
the sentence i am referring to is for "stretched pools" without a tiebreaker which is "stretch mode" if i understood the documentation correctly. I read this in the "Limitations" section exactly on the page your link refers to as well.
The reason behind having 3 datacenters is because we are having alot of k8s clusters which also need to have quorum, if i distribute the etcd nodes across 3 datacenters, the outage of one datacenter will keep the k8s cluster operational. Thats why i was explicitly referring to stretched pools, not stretch mode (still hope i understand everything right).
We do not have a single point of failure in the setup, all connections and devices are redundant. The latency between the datacenters is most likely very low (we can not measure since i am in planning stages. The connections between the datacenters are on dark fibres connected through modules directly in the Top of the Rack switches, compared to the local connectivity it will be almost the same. We have an existing similar setup between 2 datacenters where the WAN connection add below 1ms latency.
On "exceptionally large nodes", those are all identical servers, 3 per datacenter with 16 x 3.84 TB nvme disks, 128 AMD Epyc cores (on 2 sockets) and 1.5 TB memory, i would not count them as "exceptionally large".
I will read up a little more on asych replication.
Cheers Soeren
________________________________________ From: Joachim Kraftmayer <joachim.kraftmayer@clyso.com> Sent: Monday, April 28, 2025 8:23 AM To: Anthony D'Atri <anthony.datri@gmail.com> Cc: Soeren Malchow <soeren.malchow@convotis.com>; ceph-users@ceph.io < ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Stretched pool or not ?
Hi Soeren. First, I would like to clarify something. There are two options: stretched cluster and stretch mode. Sometimes this cannot be relied upon. If you have a “stretched-cluster” deployment in which much of your cluster is behind a single network component, you might need to use stretch mode to ensure data integrity. source: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#id1 The focus in this sentence is on ‘single network component’. I hope you don't have a single point of failure in your setup.
Which option is best for your requirements? Regards, Joachim
joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677
Am Mo., 28. Apr. 2025 um 02:50 Uhr schrieb Anthony D'Atri < anthony.datri@gmail.com>:
Please make list posts in plain text.
i am working on the plan for a 3 datacenter setup using ceph (in proxmox nodes).
Each datacenter has 3 physical nodes to start with and 100Gbit switches. I will also have 2 x 100 Gbit/s connectivity between the datacenters (each datacenter to each other). the physical nodes have 2 x 100Gbit/s for the public network and 2 x 100Gbit/s for the cluster network.
You almost certainly don’t need a cluster / replication network unless these are exceptionally large nodes.
About this setup i have 2 questions.
is it even necessary to evaluate a stretched cluster since the WAN
connections are as fast as the local ones (including the latency, since it is only 25km) ?
There’s more to latency than just distance. What is the measured latency? A:B, B:C, C:A?
If using a stretched pool across all 3 datacenters, what happens if one
datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ?
That sentence refers to a non-stretch cluster.
Tell us why you’re spreading across three DCs, what you’re trying to accomplish, and what your performance requirements are.
AIUI a stretch 3-site cluster requires all pools to be replicated, size=6.
Explicit stretch mode treats the mon quorum in a different way. With two OSD sites you deploy a tiebreaker at a third site, which is possibly just a cloud VM. With three OSD sites, I might speculate that one would deploy 7 mons, 2 At each OSD site + tiebreaker.
Operations on a stretch cluster can be slow. Sometimes separate clusters with asynchronous replication make more sense.
If i am on the wrong path, maybe someone has a link for me, where is can
find information on this setup ?
Cheers Soeren
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, ok, i am starting to get clearer on the wording, that seems to be one problem. The etcd database should be in virtual disks for each of the kubernetes nodes, not directly on ceph. I would plan either a "regular" replicated cluster and a stretched cluster (even if we would have to install 2 times, i think it is worth testing both), test both in behaviour and i assume an outcome of these results is also interesting for this mailing list or maybe a blog article. Thank you again for taking the time, it is much clearer now Cheers Soeren ________________________________ From: Joachim Kraftmayer <joachim.kraftmayer@clyso.com> Sent: Monday, April 28, 2025 1:11 PM To: Soeren Malchow <soeren.malchow@convotis.com> Cc: Anthony D'Atri <anthony.datri@gmail.com>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Stretched pool or not ? Hi Soeren, understood. So stretched pools need also a stretched ceph cluster. So a simple setup would be with replication size 3 for replicated pools and 3 or more ceph monitors, ... The failure domain is the datacenter. That means if you lose one DC the ceph cluster is still online. back to your original question:
If using a stretched pool across all 3 datacenters, what happens if one datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ?
If you have changed your failover domain to ‘Datacenter’, at least two DCs must be available to handle I/O operations. Do you want to use Ceph for the etcd database? Hope it helps, Joachim [https://lh7-qw.googleusercontent.com/docsz/AD_4nXeuFWFpeRWW3xA-qDZ9fTo1odW6y...] joachim.kraftmayer@clyso.com<mailto:joachim.kraftmayer@clyso.com> [https://lh7-qw.googleusercontent.com/docsz/AD_4nXcbgqtXMzhmdUBlDb9tcjGwRcVHD...] www.clyso.com<https://urldefense.com/v3/__http://www.clyso.com/__;!!KHkafc0ElKqcERD9YPg!g02qlzu3UHTr37Md2fCmnbBOA0_kR1B2PjHvhelDALqD8dPP1QSJuxIsR4TLaX8VJ4kbkQRTr29kHG9usjkr_08uiRE9s45K$> [https://lh7-qw.googleusercontent.com/docsz/AD_4nXeYECk1HXhGrVUJqpefhuHdvP4uH...] Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 [https://lh7-qw.googleusercontent.com/docsz/AD_4nXfNyeLIbOzh5A66KvA66b56OBK6t...] Am Mo., 28. Apr. 2025 um 12:15 Uhr schrieb Soeren Malchow <soeren.malchow@convotis.com<mailto:soeren.malchow@convotis.com>>: HI Joachim and Anthony, first: thanks for taking the time to answer. (now in plain text, sorry, i did not think about that) the sentence i am referring to is for "stretched pools" without a tiebreaker which is "stretch mode" if i understood the documentation correctly. I read this in the "Limitations" section exactly on the page your link refers to as well. The reason behind having 3 datacenters is because we are having alot of k8s clusters which also need to have quorum, if i distribute the etcd nodes across 3 datacenters, the outage of one datacenter will keep the k8s cluster operational. Thats why i was explicitly referring to stretched pools, not stretch mode (still hope i understand everything right). We do not have a single point of failure in the setup, all connections and devices are redundant. The latency between the datacenters is most likely very low (we can not measure since i am in planning stages. The connections between the datacenters are on dark fibres connected through modules directly in the Top of the Rack switches, compared to the local connectivity it will be almost the same. We have an existing similar setup between 2 datacenters where the WAN connection add below 1ms latency. On "exceptionally large nodes", those are all identical servers, 3 per datacenter with 16 x 3.84 TB nvme disks, 128 AMD Epyc cores (on 2 sockets) and 1.5 TB memory, i would not count them as "exceptionally large". I will read up a little more on asych replication. Cheers Soeren ________________________________________ From: Joachim Kraftmayer <joachim.kraftmayer@clyso.com<mailto:joachim.kraftmayer@clyso.com>> Sent: Monday, April 28, 2025 8:23 AM To: Anthony D'Atri <anthony.datri@gmail.com<mailto:anthony.datri@gmail.com>> Cc: Soeren Malchow <soeren.malchow@convotis.com<mailto:soeren.malchow@convotis.com>>; ceph-users@ceph.io<mailto:ceph-users@ceph.io> <ceph-users@ceph.io<mailto:ceph-users@ceph.io>> Subject: Re: [ceph-users] Re: Stretched pool or not ? Hi Soeren. First, I would like to clarify something. There are two options: stretched cluster and stretch mode. Sometimes this cannot be relied upon. If you have a “stretched-cluster” deployment in which much of your cluster is behind a single network component, you might need to use stretch mode to ensure data integrity. source: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#id1<https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/operations/stretch-mode/*id1__;Iw!!KHkafc0ElKqcERD9YPg!g02qlzu3UHTr37Md2fCmnbBOA0_kR1B2PjHvhelDALqD8dPP1QSJuxIsR4TLaX8VJ4kbkQRTr29kHG9usjkr_08uiaDvFI8S$> The focus in this sentence is on ‘single network component’. I hope you don't have a single point of failure in your setup. Which option is best for your requirements? Regards, Joachim joachim.kraftmayer@clyso.com<mailto:joachim.kraftmayer@clyso.com> www.clyso.com<https://urldefense.com/v3/__http://www.clyso.com__;!!KHkafc0ElKqcERD9YPg!g02qlzu3UHTr37Md2fCmnbBOA0_kR1B2PjHvhelDALqD8dPP1QSJuxIsR4TLaX8VJ4kbkQRTr29kHG9usjkr_08uiQyt45SG$> Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mo., 28. Apr. 2025 um 02:50 Uhr schrieb Anthony D'Atri <anthony.datri@gmail.com<mailto:anthony.datri@gmail.com>>: Please make list posts in plain text.
i am working on the plan for a 3 datacenter setup using ceph (in proxmox nodes).
Each datacenter has 3 physical nodes to start with and 100Gbit switches. I will also have 2 x 100 Gbit/s connectivity between the datacenters (each datacenter to each other). the physical nodes have 2 x 100Gbit/s for the public network and 2 x 100Gbit/s for the cluster network.
You almost certainly don’t need a cluster / replication network unless these are exceptionally large nodes.
About this setup i have 2 questions.
is it even necessary to evaluate a stretched cluster since the WAN connections are as fast as the local ones (including the latency, since it is only 25km) ?
There’s more to latency than just distance. What is the measured latency? A:B, B:C, C:A?
If using a stretched pool across all 3 datacenters, what happens if one datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ?
That sentence refers to a non-stretch cluster. Tell us why you’re spreading across three DCs, what you’re trying to accomplish, and what your performance requirements are. AIUI a stretch 3-site cluster requires all pools to be replicated, size=6. Explicit stretch mode treats the mon quorum in a different way. With two OSD sites you deploy a tiebreaker at a third site, which is possibly just a cloud VM. With three OSD sites, I might speculate that one would deploy 7 mons, 2 At each OSD site + tiebreaker. Operations on a stretch cluster can be slow. Sometimes separate clusters with asynchronous replication make more sense.
If i am on the wrong path, maybe someone has a link for me, where is can find information on this setup ?
Cheers Soeren
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
understood. So stretched pools need also a stretched ceph cluster.
The docs are a bit confusing, they refer to a stretched pool in a cluster that is not explicitly in stretch mode. We should probably not use “stretch” to describe anything that isn’t in a formal stretch mode cluster, as setting stretch mode affects behavior in certain ways.
So a simple setup would be with replication size 3 for replicated pools and 3 or more ceph monitors, ...
We want at least 2x mons per site + tiebreaker, so that not only can we form quorum, but that the cluster can operate if one crashes.
The reason behind having 3 datacenters is because we are having alot of k8s clusters which also need to have quorum, if i distribute the etcd nodes across 3 datacenters, the outage of one datacenter will keep the k8s cluster operational.
I think with K8s you could employ a strategy similar to Ceph’s stretch mode: * K8s workers and OSDs at *2* sites * 2x K8s control nodes + 1x Ceph mon at a tiebreaker site, which could even be just cloud VMs. That way the Ceph pools would only need R4 instead of R6.
The latency between the datacenters is most likely very low (we can not measure since i am in planning stages.
I know of one commercial Ceph support organization that dictates < 10ms RTT between OSD sites and < 100ms RTT to a tiebreaker mon. These thresholds might inform decisions and predictions. A quick web search asserts:
A rule of thumb is that RTT increases by approximately 1 millisecond (ms) for every 60 miles of distance.
The nuance to the formal stretch mode is the difference in how mon quorum is managed using reachability scores, and the automatic management of pools` min_size in order to maintain an operable cluster in the face of an entire DC going down. With a conventional cluster, if you have say 2 mons in one DC and 3 in another, loss of the second DC will result in an inoperable cluster unless one takes manual drastic action.
The connections between the datacenters are on dark fibres connected through modules directly in the Top of the Rack switches, compared to the local connectivity it will be almost the same. We have an existing similar setup between 2 datacenters where the WAN connection add below 1ms latency.
The same two DCs as would be in operation here?
On "exceptionally large nodes", those are all identical servers, 3 per datacenter with 16 x 3.84 TB nvme disks, 128 AMD Epyc cores (on 2 sockets) and 1.5 TB memory, i would not count them as "exceptionally large".
Gotcha. Is this a converged cluster? That’s a excess of cores and RAM just for Ceph if not.
I will read up a little more on asych replication.
RGW: multisite RBD: rbd-mirror CephFS: mirroring is fairly recent Part of the equation is having the clients be able to access the data, including if you’re solving for zero data *unavailability* vs zero *loss*. The latter is much easier than the former.
HI, the distance between the datacenters does not exceed 25km ( 15 miles ), current 2 DC setup is with one different datacenter provider but the same Dark Fibre provider. And yes, the clusters are hyperconverged, they are running proxmox. And the kubernetes nodes will run as VMs on top of Proxmox/KVM, our experience is, that 2 kubernetes control planes are not a good choice, you should have 3 or 5 to get a proper quorum. I read up on the quorum for the mon nodes in streched mode. In regards to unavailability vs. zero dataloss, we definitly prefer zero dataloss, but depending on the setup we are willing to make compromises and risk having to restore machines from backup. I will come up with a testplan (including simulating load) and try to test different scenarios and share the outcome Cheers Soeren ________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Monday, April 28, 2025 3:09 PM To: Joachim Kraftmayer <joachim.kraftmayer@clyso.com> Cc: Soeren Malchow <soeren.malchow@convotis.com>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Stretched pool or not ? understood. So stretched pools need also a stretched ceph cluster. The docs are a bit confusing, they refer to a stretched pool in a cluster that is not explicitly in stretch mode. We should probably not use “stretch” to describe anything that isn’t in a formal stretch mode cluster, as setting stretch mode affects behavior in certain ways. So a simple setup would be with replication size 3 for replicated pools and 3 or more ceph monitors, ... We want at least 2x mons per site + tiebreaker, so that not only can we form quorum, but that the cluster can operate if one crashes. The reason behind having 3 datacenters is because we are having alot of k8s clusters which also need to have quorum, if i distribute the etcd nodes across 3 datacenters, the outage of one datacenter will keep the k8s cluster operational. I think with K8s you could employ a strategy similar to Ceph’s stretch mode: * K8s workers and OSDs at *2* sites * 2x K8s control nodes + 1x Ceph mon at a tiebreaker site, which could even be just cloud VMs. That way the Ceph pools would only need R4 instead of R6. The latency between the datacenters is most likely very low (we can not measure since i am in planning stages. I know of one commercial Ceph support organization that dictates < 10ms RTT between OSD sites and < 100ms RTT to a tiebreaker mon. These thresholds might inform decisions and predictions. A quick web search asserts:
A rule of thumb is that RTT increases by approximately 1 millisecond (ms) for every 60 miles of distance.
The nuance to the formal stretch mode is the difference in how mon quorum is managed using reachability scores, and the automatic management of pools` min_size in order to maintain an operable cluster in the face of an entire DC going down. With a conventional cluster, if you have say 2 mons in one DC and 3 in another, loss of the second DC will result in an inoperable cluster unless one takes manual drastic action. The connections between the datacenters are on dark fibres connected through modules directly in the Top of the Rack switches, compared to the local connectivity it will be almost the same. We have an existing similar setup between 2 datacenters where the WAN connection add below 1ms latency. The same two DCs as would be in operation here? On "exceptionally large nodes", those are all identical servers, 3 per datacenter with 16 x 3.84 TB nvme disks, 128 AMD Epyc cores (on 2 sockets) and 1.5 TB memory, i would not count them as "exceptionally large". Gotcha. Is this a converged cluster? That’s a excess of cores and RAM just for Ceph if not. I will read up a little more on asych replication. RGW: multisite RBD: rbd-mirror CephFS: mirroring is fairly recent Part of the equation is having the clients be able to access the data, including if you’re solving for zero data *unavailability* vs zero *loss*. The latter is much easier than the former.
participants (3)
-
Anthony D'Atri
-
Joachim Kraftmayer
-
Soeren Malchow