Clarification of documentation
In the docs: https://docs.ceph.com/docs/master/radosgw/multisite/ - in the section Requirements and Assumptions There is this warning hint: "Running a single Ceph storage cluster is NOT recommended unless you have low latency WAN connections." What exactly does "single Ceph storage cluster" mean? * Just having one site? * Do I need to run multiple independent clusters across a number of n sites before I can do multi-site - so it would not be possible to start with one site and add further sites step by step? And what exactly happens with higher latency WAN connections?
It is my understanding that it refers to running a single, normal ceph cluster with it's component hosts connected over WAN. This would require OSDs to connect to other OSDs and mons over WAN for nearly every operation, and is not likely to perform acceptably.
It is possible to run a ceph cluster over a WAN if you have reliable enough WAN with sites close enough for low-ish latency. The OSiRIS project is architected that way with Ceph services spread evenly across three university sites in Michigan. There's more information and contact on their website: http://www.osris.org We had a variety of interesting WAN outages which Ceph has always handled well in terms of not losing data or our cluster definitions. Outages were at times further complicated by inconsistent pathing for cluster and backend networks such that only one or the other might be up to some sites. In all that, with 3 mons situated 1 per site, we never encountered any kind of split brain situations. Though I'm no longer involved the project is still ongoing and I'm sure if you want to reach out they (or I personally) would be happy to answer any questions. thanks, Ben On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth@gmail.com> wrote:
It is my understanding that it refers to running a single, normal ceph cluster with it's component hosts connected over WAN. This would require OSDs to connect to other OSDs and mons over WAN for nearly every operation, and is not likely to perform acceptably. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have been running Ceph over a gigabit WAN for a few months now and have been happy with it. Mine is set up with Strongswan tunnels And dynamic routing with BIRD) (although I would have used transport Mode and iBGP in hindsight). I generally have 300-500kbps flow with 5ms latency. What I specifically make sure of is the osds and clients are all on one side of the connection. That may seem odd, but it allows me to run a HA cluster with only two machines on one end. And if you think through the failure configurations is that line goes down, they pretty much all work if two machines are up and you don’t do maintenance on them during thunderstorms :-). Sent from my iPhone
On May 19, 2020, at 11:34, Benjeman Meekhof <bmeekhof@umich.edu> wrote:
It is possible to run a ceph cluster over a WAN if you have reliable enough WAN with sites close enough for low-ish latency. The OSiRIS project is architected that way with Ceph services spread evenly across three university sites in Michigan. There's more information and contact on their website: http://www.osris.org
We had a variety of interesting WAN outages which Ceph has always handled well in terms of not losing data or our cluster definitions. Outages were at times further complicated by inconsistent pathing for cluster and backend networks such that only one or the other might be up to some sites. In all that, with 3 mons situated 1 per site, we never encountered any kind of split brain situations.
Though I'm no longer involved the project is still ongoing and I'm sure if you want to reach out they (or I personally) would be happy to answer any questions.
thanks, Ben
On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth@gmail.com> wrote:
It is my understanding that it refers to running a single, normal ceph cluster with it's component hosts connected over WAN. This would require OSDs to connect to other OSDs and mons over WAN for nearly every operation, and is not likely to perform acceptably. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, May 19, 2020 at 10:34 AM Benjeman Meekhof <bmeekhof@umich.edu> wrote:
It is possible to run a ceph cluster over a WAN if you have reliable enough WAN with sites close enough for low-ish latency. The OSiRIS project is architected that way with Ceph services spread evenly across three university sites in Michigan. There's more information and contact on their website: http://www.osris.org
We had a variety of interesting WAN outages which Ceph has always handled well in terms of not losing data or our cluster definitions. Outages were at times further complicated by inconsistent pathing for cluster and backend networks such that only one or the other might be up to some sites. In all that, with 3 mons situated 1 per site, we never encountered any kind of split brain situations.
Oooh, are those stories written down anywhere? I'm working on explicitly supporting 2-site stretch clusters right now (with a "tiebreaker monitor" in a third site, and the main thing is handling those networking issues) but I imagine we'll extend it to do 3 sites in the future. If I have some real-world failure experiences to validate against that'd be good. -Greg
Though I'm no longer involved the project is still ongoing and I'm sure if you want to reach out they (or I personally) would be happy to answer any questions.
thanks, Ben
On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth@gmail.com> wrote:
It is my understanding that it refers to running a single, normal ceph cluster with it's component hosts connected over WAN. This would require OSDs to connect to other OSDs and mons over WAN for nearly every operation, and is not likely to perform acceptably. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Greg, My name's Zac and I'm the docs guy for the Ceph Foundation. I have a long-term plan to create a document that collects error codes and failure cases, but I am only one man and it will be a few months before I can begin on it. Zac Dover Ceph Docs Guy On Wed, May 20, 2020 at 4:32 AM Gregory Farnum <gfarnum@redhat.com> wrote:
On Tue, May 19, 2020 at 10:34 AM Benjeman Meekhof <bmeekhof@umich.edu> wrote:
It is possible to run a ceph cluster over a WAN if you have reliable enough WAN with sites close enough for low-ish latency. The OSiRIS project is architected that way with Ceph services spread evenly across three university sites in Michigan. There's more information and contact on their website: http://www.osris.org
We had a variety of interesting WAN outages which Ceph has always handled well in terms of not losing data or our cluster definitions. Outages were at times further complicated by inconsistent pathing for cluster and backend networks such that only one or the other might be up to some sites. In all that, with 3 mons situated 1 per site, we never encountered any kind of split brain situations.
Oooh, are those stories written down anywhere? I'm working on explicitly supporting 2-site stretch clusters right now (with a "tiebreaker monitor" in a third site, and the main thing is handling those networking issues) but I imagine we'll extend it to do 3 sites in the future. If I have some real-world failure experiences to validate against that'd be good. -Greg
Though I'm no longer involved the project is still ongoing and I'm sure if you want to reach out they (or I personally) would be happy to answer any questions.
thanks, Ben
On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth@gmail.com> wrote:
It is my understanding that it refers to running a single, normal ceph cluster with it's component hosts connected over WAN. This would require OSDs to connect to other OSDs and mons over WAN for nearly every operation, and is not likely to perform acceptably. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Zac, I have some further questions on that page: Right before the section "Delete Default Zone Group and Zone" there is another warning that says: "The following steps assume a multi-site configuration using newly installed systems that aren’t storing data yet. DO NOT DELETE the default zone and its pools if you are already using it to store data, or the data will be deleted and unrecoverable." Question: Is it a problem when I set up the multi-site configuration on a system that is already storing data - as long as I don't remove the default zone / zone group? If it is possible to do so, another sentence that says "You can still setup the multi-site with new, non-default zone groups / zones when the system already was storing data". Then, the section starts with the instruction "Delete the default zone if it exists. Make sure to remove it from the default zone group first.". I wonder whether it is really necessary to do that because the section "Create a Secondary Zone" says "Delete the default zone if needed." which sounds like it is optional to do so. In the section "Create a System User": There is the instruction "Finally, add the system user to the master zone." The command line for that displayed there is # radosgw-admin zone modify --rgw-zone=us-east --access-key={access-key} --secret={secret} # radosgw-admin period update --commit Then, directly after that follows the section "Update the Period" which also has an instruction to execute the command: # radosgw-admin period update --commit It seems that the first ocurrence of that command I mentioned is redundant. In the section "Changing the Metadata Master Zone" there is this warning: "Care must be taken when changing which zone is the metadata master. If a zone has not finished syncing metadata from the current master zone, it will be unable to serve any remaining entries when promoted to master and those changes will be lost." Question: Is there no configuration option that prevents this (at least requiring something like a "--force" flag) while sync is still in progress? This does not sound like a "Secure-by-Default" system.
You need a third monitor in order to form a quorum if one of the two sites goes down. With only two sites, there is no safe way for them to decide who is down. On Tue, May 19, 2020 at 3:11 PM CodingSpiderFox <codingspiderfox@gmail.com> wrote:
What does tiebreaker monitor mean? What exactly is its purpose? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Benjeman Meekhof
-
Brian Topping
-
CodingSpiderFox
-
Gregory Farnum
-
John Zachary Dover
-
Nathan Fish