On Tue, May 19, 2020 at 10:34 AM Benjeman Meekhof <bmeekhof@umich.edu> wrote:
It is possible to run a ceph cluster over a WAN if you have reliable enough WAN with sites close enough for low-ish latency. The OSiRIS project is architected that way with Ceph services spread evenly across three university sites in Michigan. There's more information and contact on their website: http://www.osris.org
We had a variety of interesting WAN outages which Ceph has always handled well in terms of not losing data or our cluster definitions. Outages were at times further complicated by inconsistent pathing for cluster and backend networks such that only one or the other might be up to some sites. In all that, with 3 mons situated 1 per site, we never encountered any kind of split brain situations.
Oooh, are those stories written down anywhere? I'm working on explicitly supporting 2-site stretch clusters right now (with a "tiebreaker monitor" in a third site, and the main thing is handling those networking issues) but I imagine we'll extend it to do 3 sites in the future. If I have some real-world failure experiences to validate against that'd be good. -Greg
Though I'm no longer involved the project is still ongoing and I'm sure if you want to reach out they (or I personally) would be happy to answer any questions.
thanks, Ben
On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth@gmail.com> wrote:
It is my understanding that it refers to running a single, normal ceph cluster with it's component hosts connected over WAN. This would require OSDs to connect to other OSDs and mons over WAN for nearly every operation, and is not likely to perform acceptably. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io