19 May
2020
19 May
'20
10:26 p.m.
In the docs: https://docs.ceph.com/docs/master/radosgw/multisite/ - in the section
Requirements and Assumptions
There is this warning hint:
"Running a single Ceph storage cluster is NOT recommended unless you have low latency
WAN connections."
What exactly does "single Ceph storage cluster" mean?
* Just having one site?
* Do I need to run multiple independent clusters across a number of n sites before I can
do multi-site - so it would not be possible to start with one site and add further sites
step by step?
And what exactly happens with higher latency WAN connections?
19 May
19 May
10:32 p.m.
It is my understanding that it refers to running a single, normal ceph
cluster with it's component hosts connected over WAN. This would
require OSDs to connect to other OSDs and mons over WAN for nearly
every operation, and is not likely to perform acceptably.
11:03 p.m.
It is possible to run a ceph cluster over a WAN if you have reliable
enough WAN with sites close enough for low-ish latency. The OSiRIS
project is architected that way with Ceph services spread evenly
across three university sites in Michigan. There's more information
and contact on their website: http://www.osris.org
We had a variety of interesting WAN outages which Ceph has always
handled well in terms of not losing data or our cluster definitions.
Outages were at times further complicated by inconsistent pathing for
cluster and backend networks such that only one or the other might be
up to some sites. In all that, with 3 mons situated 1 per site, we
never encountered any kind of split brain situations.
Though I'm no longer involved the project is still ongoing and I'm
sure if you want to reach out they (or I personally) would be happy to
answer any questions.
thanks,
Ben
On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth(a)gmail.com> wrote:
>
> It is my understanding that it refers to running a single, normal ceph
> cluster with it's component hosts connected over WAN. This would
> require OSDs to connect to other OSDs and mons over WAN for nearly
> every operation, and is not likely to perform acceptably.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:39 p.m.
I have been running Ceph over a gigabit WAN for a few months now and have been happy with
it. Mine is set up with Strongswan tunnels And dynamic routing with BIRD) (although I
would have used transport Mode and iBGP in hindsight). I generally have 300-500kbps flow
with 5ms latency.
What I specifically make sure of is the osds and clients are all on one side of the
connection. That may seem odd, but it allows me to run a HA cluster with only two machines
on one end. And if you think through the failure configurations is that line goes down,
they pretty much all work if two machines are up and you don’t do maintenance on them
during thunderstorms :-).
Sent from my iPhone
> On May 19, 2020, at 11:34, Benjeman Meekhof <bmeekhof(a)umich.edu> wrote:
>
> It is possible to run a ceph cluster over a WAN if you have reliable
> enough WAN with sites close enough for low-ish latency. The OSiRIS
> project is architected that way with Ceph services spread evenly
> across three university sites in Michigan. There's more information
> and contact on their website: http://www.osris.org
>
> We had a variety of interesting WAN outages which Ceph has always
> handled well in terms of not losing data or our cluster definitions.
> Outages were at times further complicated by inconsistent pathing for
> cluster and backend networks such that only one or the other might be
> up to some sites. In all that, with 3 mons situated 1 per site, we
> never encountered any kind of split brain situations.
>
> Though I'm no longer involved the project is still ongoing and I'm
> sure if you want to reach out they (or I personally) would be happy to
> answer any questions.
>
> thanks,
> Ben
>
>> On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth(a)gmail.com> wrote:
>>
>> It is my understanding that it refers to running a single, normal ceph
>> cluster with it's component hosts connected over WAN. This would
>> require OSDs to connect to other OSDs and mons over WAN for nearly
>> every operation, and is not likely to perform acceptably.
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:59 p.m.
On Tue, May 19, 2020 at 10:34 AM Benjeman Meekhof <bmeekhof(a)umich.edu> wrote:
It is possible to run a ceph cluster over a WAN if you have reliable
enough WAN with sites close enough for low-ish latency. The OSiRIS
project is architected that way with Ceph services spread evenly
across three university sites in Michigan. There's more information
and contact on their website: http://www.osris.org
We had a variety of interesting WAN outages which Ceph has always
handled well in terms of not losing data or our cluster definitions.
Outages were at times further complicated by inconsistent pathing for
cluster and backend networks such that only one or the other might be
up to some sites. In all that, with 3 mons situated 1 per site, we
never encountered any kind of split brain situations.
Oooh, are those stories written down anywhere? I'm working on
explicitly supporting 2-site stretch clusters right now (with a
"tiebreaker monitor" in a third site, and the main thing is handling
those networking issues) but I imagine we'll extend it to do 3 sites
in the future. If I have some real-world failure experiences to
validate against that'd be good.
-Greg
Though I'm no longer involved the project is still ongoing and I'm
sure if you want to reach out they (or I personally) would be happy to
answer any questions.
thanks,
Ben
On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth(a)gmail.com> wrote:
It is my understanding that it refers to running a single, normal ceph
cluster with it's component hosts connected over WAN. This would
require OSDs to connect to other OSDs and mons over WAN for nearly
every operation, and is not likely to perform acceptably.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
20 May
20 May
12:20 a.m.
Greg,
My name's Zac and I'm the docs guy for the Ceph Foundation. I have a
long-term plan to create a document that collects error codes and failure
cases, but I am only one man and it will be a few months before I can begin
on it.
Zac Dover
Ceph Docs Guy
On Wed, May 20, 2020 at 4:32 AM Gregory Farnum <gfarnum(a)redhat.com> wrote:
On Tue, May 19, 2020 at 10:34 AM Benjeman Meekhof
<bmeekhof(a)umich.edu>
wrote:
It is possible to run a ceph cluster over a WAN if you have reliable
enough WAN with sites close enough for low-ish latency. The OSiRIS
project is architected that way with Ceph services spread evenly
across three university sites in Michigan. There's more information
and contact on their website: http://www.osris.org
We had a variety of interesting WAN outages which Ceph has always
handled well in terms of not losing data or our cluster definitions.
Outages were at times further complicated by inconsistent pathing for
cluster and backend networks such that only one or the other might be
up to some sites. In all that, with 3 mons situated 1 per site, we
never encountered any kind of split brain situations.
Oooh, are those stories written down anywhere? I'm working on
explicitly supporting 2-site stretch clusters right now (with a
"tiebreaker monitor" in a third site, and the main thing is handling
those networking issues) but I imagine we'll extend it to do 3 sites
in the future. If I have some real-world failure experiences to
validate against that'd be good.
-Greg
Though I'm no longer involved the project is still ongoing and I'm
sure if you want to reach out they (or I personally) would be happy to
answer any questions.
thanks,
Ben
On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth(a)gmail.com> wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
It is my understanding that it refers to running a single, normal ceph
cluster with it's component hosts connected over WAN. This would
require OSDs to connect to other OSDs and mons over WAN for nearly
every operation, and is not likely to perform acceptably.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:07 a.m.
Hello Zac,
I have some further questions on that page:
Right before the section "Delete Default Zone Group and Zone" there is another
warning that says:
"The following steps assume a multi-site configuration using newly installed systems
that aren’t storing data yet. DO NOT DELETE the default zone and its pools if you are
already using it to store data, or the data will be deleted and unrecoverable."
Question: Is it a problem when I set up the multi-site configuration on a system that is
already storing data - as long as I don't remove the default zone / zone group?
If it is possible to do so, another sentence that says "You can still setup the
multi-site with new, non-default zone groups / zones when the system already was storing
data".
Then, the section starts with the instruction "Delete the default zone if it exists.
Make sure to remove it from the default zone group first.".
I wonder whether it is really necessary to do that because the section "Create a
Secondary Zone" says "Delete the default zone if needed." which sounds like
it is optional to do so.
In the section "Create a System User":
There is the instruction "Finally, add the system user to the master zone."
The command line for that displayed there is
# radosgw-admin zone modify --rgw-zone=us-east --access-key={access-key}
--secret={secret}
# radosgw-admin period update --commit
Then, directly after that follows the section "Update the Period" which also has
an instruction to execute the command:
# radosgw-admin period update --commit
It seems that the first ocurrence of that command I mentioned is redundant.
In the section "Changing the Metadata Master Zone" there is this warning:
"Care must be taken when changing which zone is the metadata master. If a zone has
not finished syncing metadata from the current master zone, it will be unable to serve any
remaining entries when promoted to master and those changes will be lost."
Question: Is there no configuration option that prevents this (at least requiring
something like a "--force" flag) while sync is still in progress? This does not
sound like a "Secure-by-Default" system.
12:42 a.m.
You need a third monitor in order to form a quorum if one of the two
sites goes down. With only two sites, there is no safe way for them to
decide who is down.
On Tue, May 19, 2020 at 3:11 PM CodingSpiderFox
<codingspiderfox(a)gmail.com> wrote:
>
> What does tiebreaker monitor mean? What exactly is its purpose?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1446
days inactive
1446
days old
10 comments
6 participants
participants (6)
-
Benjeman Meekhof -
Brian Topping -
CodingSpiderFox -
Gregory Farnum -
John Zachary Dover -
Nathan Fish