28 Jan
2020
28 Jan
'20
10:19 a.m.
Say one is forced to move a production cluster (4 nodes) to a different
datacenter. What options do I have, other than just turning it off at
the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time?
28 Jan
28 Jan
12:29 p.m.
On 1/28/20 11:19 AM, Marc Roos wrote:
Say one is forced to move a production cluster (4 nodes) to a different
datacenter. What options do I have, other than just turning it off at
the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time?
I did this ones. This cluster was running IPv6-only (still is) and thus
I had the flexibility of new IPs.
First I temporarily moved the MONs from hardware to Virtual. MONMAP went
from 3 to 5 MONs.
Then I moved the MONs one by one to the new DC and then removed the 2
additional VMs.
Then I set the 'noout' flag and moved the OSD nodes one by one. These
datacenters were located very close thus each node could be moved within
20 minutes.
Wait for recovery to finish and then move the next node.
Keep in mind that there is/might be additional latency between the two
datacenters.
Wido
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
1:03 p.m.
We did this as well, pretty much the same as Wido.
We had a fiber connection with good latency between the locations.
We installed a virtual monitor in the destination datacenter to always
keep quorum then we
simply moved one node at a time after setting noout.
When we took a node up on the destination we had a small moving of data
then the cluster
was back to healthy again.
We had a higher apply and commit latency until we all the nodes was on
the destination
side but we never noticed any performance issues that caused issues for us.
Best regards
On 1/28/20 1:30 PM, Wido den Hollander wrote:
On 1/28/20 11:19 AM, Marc Roos wrote:
Say one is forced to move a production cluster (4
nodes) to a different
datacenter. What options do I have, other than just turning it off at
the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time?
I did this ones. This
cluster was running IPv6-only (still is) and thus
I had the flexibility of new IPs.
First I temporarily moved the MONs from hardware to Virtual. MONMAP went
from 3 to 5 MONs.
Then I moved the MONs one by one to the new DC and then removed the 2
additional VMs.
Then I set the 'noout' flag and moved the OSD nodes one by one. These
datacenters were located very close thus each node could be moved within
20 minutes.
Wait for recovery to finish and then move the next node.
Keep in mind that there is/might be additional latency between the two
datacenters.
Wido
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:39 p.m.
And us too, exactly as below. One at a time then wait for things to
recover before moving the next host. We didn't have any issues with this
approach either.
Regards,
Simon.
On 28/01/2020 13:03, Tobias Urdin wrote:
> We did this as well, pretty much the same as Wido.
> We had a fiber connection with good latency between the locations.
>
> We installed a virtual monitor in the destination datacenter to always
> keep quorum then we
> simply moved one node at a time after setting noout.
>
> When we took a node up on the destination we had a small moving of data
> then the cluster
> was back to healthy again.
>
> We had a higher apply and commit latency until we all the nodes was on
> the destination
> side but we never noticed any performance issues that caused issues for
> us.
30 Jan
30 Jan
6:12 p.m.
I have osd nodes combined with mds,mgr and mon's. There are also running
a few VM's on them with libvirt, however client en cluster on ipv4 (and
no experience with ipv6). cluster network is on a switch not connected
to the internet.
- I should enable again ipv6
- enable forwarding so cluster communication is being routed, through
the client interfaces?
- test if the connection works between new and old
- then add two vm's with monitors bringing total to 5
- then move one osd node with mon to the new location.
- wait for recovery
- then move the two vm mon's to the new location osd node, so there are
3 there.
- move an osd node to the new location
- wait for recovery
Etc.
Something like this? What is the idea about having 5 monitors in this
migration?
-----Original Message-----
To: ceph-users
Subject: Re: [ceph-users] moving small production cluster to different
datacenter
On 1/28/20 11:19 AM, Marc Roos wrote:
Say one is forced to move a production cluster (4 nodes) to a
different datacenter. What options do I have, other than just turning
it off at the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time?
I did this ones. This cluster was running IPv6-only (still is) and thus
I had the flexibility of new IPs.
First I temporarily moved the MONs from hardware to Virtual. MONMAP went
from 3 to 5 MONs.
Then I moved the MONs one by one to the new DC and then removed the 2
additional VMs.
Then I set the 'noout' flag and moved the OSD nodes one by one. These
datacenters were located very close thus each node could be moved within
20 minutes.
Wait for recovery to finish and then move the next node.
Keep in mind that there is/might be additional latency between the two
datacenters.
Wido
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
28 Jan
28 Jan
5:56 p.m.
I did this, but with the benefit of taking the network with me, just a forklift from one
datacenter to the next.
Shutdown the clients, then OSDs, then MDS/MON/MGRs, then switches.
Reverse order back up,
On Jan 28, 2020, at 4:19 AM, Marc Roos
<M.Roos(a)f1-outsourcing.eu> wrote:
Say one is forced to move a production cluster (4 nodes) to a different
datacenter. What options do I have, other than just turning it off at
the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1555
days inactive
1557
days old
5 comments
5 participants
participants (5)
-
Marc Roos -
Reed Dier -
Simon Ironside -
Tobias Urdin -
Wido den Hollander