moving small production cluster to different datacenter
Say one is forced to move a production cluster (4 nodes) to a different datacenter. What options do I have, other than just turning it off at the old location and on on the new location? Maybe buying some extra nodes, and move one node at a time?
On 1/28/20 11:19 AM, Marc Roos wrote:
Say one is forced to move a production cluster (4 nodes) to a different datacenter. What options do I have, other than just turning it off at the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time?
I did this ones. This cluster was running IPv6-only (still is) and thus I had the flexibility of new IPs. First I temporarily moved the MONs from hardware to Virtual. MONMAP went from 3 to 5 MONs. Then I moved the MONs one by one to the new DC and then removed the 2 additional VMs. Then I set the 'noout' flag and moved the OSD nodes one by one. These datacenters were located very close thus each node could be moved within 20 minutes. Wait for recovery to finish and then move the next node. Keep in mind that there is/might be additional latency between the two datacenters. Wido
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We did this as well, pretty much the same as Wido. We had a fiber connection with good latency between the locations. We installed a virtual monitor in the destination datacenter to always keep quorum then we simply moved one node at a time after setting noout. When we took a node up on the destination we had a small moving of data then the cluster was back to healthy again. We had a higher apply and commit latency until we all the nodes was on the destination side but we never noticed any performance issues that caused issues for us. Best regards On 1/28/20 1:30 PM, Wido den Hollander wrote:
On 1/28/20 11:19 AM, Marc Roos wrote:
Say one is forced to move a production cluster (4 nodes) to a different datacenter. What options do I have, other than just turning it off at the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time? I did this ones. This cluster was running IPv6-only (still is) and thus I had the flexibility of new IPs.
First I temporarily moved the MONs from hardware to Virtual. MONMAP went from 3 to 5 MONs.
Then I moved the MONs one by one to the new DC and then removed the 2 additional VMs.
Then I set the 'noout' flag and moved the OSD nodes one by one. These datacenters were located very close thus each node could be moved within 20 minutes.
Wait for recovery to finish and then move the next node.
Keep in mind that there is/might be additional latency between the two datacenters.
Wido
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
And us too, exactly as below. One at a time then wait for things to recover before moving the next host. We didn't have any issues with this approach either. Regards, Simon. On 28/01/2020 13:03, Tobias Urdin wrote:
We did this as well, pretty much the same as Wido. We had a fiber connection with good latency between the locations.
We installed a virtual monitor in the destination datacenter to always keep quorum then we simply moved one node at a time after setting noout.
When we took a node up on the destination we had a small moving of data then the cluster was back to healthy again.
We had a higher apply and commit latency until we all the nodes was on the destination side but we never noticed any performance issues that caused issues for us.
I have osd nodes combined with mds,mgr and mon's. There are also running a few VM's on them with libvirt, however client en cluster on ipv4 (and no experience with ipv6). cluster network is on a switch not connected to the internet. - I should enable again ipv6 - enable forwarding so cluster communication is being routed, through the client interfaces? - test if the connection works between new and old - then add two vm's with monitors bringing total to 5 - then move one osd node with mon to the new location. - wait for recovery - then move the two vm mon's to the new location osd node, so there are 3 there. - move an osd node to the new location - wait for recovery Etc. Something like this? What is the idea about having 5 monitors in this migration? -----Original Message----- To: ceph-users Subject: Re: [ceph-users] moving small production cluster to different datacenter On 1/28/20 11:19 AM, Marc Roos wrote:
Say one is forced to move a production cluster (4 nodes) to a different datacenter. What options do I have, other than just turning it off at the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time?
I did this ones. This cluster was running IPv6-only (still is) and thus I had the flexibility of new IPs. First I temporarily moved the MONs from hardware to Virtual. MONMAP went from 3 to 5 MONs. Then I moved the MONs one by one to the new DC and then removed the 2 additional VMs. Then I set the 'noout' flag and moved the OSD nodes one by one. These datacenters were located very close thus each node could be moved within 20 minutes. Wait for recovery to finish and then move the next node. Keep in mind that there is/might be additional latency between the two datacenters. Wido
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I did this, but with the benefit of taking the network with me, just a forklift from one datacenter to the next. Shutdown the clients, then OSDs, then MDS/MON/MGRs, then switches. Reverse order back up,
On Jan 28, 2020, at 4:19 AM, Marc Roos <M.Roos@f1-outsourcing.eu> wrote:
Say one is forced to move a production cluster (4 nodes) to a different datacenter. What options do I have, other than just turning it off at the old location and on on the new location?
Maybe buying some extra nodes, and move one node at a time?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Marc Roos
-
Reed Dier
-
Simon Ironside
-
Tobias Urdin
-
Wido den Hollander