Live migrate RBD image with a client using it
Hello guys, We have been reading the docs, and trying to reproduce that process in our Ceph cluster. However, we always receive the following message: ``` librbd::Migration: prepare: image has watchers - not migrating rbd: preparing migration failed: (16) Device or resource busy ``` We tested with both RDB blocks mounted with kRBD, and with librbd via KVM/qEMU system. Both cases result in the same result. For KRBD, we understood that it is not supported right now, but for librbd it seems that it should be supported somehow. How do you guys handle those situations? We have the following use cases that might need an image to migrate between pools while the client is still consuming it. - RBD images that are consumed via the iSCSI gateways - RBD images mounted (rbd map) in hosts - RBD images used by KVM/Libvirt Does Ceph support a live migration of images between pools while the clients/consumers are still using those volumes? [1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/
Hi, the docs you mentioned also state:
All clients using the source image must be stopped prior to preparing a live-migration. The prepare step will fail if it finds any running clients with the image open in read/write mode. Once the prepare step is complete, the clients can be restarted using the new target image name. Attempting to restart the clients using the source image name will result in failure.
So I don't think you can live-migrate without interruption, at least not at the moment. Regards, Eugen Zitat von Work Ceph <work.ceph.user.mailing@gmail.com>:
Hello guys,
We have been reading the docs, and trying to reproduce that process in our Ceph cluster. However, we always receive the following message:
```
librbd::Migration: prepare: image has watchers - not migrating
rbd: preparing migration failed: (16) Device or resource busy
```
We tested with both RDB blocks mounted with kRBD, and with librbd via KVM/qEMU system. Both cases result in the same result. For KRBD, we understood that it is not supported right now, but for librbd it seems that it should be supported somehow.
How do you guys handle those situations?
We have the following use cases that might need an image to migrate between pools while the client is still consuming it.
- RBD images that are consumed via the iSCSI gateways - RBD images mounted (rbd map) in hosts - RBD images used by KVM/Libvirt
Does Ceph support a live migration of images between pools while the clients/consumers are still using those volumes?
[1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Exactly, I have seen that. However, that also means that it is not a "process" then, right? Am I missing something? If we need a live process, where the clients cannot unmpa the volumes, what do you guys recommend? On Wed, Apr 12, 2023 at 10:01 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
the docs you mentioned also state:
All clients using the source image must be stopped prior to preparing a live-migration. The prepare step will fail if it finds any running clients with the image open in read/write mode. Once the prepare step is complete, the clients can be restarted using the new target image name. Attempting to restart the clients using the source image name will result in failure.
So I don't think you can live-migrate without interruption, at least not at the moment.
Regards, Eugen
Zitat von Work Ceph <work.ceph.user.mailing@gmail.com>:
Hello guys,
We have been reading the docs, and trying to reproduce that process in our Ceph cluster. However, we always receive the following message:
```
librbd::Migration: prepare: image has watchers - not migrating
rbd: preparing migration failed: (16) Device or resource busy
```
We tested with both RDB blocks mounted with kRBD, and with librbd via KVM/qEMU system. Both cases result in the same result. For KRBD, we understood that it is not supported right now, but for librbd it seems that it should be supported somehow.
How do you guys handle those situations?
We have the following use cases that might need an image to migrate between pools while the client is still consuming it.
- RBD images that are consumed via the iSCSI gateways - RBD images mounted (rbd map) in hosts - RBD images used by KVM/Libvirt
Does Ceph support a live migration of images between pools while the clients/consumers are still using those volumes?
[1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 4/12/23 19:09, Work Ceph wrote:
Exactly, I have seen that. However, that also means that it is not a "process" then, right? Am I missing something?
If we need a live process, where the clients cannot unmpa the volumes, what do you guys recommend?
We have performed a "live migration" between two ceph cluster in our openstack setup in the past. Cinder and glance were configured with ceph pools as data backend. It might work for other virtualization solution, too. The blocker is usually the fact that the virtual machines keeps a lock on the rbd image by keeping its header object open. We solved this problem by performing a live migration to a different host. The necessary steps are: 1. setup rbd mirroring for source and target (image based mirroring, not the complete pool) for each virtual machine: 2. enable mirroring for all rbd volumes used by the VM 3. wait until mirroring is complete and data is synced 4. pause the virtual machine (prevents further writes to the rbds!) 5. in case of openstack update the cinder database with the new ceph cluster information (yes, cinder database contains the mon ip address for all images...) 6. demote rbd image on old pools and promote it on new pool 7. live migrate paused instance to another host 8. resume instance 9. after successful migration move old image to trash This will not work for kernel mapped rbd images. And given the complexity of the process I would advise to perform a cold migration if possible. I've written a perl script to perform the per image steps, not sure whether it will still work as expected. There's also a short period between steps 4 and 8 without an active virtual machine. Another solution for libvirt based virtual machines is block migration within qemu itself. Proxmox is using this for storage migrations. Works quite well within proxmox (btw thx for the great software), but I haven't done it manually yet. YMMV. Best regards, Burkhard Linke
I've used a similar process with great success for capacity management -- moving volumes from very full clusters to ones with more free space. There was a weighting system to direct new volumes where there was space, but, to forestall full ratio problems due to organic growth of existing thin-provisioned volumes sometimes explicit moving was needed. Unattached volumes were the low-hanging fruit, they could be moved at will, ideally with a lock preventing attachment during the process. This was libvirt-based, but before QEMU had its own migration. I'd power off the VM where the below pauses and migrates, so when it powered back up it got the new attachment. We had a service wrapped around rbd-mirror that handled the state transitions and queueing. I did at most two volumes in parallel, and had to tweak rbd-mirror options to prevent an hours-long catch-up before the primary/secondary could be swapped. This was on Luminous. Of course, that isn't live migration, but it's some context.
We have performed a "live migration" between two ceph cluster in our openstack setup in the past. Cinder and glance were configured with ceph pools as data backend.
It might work for other virtualization solution, too. The blocker is usually the fact that the virtual machines keeps a lock on the rbd image by keeping its header object open. We solved this problem by performing a live migration to a different host.
The necessary steps are:
1. setup rbd mirroring for source and target (image based mirroring, not the complete pool)
for each virtual machine:
2. enable mirroring for all rbd volumes used by the VM
3. wait until mirroring is complete and data is synced
4. pause the virtual machine (prevents further writes to the rbds!)
5. in case of openstack update the cinder database with the new ceph cluster information (yes, cinder database contains the mon ip address for all images...)
6. demote rbd image on old pools and promote it on new pool
7. live migrate paused instance to another host
8. resume instance
9. after successful migration move old image to trash
This will not work for kernel mapped rbd images. And given the complexity of the process I would advise to perform a cold migration if possible. I've written a perl script to perform the per image steps, not sure whether it will still work as expected. There's also a short period between steps 4 and 8 without an active virtual machine.
Another solution for libvirt based virtual machines is block migration within qemu itself. Proxmox is using this for storage migrations. Works quite well within proxmox (btw thx for the great software), but I haven't done it manually yet. YMMV.
Best regards,
Burkhard Linke
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
For KVM virtual machines, one of my coworkers worked out a way to live migrate a VM with its storage to another node on the same cluster with storage in a different pool. This requires that the VM can be live migrated to a new host that has access to the same Ceph cluster, and enough bandwidth between the two hosts (or an idle enough VM) that the RAM and disk replication can keep up with the operation of the server. This probably originated from a webpage somewhere, but I didn't find it after several attempts. The steps were: 1) Create an empty RBD in the new pool that is the same size as the RBD in the existing pool qemu-img info rbd:sourcepool/rbdname qemu-img create -f rbd rbd:targetpool/rbdname <size>G 2) Dump a copy of the libvirt XML file for the virtual machine virsh dumpxml vmname > vmname-migrate.xml 3) Edit the dumped XML file to update the pool name <source protocol='rbd' name='targetpool/rbdname'> 4) Migrate the virtual machine to another host in the same cluster virsh migrate --live --persistent --copy-storage-all --verbose --xml vmname-migrate.xml vmname qemu+ssh://target_host/system 5) After migration is complete and you've confirmed the VM is running on the target host, delete or rename the rbd image on the old pool to make sure the VM cannot accidentally boot from it again later rbd mv sourcepool/rbdname sourcepool/rbdname-old-migrated 6) Check the VM definition on the target node to make sure the pool was updated. I think we had cases where after the migration the VM was running fine but the configuration still referred to the old pool, so when we rebooted the VM it tried to find the image on the old pool again. 7) A good test to run after the migration is complete might be to create a snapshot of the RBD image on the target pool, create a clone of the VM that will boot from that snapshot (clone VM network disconnected or removed to avoid an IP conflict), and make sure it can boot successfully. I don't know if this was related, but we had two VMs around this time period that lost their boot sector or partition table or something like that. One of them I corrected by running testdisk, the other another coworker was able to quickly rebuild from a backup. This same general approach should work regardless of if the source and target images in the XML are using protocol=rbd, rbd map, mounted on iscsi, or even a different storage/replication method entirely like a qcow image, LVM partition, or DRBD. These will involve different steps to find the source image size and create the destination image, and different changes to the vmname-migrate.xml disk section. We have used this approach to migrate a VM between two Ceph clusters, for example, which involved also changing the monitor IP entries in the XML. On 4/12/23 05:12, Work Ceph wrote:
Hello guys,
We have been reading the docs, and trying to reproduce that process in our Ceph cluster. However, we always receive the following message:
```
librbd::Migration: prepare: image has watchers - not migrating
rbd: preparing migration failed: (16) Device or resource busy
```
We tested with both RDB blocks mounted with kRBD, and with librbd via KVM/qEMU system. Both cases result in the same result. For KRBD, we understood that it is not supported right now, but for librbd it seems that it should be supported somehow.
How do you guys handle those situations?
We have the following use cases that might need an image to migrate between pools while the client is still consuming it.
- RBD images that are consumed via the iSCSI gateways - RBD images mounted (rbd map) in hosts - RBD images used by KVM/Libvirt
Does Ceph support a live migration of images between pools while the clients/consumers are still using those volumes?
[1]https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/ _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Nelson Hicks Information Technology SOCKET (573) 817-0000 ext. 210 nelsonh@socket.net
participants (5)
-
Anthony D'Atri
-
Burkhard Linke
-
Eugen Block
-
Nelson Hicks
-
Work Ceph