Unable to enable RBD-Mirror Snapshot on image when VM is using RBD
This is a odd one. I don't hit it all the time so I don't think its expected behavior. Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a KVM VM. Other times I hit the following error, the only way I can get around it is to power down the KVM VM. root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument
On Fri, Jan 29, 2021 at 9:34 AM Adam Boyhan <adamb@medent.com> wrote:
This is a odd one. I don't hit it all the time so I don't think its expected behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a KVM VM. Other times I hit the following error, the only way I can get around it is to power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument
I suspect that you have the exclusive-lock feature enabled and the QEMU hypervisor host has a pre-octopus version of librbd.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
That makes sense. Appreciate it. From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Friday, January 29, 2021 9:39:28 AM Subject: Re: [ceph-users] Unable to enable RBD-Mirror Snapshot on image when VM is using RBD On Fri, Jan 29, 2021 at 9:34 AM Adam Boyhan <adamb@medent.com> wrote:
This is a odd one. I don't hit it all the time so I don't think its expected behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a KVM VM. Other times I hit the following error, the only way I can get around it is to power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument
I suspect that you have the exclusive-lock feature enabled and the QEMU hypervisor host has a pre-octopus version of librbd.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Spot on, once I upgraded the client to 15.2.8 I was able to enable rbd-mirror snapshots and create them while the VM was running. However, I have noticed that I am also able to break replication when the rbd is being used by a KVM VM. While writing data in the VM, I took a rbd snapshot, then kicked off a mirror snapshot. I took the rbd snapshot and mirror snapshot from one the osd nodes. Primary vm-101-disk-0: global_id: 68fe424e-3683-48af-ba39-2b0b4726d918 state: up+stopped description: local image is primary service: admin on Ccscephmon1 last_update: 2021-01-29 10:44:40 peer_sites: name: bunker state: up+error description: failed to copy snapshots from remote to local image last_update: 2021-01-29 10:44:50 root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-101-disk-0 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 1581 .mirror.primary.68fe424e-3683-48af-ba39-2b0b4726d918.3f9da1d2-7f23-4a75-865e-3af062be93be 2 TiB Fri Jan 29 10:28:53 2021 mirror (primary peer_uuids:[35c97c1f-a897-4635-abf6-dbd4085c89bd]) 1584 Test 2 TiB Fri Jan 29 10:37:37 2021 user 1585 .mirror.primary.68fe424e-3683-48af-ba39-2b0b4726d918.24744afa-2d9f-4958-8775-c23b0b7f383c 2 TiB Fri Jan 29 10:37:47 2021 mirror (primary peer_uuids:[35c97c1f-a897-4635-abf6-dbd4085c89bd]) Secondary vm-101-disk-0: global_id: 68fe424e-3683-48af-ba39-2b0b4726d918 state: up+error description: failed to copy snapshots from remote to local image service: admin on Bunkcephtest1 last_update: 2021-01-29 10:47:12 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-29 10:47:10 root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-101-disk-0 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 1552 Test 2 TiB Fri Jan 29 10:28:53 2021 user 1553 .mirror.non_primary.68fe424e-3683-48af-ba39-2b0b4726d918.0cd471db-7ed7-4700-8da0-8a82bc2d23a5 2 TiB Fri Jan 29 10:28:54 2021 mirror (non-primary peer_uuids:[] adcedc94-b1a1-4508-8402-db4ec0dc3085:1581 copied) I tried restarting the rbd-mirror services but no luck. I ended up issuing a resync and that got things moving until I repeated the steps again. I have the option of using KRBD but not sure if that will help in this situation. From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Friday, January 29, 2021 9:39:28 AM Subject: Re: [ceph-users] Unable to enable RBD-Mirror Snapshot on image when VM is using RBD On Fri, Jan 29, 2021 at 9:34 AM Adam Boyhan <adamb@medent.com> wrote:
This is a odd one. I don't hit it all the time so I don't think its expected behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a KVM VM. Other times I hit the following error, the only way I can get around it is to power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument
I suspect that you have the exclusive-lock feature enabled and the QEMU hypervisor host has a pre-octopus version of librbd.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Doing some testing it seems this issue only happens about 10% of the time. Haven't really correlated anything that could be the cause. A resync gets things going again. Anyway to further determine what this means? description: failed to copy snapshots from remote to local image From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Friday, January 29, 2021 11:00:01 AM Subject: [ceph-users] Re: Unable to enable RBD-Mirror Snapshot on image when VM is using RBD Spot on, once I upgraded the client to 15.2.8 I was able to enable rbd-mirror snapshots and create them while the VM was running. However, I have noticed that I am also able to break replication when the rbd is being used by a KVM VM. While writing data in the VM, I took a rbd snapshot, then kicked off a mirror snapshot. I took the rbd snapshot and mirror snapshot from one the osd nodes. Primary vm-101-disk-0: global_id: 68fe424e-3683-48af-ba39-2b0b4726d918 state: up+stopped description: local image is primary service: admin on Ccscephmon1 last_update: 2021-01-29 10:44:40 peer_sites: name: bunker state: up+error description: failed to copy snapshots from remote to local image last_update: 2021-01-29 10:44:50 root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-101-disk-0 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 1581 .mirror.primary.68fe424e-3683-48af-ba39-2b0b4726d918.3f9da1d2-7f23-4a75-865e-3af062be93be 2 TiB Fri Jan 29 10:28:53 2021 mirror (primary peer_uuids:[35c97c1f-a897-4635-abf6-dbd4085c89bd]) 1584 Test 2 TiB Fri Jan 29 10:37:37 2021 user 1585 .mirror.primary.68fe424e-3683-48af-ba39-2b0b4726d918.24744afa-2d9f-4958-8775-c23b0b7f383c 2 TiB Fri Jan 29 10:37:47 2021 mirror (primary peer_uuids:[35c97c1f-a897-4635-abf6-dbd4085c89bd]) Secondary vm-101-disk-0: global_id: 68fe424e-3683-48af-ba39-2b0b4726d918 state: up+error description: failed to copy snapshots from remote to local image service: admin on Bunkcephtest1 last_update: 2021-01-29 10:47:12 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-29 10:47:10 root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-101-disk-0 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 1552 Test 2 TiB Fri Jan 29 10:28:53 2021 user 1553 .mirror.non_primary.68fe424e-3683-48af-ba39-2b0b4726d918.0cd471db-7ed7-4700-8da0-8a82bc2d23a5 2 TiB Fri Jan 29 10:28:54 2021 mirror (non-primary peer_uuids:[] adcedc94-b1a1-4508-8402-db4ec0dc3085:1581 copied) I tried restarting the rbd-mirror services but no luck. I ended up issuing a resync and that got things moving until I repeated the steps again. I have the option of using KRBD but not sure if that will help in this situation. From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Friday, January 29, 2021 9:39:28 AM Subject: Re: [ceph-users] Unable to enable RBD-Mirror Snapshot on image when VM is using RBD On Fri, Jan 29, 2021 at 9:34 AM Adam Boyhan <adamb@medent.com> wrote:
This is a odd one. I don't hit it all the time so I don't think its expected behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a KVM VM. Other times I hit the following error, the only way I can get around it is to power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument
I suspect that you have the exclusive-lock feature enabled and the QEMU hypervisor host has a pre-octopus version of librbd.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Adam Boyhan
-
Jason Dillaman