29 Jan
2021
29 Jan
'21
8:03 p.m.
This is a odd one. I don't hit it all the time so I don't think its expected
behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a
KVM VM. Other times I hit the following error, the only way I can get around it is to
power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1
librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot:
failed to create mirror snapshot: (22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200
handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid
argument
2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot
enable mirroring: (22) Invalid argument
29 Jan
29 Jan
8:09 p.m.
On Fri, Jan 29, 2021 at 9:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
This is a odd one. I don't hit it all the time so I don't think its expected
behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by
a KVM VM. Other times I hit the following error, the only way I can get around it is to
power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1
librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot:
failed to create mirror snapshot: (22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest:
0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot:
(22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot
enable mirroring: (22) Invalid argument
I suspect that you have the exclusive-lock feature enabled and the
QEMU hypervisor host has a pre-octopus version of librbd.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
8:12 p.m.
That makes sense. Appreciate it.
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Friday, January 29, 2021 9:39:28 AM
Subject: Re: [ceph-users] Unable to enable RBD-Mirror Snapshot on image when VM is using
RBD
On Fri, Jan 29, 2021 at 9:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
This is a odd one. I don't hit it all the time so I don't think its expected
behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by
a KVM VM. Other times I hit the following error, the only way I can get around it is to
power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1
librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot:
failed to create mirror snapshot: (22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest:
0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot:
(22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot
enable mirroring: (22) Invalid argument
I suspect that you have the exclusive-lock feature enabled and the
QEMU hypervisor host has a pre-octopus version of librbd.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
9:30 p.m.
Spot on, once I upgraded the client to 15.2.8 I was able to enable rbd-mirror snapshots
and create them while the VM was running.
However, I have noticed that I am also able to break replication when the rbd is being
used by a KVM VM.
While writing data in the VM, I took a rbd snapshot, then kicked off a mirror snapshot. I
took the rbd snapshot and mirror snapshot from one the osd nodes.
Primary
vm-101-disk-0:
global_id: 68fe424e-3683-48af-ba39-2b0b4726d918
state: up+stopped
description: local image is primary
service: admin on Ccscephmon1
last_update: 2021-01-29 10:44:40
peer_sites:
name: bunker
state: up+error
description: failed to copy snapshots from remote to local image
last_update: 2021-01-29 10:44:50
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-101-disk-0
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
1581
.mirror.primary.68fe424e-3683-48af-ba39-2b0b4726d918.3f9da1d2-7f23-4a75-865e-3af062be93be
2 TiB Fri Jan 29 10:28:53 2021 mirror (primary
peer_uuids:[35c97c1f-a897-4635-abf6-dbd4085c89bd])
1584 Test 2 TiB Fri Jan 29 10:37:37 2021 user
1585
.mirror.primary.68fe424e-3683-48af-ba39-2b0b4726d918.24744afa-2d9f-4958-8775-c23b0b7f383c
2 TiB Fri Jan 29 10:37:47 2021 mirror (primary
peer_uuids:[35c97c1f-a897-4635-abf6-dbd4085c89bd])
Secondary
vm-101-disk-0:
global_id: 68fe424e-3683-48af-ba39-2b0b4726d918
state: up+error
description: failed to copy snapshots from remote to local image
service: admin on Bunkcephtest1
last_update: 2021-01-29 10:47:12
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-29 10:47:10
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-101-disk-0
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
1552 Test 2 TiB Fri Jan 29 10:28:53 2021 user
1553
.mirror.non_primary.68fe424e-3683-48af-ba39-2b0b4726d918.0cd471db-7ed7-4700-8da0-8a82bc2d23a5
2 TiB Fri Jan 29 10:28:54 2021 mirror (non-primary peer_uuids:[]
adcedc94-b1a1-4508-8402-db4ec0dc3085:1581 copied)
I tried restarting the rbd-mirror services but no luck.
I ended up issuing a resync and that got things moving until I repeated the steps again.
I have the option of using KRBD but not sure if that will help in this situation.
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Friday, January 29, 2021 9:39:28 AM
Subject: Re: [ceph-users] Unable to enable RBD-Mirror Snapshot on image when VM is using
RBD
On Fri, Jan 29, 2021 at 9:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
This is a odd one. I don't hit it all the time so I don't think its expected
behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by
a KVM VM. Other times I hit the following error, the only way I can get around it is to
power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1
librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot:
failed to create mirror snapshot: (22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest:
0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot:
(22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot
enable mirroring: (22) Invalid argument
I suspect that you have the exclusive-lock feature enabled and the
QEMU hypervisor host has a pre-octopus version of librbd.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
1 Feb
1 Feb
8:04 p.m.
Doing some testing it seems this issue only happens about 10% of the time. Haven't
really correlated anything that could be the cause.
A resync gets things going again.
Anyway to further determine what this means?
description: failed to copy snapshots from remote to local image
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Friday, January 29, 2021 11:00:01 AM
Subject: [ceph-users] Re: Unable to enable RBD-Mirror Snapshot on image when VM is using
RBD
Spot on, once I upgraded the client to 15.2.8 I was able to enable rbd-mirror snapshots
and create them while the VM was running.
However, I have noticed that I am also able to break replication when the rbd is being
used by a KVM VM.
While writing data in the VM, I took a rbd snapshot, then kicked off a mirror snapshot. I
took the rbd snapshot and mirror snapshot from one the osd nodes.
Primary
vm-101-disk-0:
global_id: 68fe424e-3683-48af-ba39-2b0b4726d918
state: up+stopped
description: local image is primary
service: admin on Ccscephmon1
last_update: 2021-01-29 10:44:40
peer_sites:
name: bunker
state: up+error
description: failed to copy snapshots from remote to local image
last_update: 2021-01-29 10:44:50
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-101-disk-0
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
1581
.mirror.primary.68fe424e-3683-48af-ba39-2b0b4726d918.3f9da1d2-7f23-4a75-865e-3af062be93be
2 TiB Fri Jan 29 10:28:53 2021 mirror (primary
peer_uuids:[35c97c1f-a897-4635-abf6-dbd4085c89bd])
1584 Test 2 TiB Fri Jan 29 10:37:37 2021 user
1585
.mirror.primary.68fe424e-3683-48af-ba39-2b0b4726d918.24744afa-2d9f-4958-8775-c23b0b7f383c
2 TiB Fri Jan 29 10:37:47 2021 mirror (primary
peer_uuids:[35c97c1f-a897-4635-abf6-dbd4085c89bd])
Secondary
vm-101-disk-0:
global_id: 68fe424e-3683-48af-ba39-2b0b4726d918
state: up+error
description: failed to copy snapshots from remote to local image
service: admin on Bunkcephtest1
last_update: 2021-01-29 10:47:12
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-29 10:47:10
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-101-disk-0
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
1552 Test 2 TiB Fri Jan 29 10:28:53 2021 user
1553
.mirror.non_primary.68fe424e-3683-48af-ba39-2b0b4726d918.0cd471db-7ed7-4700-8da0-8a82bc2d23a5
2 TiB Fri Jan 29 10:28:54 2021 mirror (non-primary peer_uuids:[]
adcedc94-b1a1-4508-8402-db4ec0dc3085:1581 copied)
I tried restarting the rbd-mirror services but no luck.
I ended up issuing a resync and that got things moving until I repeated the steps again.
I have the option of using KRBD but not sure if that will help in this situation.
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Friday, January 29, 2021 9:39:28 AM
Subject: Re: [ceph-users] Unable to enable RBD-Mirror Snapshot on image when VM is using
RBD
On Fri, Jan 29, 2021 at 9:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
This is a odd one. I don't hit it all the time so I don't think its expected
behavior.
Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by
a KVM VM. Other times I hit the following error, the only way I can get around it is to
power down the KVM VM.
root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1
librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot:
failed to create mirror snapshot: (22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest:
0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot:
(22) Invalid argument
2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot
enable mirroring: (22) Invalid argument
I suspect that you have the exclusive-lock feature enabled and the
QEMU hypervisor host has a pre-octopus version of librbd.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1180
days inactive
1183
days old
4 comments
2 participants
participants (2)
-
Adam Boyhan -
Jason Dillaman