20 Jan
2021
20 Jan
'21
8:23 p.m.
I have been doing some testing with RBD-Mirror Snapshots to a remote Ceph cluster.
Does anyone know if the images on the remote cluster can be utilized in anyway? Would love
the ability to clone them, or even readonly would be nice.
20 Jan
20 Jan
8:35 p.m.
Have you tried just using them?
(RO, if you do RW things might go crazy, would be nice to try though).
You might be able to create a clone too, and I guess worst case just cp/deep
cp.
I'm interested in your findings btw. I'd be greatful if you share them :)
Thanks!
On 01/20 14:23, Adam Boyhan wrote:
I have been doing some testing with RBD-Mirror
Snapshots to a remote Ceph cluster.
Does anyone know if the images on the remote cluster can be utilized in anyway? Would
love the ability to clone them, or even readonly would be nice.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
8:45 p.m.
Can you describe your Ceph deployment?
On Wed, Jan 20, 2021 at 11:24 AM Adam Boyhan <adamb(a)medent.com> wrote:
I have been doing some testing with RBD-Mirror
Snapshots to a remote Ceph
cluster.
Does anyone know if the images on the remote cluster can be utilized in
anyway? Would love the ability to clone them, or even readonly would be
nice.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
This e-mail and all information in, attached to, or linked via this
e-mail (together the ‘e-mail’) is confidential and may be legally
privileged. It is intended solely for the intended addressee(s). Access to,
or any onward transmission, of this e-mail by any other person is not
authorised. If you are not the intended recipient, you are requested to
immediately alert the sender of this e-mail and to immediately delete this
e-mail. Any disclosure in any form of all or part of this e-mail, or of any
the parties to it, including any copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be
unlawful.
This e-mail is not, and is not intended to be, and should
not be construed as being, (a) any offer, solicitation, or promotion of any
kind; (b) the basis of any investment or other decision(s); (c) any
recommendation to buy, sell or transact in any manner any good(s),
product(s) or service(s), nor engage in any investment(s) or other
transaction(s) or activities; or (d) the provision of, or related to, any
advisory service(s) or activities, including regarding any investment, tax,
legal, financial, accounting, consulting or any other related service(s).
8:54 p.m.
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are
the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
9 p.m.
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
Two separate 4 node clusters with 10 OSD's in each
node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:09 p.m.
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
Two separate 4 node clusters with 10 OSD's in each
node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:28 p.m.
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
Two separate 4 node clusters with 10 OSD's in
each node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:42 p.m.
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
Two separate 4 node clusters with 10 OSD's in
each node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
21 Jan
21 Jan
2:34 p.m.
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
Two separate 4 node clusters with 10 OSD's in
each node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:25 p.m.
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1
2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1
2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
Two separate 4 node clusters with 10 OSD's in
each node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:39 p.m.
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
Two separate 4 node clusters with 10 OSD's in
each node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:42 p.m.
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1
2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1
2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
Two separate 4 node clusters with 10 OSD's in
each node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:51 p.m.
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
Two separate 4 node clusters with 10 OSD's in
each node. Micron 9300
NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
white papers.
When I attempt to protect the snapshot on a remote image, it errors
with read only.
root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:01 p.m.
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME
SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper
2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1
2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1
2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> Two separate 4 node clusters with 10 OSD's in each node. Micron 9300
> NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> white papers.
>
> When I attempt to protect the snapshot on a remote image, it errors
> with read only.
>
> root@Bunkcephmon2:~# rbd snap protect
> CephTestPool1/vm-100-disk-0@TestSnapper1
> rbd: protecting snap failed: (30) Read-only file system
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
9:11 p.m.
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> Two separate 4 node clusters with 10 OSD's in each node. Micron 9300
> NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> white papers.
>
> When I attempt to protect the snapshot on a remote image, it errors
> with read only.
>
> root@Bunkcephmon2:~# rbd snap protect
> CephTestPool1/vm-100-disk-0@TestSnapper1
> rbd: protecting snap failed: (30) Read-only file system
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
22 Jan
22 Jan
7:50 p.m.
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the
larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary
array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back
in the same position.
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may
maintain read-only images and snapshots as templates in one pool, and writeable clones in
another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0
validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
From: "Eugen Block" <eblock(a)nde.ag>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:00:54 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote
cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> Two separate 4 node clusters with 10 OSD's in each node. Micron 9300
> NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> white papers.
>
> When I attempt to protect the snapshot on a remote image, it errors
> with read only.
>
> root@Bunkcephmon2:~# rbd snap protect
> CephTestPool1/vm-100-disk-0@TestSnapper1
> rbd: protecting snap failed: (30) Read-only file system
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
8:11 p.m.
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out,
the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am
back in the same position.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
>
> That's what I though as well, specially based on this.
>
>
>
> Note
>
> You may clone a snapshot from one pool to an image in another pool. For example, you
may maintain read-only images and snapshots as templates in one pool, and writeable clones
in another pool.
>
> root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest:
0x55c7cf8417f0 validate_parent: parent snapshot must be protected
>
> root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
> rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
>
> From: "Eugen Block" <eblock(a)nde.ag>
> To: "adamb" <adamb(a)medent.com>
> Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> Sent: Wednesday, January 20, 2021 3:00:54 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> But you should be able to clone the mirrored snapshot on the remote
> cluster even though it’s not protected, IIRC.
>
>
> Zitat von Adam Boyhan <adamb(a)medent.com>om>:
>
> > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300
> > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> > white papers.
> >
> > When I attempt to protect the snapshot on a remote image, it errors
> > with read only.
> >
> > root@Bunkcephmon2:~# rbd snap protect
> > CephTestPool1/vm-100-disk-0@TestSnapper1
> > rbd: protecting snap failed: (30) Read-only file system
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
--
Jason
8:32 p.m.
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out,
the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am
back in the same position.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
Primary Site
root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1
10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:42:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an
issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
root@Bunkcephmon2:~# rbd ls CephTestPool2
vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with
Ceph and NVMe.
Appreciate it!
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Wednesday, January 20, 2021 3:28:39 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
>
> That's what I though as well, specially based on this.
>
>
>
> Note
>
> You may clone a snapshot from one pool to an image in another pool. For example, you
may maintain read-only images and snapshots as templates in one pool, and writeable clones
in another pool.
>
> root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest:
0x55c7cf8417f0 validate_parent: parent snapshot must be protected
>
> root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
> rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so
that the protection status replicates or (2) utilize RBD clone v2
which doesn't require protection but does require Mimic or later
clients [1].
>
> From: "Eugen Block" <eblock(a)nde.ag>
> To: "adamb" <adamb(a)medent.com>
> Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> Sent: Wednesday, January 20, 2021 3:00:54 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> But you should be able to clone the mirrored snapshot on the remote
> cluster even though it’s not protected, IIRC.
>
>
> Zitat von Adam Boyhan <adamb(a)medent.com>om>:
>
> > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300
> > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> > white papers.
> >
> > When I attempt to protect the snapshot on a remote image, it errors
> > with read only.
> >
> > root@Bunkcephmon2:~# rbd snap protect
> > CephTestPool1/vm-100-disk-0@TestSnapper1
> > rbd: protecting snap failed: (30) Read-only file system
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
--
Jason
--
Jason
9:02 p.m.
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph
daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out,
the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am
back in the same position.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
>
> Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
> Primary Site
> root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
>
> root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Ccscephtest1:~# mount /dev/nbd0 /usr2
>
> Secondary Site
> root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
>
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
>
>
>
> ________________________________
> From: "adamb" <adamb(a)medent.com>
> To: "dillaman" <dillaman(a)redhat.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Wednesday, January 20, 2021 3:42:46 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> Awesome information. I new I had to be missing something.
>
> All of my clients will be far newer than mimic so I don't think that will be an
issue.
>
> Added the following to my ceph.conf on both clusters.
>
> rbd_default_clone_format = 2
>
> root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> root@Bunkcephmon2:~# rbd ls CephTestPool2
> vm-100-disk-0-CLONE
>
> I am sure I will be back with more questions. Hoping to replace our Nimble storage
with Ceph and NVMe.
>
> Appreciate it!
>
> ________________________________
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Wednesday, January 20, 2021 3:28:39 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > That's what I though as well, specially based on this.
> >
> >
> >
> > Note
> >
> > You may clone a snapshot from one pool to an image in another pool. For
example, you may maintain read-only images and snapshots as templates in one pool, and
writeable clones in another pool.
> >
> > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest:
0x55c7cf8417f0 validate_parent: parent snapshot must be protected
> >
> > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
> > rbd: protecting snap failed: (30) Read-only file system
>
> You have two options: (1) protect the snapshot on the primary image so
> that the protection status replicates or (2) utilize RBD clone v2
> which doesn't require protection but does require Mimic or later
> clients [1].
>
> >
> > From: "Eugen Block" <eblock(a)nde.ag>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > But you should be able to clone the mirrored snapshot on the remote
> > cluster even though it’s not protected, IIRC.
> >
> >
> > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> >
> > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300
> > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> > > white papers.
> > >
> > > When I attempt to protect the snapshot on a remote image, it errors
> > > with read only.
> > >
> > > root@Bunkcephmon2:~# rbd snap protect
> > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > rbd: protecting snap failed: (30) Read-only file system
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
>
> --
> Jason
>
--
Jason
--
Jason
9:29 p.m.
I will have to do some looking into how that is done on Proxmox, but most definitely.
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:02:23 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out,
the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am
back in the same position.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:25:11 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
>
> Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status
CephTestPool1/vm-100-disk-0")?
> Primary Site
> root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
>
> root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Ccscephtest1:~# mount /dev/nbd0 /usr2
>
> Secondary Site
> root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
>
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
>
>
>
> ________________________________
> From: "adamb" <adamb(a)medent.com>
> To: "dillaman" <dillaman(a)redhat.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Wednesday, January 20, 2021 3:42:46 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> Awesome information. I new I had to be missing something.
>
> All of my clients will be far newer than mimic so I don't think that will be an
issue.
>
> Added the following to my ceph.conf on both clusters.
>
> rbd_default_clone_format = 2
>
> root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> root@Bunkcephmon2:~# rbd ls CephTestPool2
> vm-100-disk-0-CLONE
>
> I am sure I will be back with more questions. Hoping to replace our Nimble storage
with Ceph and NVMe.
>
> Appreciate it!
>
> ________________________________
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Wednesday, January 20, 2021 3:28:39 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > That's what I though as well, specially based on this.
> >
> >
> >
> > Note
> >
> > You may clone a snapshot from one pool to an image in another pool. For
example, you may maintain read-only images and snapshots as templates in one pool, and
writeable clones in another pool.
> >
> > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest:
0x55c7cf8417f0 validate_parent: parent snapshot must be protected
> >
> > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1
> > rbd: protecting snap failed: (30) Read-only file system
>
> You have two options: (1) protect the snapshot on the primary image so
> that the protection status replicates or (2) utilize RBD clone v2
> which doesn't require protection but does require Mimic or later
> clients [1].
>
> >
> > From: "Eugen Block" <eblock(a)nde.ag>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > But you should be able to clone the mirrored snapshot on the remote
> > cluster even though it’s not protected, IIRC.
> >
> >
> > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> >
> > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300
> > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> > > white papers.
> > >
> > > When I attempt to protect the snapshot on a remote image, it errors
> > > with read only.
> > >
> > > root@Bunkcephmon2:~# rbd snap protect
> > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > rbd: protecting snap failed: (30) Read-only file system
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
>
> --
> Jason
>
--
Jason
--
Jason
9:44 p.m.
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb(a)medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:02:23 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph
daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out,
the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am
back in the same position.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> After the resync finished. I can mount it now.
>
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
>
> Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
> ________________________________
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 9:25:11 AM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > When cloning the snapshot on the remote cluster I can't see my ext4
filesystem.
> >
> > Using the same exact snapshot on both sides. Shouldn't this be consistent?
>
> Yes. Has the replication process completed ("rbd mirror image status
> CephTestPool1/vm-100-disk-0")?
>
> > Primary Site
> > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
> >
> > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> >
> > Secondary Site
> > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
> >
> > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
> >
> >
> >
> > ________________________________
> > From: "adamb" <adamb(a)medent.com>
> > To: "dillaman" <dillaman(a)redhat.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Wednesday, January 20, 2021 3:42:46 PM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > Awesome information. I new I had to be missing something.
> >
> > All of my clients will be far newer than mimic so I don't think that will
be an issue.
> >
> > Added the following to my ceph.conf on both clusters.
> >
> > rbd_default_clone_format = 2
> >
> > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > root@Bunkcephmon2:~# rbd ls CephTestPool2
> > vm-100-disk-0-CLONE
> >
> > I am sure I will be back with more questions. Hoping to replace our Nimble
storage with Ceph and NVMe.
> >
> > Appreciate it!
> >
> > ________________________________
> > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Wednesday, January 20, 2021 3:28:39 PM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
> > >
> > > That's what I though as well, specially based on this.
> > >
> > >
> > >
> > > Note
> > >
> > > You may clone a snapshot from one pool to an image in another pool. For
example, you may maintain read-only images and snapshots as templates in one pool, and
writeable clones in another pool.
> > >
> > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest:
0x55c7cf8417f0 validate_parent: parent snapshot must be protected
> > >
> > > root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
> > > rbd: protecting snap failed: (30) Read-only file system
> >
> > You have two options: (1) protect the snapshot on the primary image so
> > that the protection status replicates or (2) utilize RBD clone v2
> > which doesn't require protection but does require Mimic or later
> > clients [1].
> >
> > >
> > > From: "Eugen Block" <eblock(a)nde.ag>
> > > To: "adamb" <adamb(a)medent.com>
> > > Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt
Wilder" <matt.wilder(a)bitmex.com>
> > > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > >
> > > But you should be able to clone the mirrored snapshot on the remote
> > > cluster even though it’s not protected, IIRC.
> > >
> > >
> > > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> > >
> > > > Two separate 4 node clusters with 10 OSD's in each node. Micron
9300
> > > > NVMe's are the OSD drives. Heavily based on the
Micron/Supermicro
> > > > white papers.
> > > >
> > > > When I attempt to protect the snapshot on a remote image, it errors
> > > > with read only.
> > > >
> > > > root@Bunkcephmon2:~# rbd snap protect
> > > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > rbd: protecting snap failed: (30) Read-only file system
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
> >
> > --
> > Jason
> >
>
>
> --
> Jason
--
Jason
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
26 Jan
26 Jan
7:51 p.m.
Did some testing with clients running 16.1. I setup two different clients, each one
dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1
build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:44:26 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb(a)medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:02:23 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out,
the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am
back in the same position.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> After the resync finished. I can mount it now.
>
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
>
> Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
> ________________________________
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 9:25:11 AM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > When cloning the snapshot on the remote cluster I can't see my ext4
filesystem.
> >
> > Using the same exact snapshot on both sides. Shouldn't this be consistent?
>
> Yes. Has the replication process completed ("rbd mirror image status
> CephTestPool1/vm-100-disk-0")?
>
> > Primary Site
> > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
> >
> > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> >
> > Secondary Site
> > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
> >
> > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
> >
> >
> >
> > ________________________________
> > From: "adamb" <adamb(a)medent.com>
> > To: "dillaman" <dillaman(a)redhat.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Wednesday, January 20, 2021 3:42:46 PM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > Awesome information. I new I had to be missing something.
> >
> > All of my clients will be far newer than mimic so I don't think that will
be an issue.
> >
> > Added the following to my ceph.conf on both clusters.
> >
> > rbd_default_clone_format = 2
> >
> > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > root@Bunkcephmon2:~# rbd ls CephTestPool2
> > vm-100-disk-0-CLONE
> >
> > I am sure I will be back with more questions. Hoping to replace our Nimble
storage with Ceph and NVMe.
> >
> > Appreciate it!
> >
> > ________________________________
> > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Wednesday, January 20, 2021 3:28:39 PM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
> > >
> > > That's what I though as well, specially based on this.
> > >
> > >
> > >
> > > Note
> > >
> > > You may clone a snapshot from one pool to an image in another pool. For
example, you may maintain read-only images and snapshots as templates in one pool, and
writeable clones in another pool.
> > >
> > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest:
0x55c7cf8417f0 validate_parent: parent snapshot must be protected
> > >
> > > root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
> > > rbd: protecting snap failed: (30) Read-only file system
> >
> > You have two options: (1) protect the snapshot on the primary image so
> > that the protection status replicates or (2) utilize RBD clone v2
> > which doesn't require protection but does require Mimic or later
> > clients [1].
> >
> > >
> > > From: "Eugen Block" <eblock(a)nde.ag>
> > > To: "adamb" <adamb(a)medent.com>
> > > Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt
Wilder" <matt.wilder(a)bitmex.com>
> > > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > >
> > > But you should be able to clone the mirrored snapshot on the remote
> > > cluster even though it’s not protected, IIRC.
> > >
> > >
> > > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> > >
> > > > Two separate 4 node clusters with 10 OSD's in each node. Micron
9300
> > > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> > > > white papers.
> > > >
> > > > When I attempt to protect the snapshot on a remote image, it errors
> > > > with read only.
> > > >
> > > > root@Bunkcephmon2:~# rbd snap protect
> > > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > rbd: protecting snap failed: (30) Read-only file system
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
> >
> > --
> > Jason
> >
>
>
> --
> Jason
--
Jason
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
27 Jan
27 Jan
1:27 p.m.
Doing some more testing.
I can demote the rbd image on the primary, promote on the secondary and the image looks
great. I can map it, mount it, and it looks just like it should.
However, the rbd snapshots are still unusable on the secondary even when promoted. I went
as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd snapshot
still won't work either. Both snapshots look and work great on the primary.
If I request a resync from the secondary the rbd snapshots start working just like the
primary, but only if I request a resync.
I get the same exact results whether I am using a 15.2.8 or 16.1 client.
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Tuesday, January 26, 2021 1:51:13 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one
dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1
build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:44:26 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb(a)medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:02:23 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out,
the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am
back in the same position.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1
- Allowed the next RBD-Mirror scheduled snap to complete
- At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 11:46:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring
/etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
/dev/nbd0
root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 9:42:26 AM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> After the resync finished. I can mount it now.
>
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
>
> Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as
a first step, but perhaps there are some extra guardrails we can put
on the system to prevent premature usage if the sync status doesn't
indicate that it's complete.
> ________________________________
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 9:25:11 AM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > When cloning the snapshot on the remote cluster I can't see my ext4
filesystem.
> >
> > Using the same exact snapshot on both sides. Shouldn't this be consistent?
>
> Yes. Has the replication process completed ("rbd mirror image status
> CephTestPool1/vm-100-disk-0")?
>
> > Primary Site
> > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
> >
> > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> >
> > Secondary Site
> > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
> >
> > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
> >
> >
> >
> > ________________________________
> > From: "adamb" <adamb(a)medent.com>
> > To: "dillaman" <dillaman(a)redhat.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Wednesday, January 20, 2021 3:42:46 PM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > Awesome information. I new I had to be missing something.
> >
> > All of my clients will be far newer than mimic so I don't think that will
be an issue.
> >
> > Added the following to my ceph.conf on both clusters.
> >
> > rbd_default_clone_format = 2
> >
> > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > root@Bunkcephmon2:~# rbd ls CephTestPool2
> > vm-100-disk-0-CLONE
> >
> > I am sure I will be back with more questions. Hoping to replace our Nimble
storage with Ceph and NVMe.
> >
> > Appreciate it!
> >
> > ________________________________
> > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Wednesday, January 20, 2021 3:28:39 PM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com> wrote:
> > >
> > > That's what I though as well, specially based on this.
> > >
> > >
> > >
> > > Note
> > >
> > > You may clone a snapshot from one pool to an image in another pool. For
example, you may maintain read-only images and snapshots as templates in one pool, and
writeable clones in another pool.
> > >
> > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest:
0x55c7cf8417f0 validate_parent: parent snapshot must be protected
> > >
> > > root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
> > > rbd: protecting snap failed: (30) Read-only file system
> >
> > You have two options: (1) protect the snapshot on the primary image so
> > that the protection status replicates or (2) utilize RBD clone v2
> > which doesn't require protection but does require Mimic or later
> > clients [1].
> >
> > >
> > > From: "Eugen Block" <eblock(a)nde.ag>
> > > To: "adamb" <adamb(a)medent.com>
> > > Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt
Wilder" <matt.wilder(a)bitmex.com>
> > > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > >
> > > But you should be able to clone the mirrored snapshot on the remote
> > > cluster even though it’s not protected, IIRC.
> > >
> > >
> > > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> > >
> > > > Two separate 4 node clusters with 10 OSD's in each node. Micron
9300
> > > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro
> > > > white papers.
> > > >
> > > > When I attempt to protect the snapshot on a remote image, it errors
> > > > with read only.
> > > >
> > > > root@Bunkcephmon2:~# rbd snap protect
> > > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > rbd: protecting snap failed: (30) Read-only file system
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
> >
> > --
> > Jason
> >
>
>
> --
> Jason
--
Jason
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
28 Jan
28 Jan
4:31 p.m.
On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb(a)medent.com> wrote:
Doing some more testing.
I can demote the rbd image on the primary, promote on the secondary and the image looks
great. I can map it, mount it, and it looks just like it should.
However, the rbd snapshots are still unusable on the secondary even when promoted. I
went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd
snapshot still won't work either. Both snapshots look and work great on the primary.
If I request a resync from the secondary the rbd snapshots start working just like the
primary, but only if I request a resync.
I get the same exact results whether I am using a 15.2.8 or 16.1 client.
Thanks. I've opened a tracker ticket
(https://tracker.ceph.com/issues/49037) for this.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Tuesday, January 26, 2021 1:51:13 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one
dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1
build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific
(rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific
(rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:44:26 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:02:23 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out,
the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also
had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and
also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect
during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am
back in the same position.
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:11:31 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
vm-100-disk-1:
global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
state: up+replaying
description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
service: admin on Bunkcephmon1
last_update: 2021-01-21 15:06:24
peer_sites:
name: ccs
state: up+stopped
description: local image is primary
last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Bunkcephtest1:~# blkid /dev/nbd0
root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or
helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
/dev/nbd0
root@Ccscephtest1:~# blkid /dev/nbd0
/dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4"
root@Ccscephtest1:~# mount /dev/nbd0 /usr2
root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
/dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
Sent: Thursday, January 21, 2021 3:01:46 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> I was able to trigger the issue again.
>
> - On the primary I created a snap called TestSnapper for disk vm-100-disk-1
> - Allowed the next RBD-Mirror scheduled snap to complete
> - At this point the snapshot is showing up on the remote side.
>
> root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
> vm-100-disk-1:
> global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
> state: up+replaying
> description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
> service: admin on Bunkcephmon1
> last_update: 2021-01-21 11:46:24
> peer_sites:
> name: ccs
> state: up+stopped
> description: local image is primary
> last_update: 2021-01-21 11:46:28
>
> root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
> SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
> 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
> 11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
>
> Seems like the sync is complete, So I then clone it, map it and attempt to mount
it.
Can you run "snap ls --all" on the non-primary cluster? The
non-primary snapshot will list its status. On my cluster (with a much
smaller image):
#
# CLUSTER 1
#
$ rbd --cluster cluster1 create --size 1G mirror/image1
$ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
Mirroring enabled
$ rbd --cluster cluster1 device map -t nbd mirror/image1
/dev/nbd0
$ mkfs.ext4 /dev/nbd0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
$ blkid /dev/nbd0
/dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ rbd --cluster cluster1 snap create mirror/image1@fs
Creating snap: 100% complete...done.
$ rbd --cluster cluster1 mirror image snapshot mirror/image1
Snapshot ID: 6
$ rbd --cluster cluster1 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:24 2021
user
6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
#
# CLUSTER 2
#
$ rbd --cluster cluster2 mirror image status mirror/image1
image1:
global_id: f9f692b8-2405-416c-9247-5628e303947a
state: up+replaying
description: replaying,
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
service: mirror.0 on cube-1
last_update: 2021-01-21 14:51:18
peer_sites:
name: cluster1
state: up+stopped
description: local image is primary
last_update: 2021-01-21 14:51:27
$ rbd --cluster cluster2 snap ls --all mirror/image1
SNAPID NAME
SIZE PROTECTED TIMESTAMP
NAMESPACE
5 fs
1 GiB Thu Jan 21 14:50:52
2021 user
6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
$ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
mirror/image1@fs mirror/image2
$ rbd --cluster cluster2 device map -t nbd mirror/image2
/dev/nbd1
$ blkid /dev/nbd1
/dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
BLOCK_SIZE="4096" TYPE="ext4"
$ mount /dev/nbd1 /mnt/
$ mount | grep nbd
/dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
>
> On the primary still no issues
>
> root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
> root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id
admin --keyring /etc/ceph/ceph.client.admin.keyring
> /dev/nbd0
> root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
>
>
>
>
>
> ________________________________
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 9:42:26 AM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > After the resync finished. I can mount it now.
> >
> > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> >
> > Makes me a bit nervous how it got into that position and everything appeared
ok.
>
> We unfortunately need to create the snapshots that are being synced as
> a first step, but perhaps there are some extra guardrails we can put
> on the system to prevent premature usage if the sync status doesn't
> indicate that it's complete.
>
> > ________________________________
> > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Thursday, January 21, 2021 9:25:11 AM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com> wrote:
> > >
> > > When cloning the snapshot on the remote cluster I can't see my ext4
filesystem.
> > >
> > > Using the same exact snapshot on both sides. Shouldn't this be
consistent?
> >
> > Yes. Has the replication process completed ("rbd mirror image status
> > CephTestPool1/vm-100-disk-0")?
> >
> > > Primary Site
> > > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> > > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
> > >
> > > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id
admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > /dev/nbd0
> > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> > >
> > > Secondary Site
> > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep
TestSnapper1
> > > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
> > >
> > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id
admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > /dev/nbd0
> > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0,
missing codepage or helper program, or other error.
> > >
> > >
> > >
> > > ________________________________
> > > From: "adamb" <adamb(a)medent.com>
> > > To: "dillaman" <dillaman(a)redhat.com>
> > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > > Sent: Wednesday, January 20, 2021 3:42:46 PM
> > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > >
> > > Awesome information. I new I had to be missing something.
> > >
> > > All of my clients will be far newer than mimic so I don't think that
will be an issue.
> > >
> > > Added the following to my ceph.conf on both clusters.
> > >
> > > rbd_default_clone_format = 2
> > >
> > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool2/vm-100-disk-0-CLONE
> > > root@Bunkcephmon2:~# rbd ls CephTestPool2
> > > vm-100-disk-0-CLONE
> > >
> > > I am sure I will be back with more questions. Hoping to replace our Nimble
storage with Ceph and NVMe.
> > >
> > > Appreciate it!
> > >
> > > ________________________________
> > > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > > To: "adamb" <adamb(a)medent.com>
> > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > > Sent: Wednesday, January 20, 2021 3:28:39 PM
> > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > >
> > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com>
wrote:
> > > >
> > > > That's what I though as well, specially based on this.
> > > >
> > > >
> > > >
> > > > Note
> > > >
> > > > You may clone a snapshot from one pool to an image in another pool.
For example, you may maintain read-only images and snapshots as templates in one pool, and
writeable clones in another pool.
> > > >
> > > > root@Bunkcephmon2:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE
> > > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1
librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be
protected
> > > >
> > > > root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > rbd: protecting snap failed: (30) Read-only file system
> > >
> > > You have two options: (1) protect the snapshot on the primary image so
> > > that the protection status replicates or (2) utilize RBD clone v2
> > > which doesn't require protection but does require Mimic or later
> > > clients [1].
> > >
> > > >
> > > > From: "Eugen Block" <eblock(a)nde.ag>
> > > > To: "adamb" <adamb(a)medent.com>
> > > > Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt
Wilder" <matt.wilder(a)bitmex.com>
> > > > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > > >
> > > > But you should be able to clone the mirrored snapshot on the remote
> > > > cluster even though it’s not protected, IIRC.
> > > >
> > > >
> > > > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> > > >
> > > > > Two separate 4 node clusters with 10 OSD's in each node.
Micron 9300
> > > > > NVMe's are the OSD drives. Heavily based on the
Micron/Supermicro
> > > > > white papers.
> > > > >
> > > > > When I attempt to protect the snapshot on a remote image, it
errors
> > > > > with read only.
> > > > >
> > > > > root@Bunkcephmon2:~# rbd snap protect
> > > > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > > rbd: protecting snap failed: (30) Read-only file system
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > >
> > > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
> > >
> > > --
> > > Jason
> > >
> >
> >
> > --
> > Jason
>
>
>
> --
> Jason
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
6:53 p.m.
On Thu, Jan 28, 2021 at 10:31 AM Jason Dillaman <jdillama(a)redhat.com> wrote:
On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> Doing some more testing.
>
> I can demote the rbd image on the primary, promote on the secondary and the image
looks great. I can map it, mount it, and it looks just like it should.
>
> However, the rbd snapshots are still unusable on the secondary even when promoted.
I went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd
snapshot still won't work either. Both snapshots look and work great on the primary.
>
> If I request a resync from the secondary the rbd snapshots start working just like
the primary, but only if I request a resync.
>
> I get the same exact results whether I am using a 15.2.8 or 16.1 client.
Any chance I can get you to retry your test w/ the object-map /
fast-diff disabled on the primary image (the features will be
propagated to the non-primary image on create / next mirror snapshot)?
I was able to recreate the issue on when the object-map is enabled and
I'll hopefully get a fix for that issue up for review today. I just
want to verify that it is the same issue that you are hitting.
Thanks. I've opened a tracker ticket
(https://tracker.ceph.com/issues/49037) for this.
--
Jason
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Tuesday, January 26, 2021 1:51:13 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one
dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1
build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific
(rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific
(rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:44:26 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:02:23 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
>
> I have been doing a lot of testing.
>
> The size of the RBD image doesn't have any effect.
>
> I run into the issue once I actually write data to the rbd. The more data I write
out, the larger the chance of reproducing the issue.
>
> I seem to hit the issue of missing the filesystem all together the most, but I have
also had a few instances where some of the data was simply missing.
>
> I monitor the mirror status on the remote cluster until the snapshot is 100% copied
and also make sure all the IO is done. My setup has no issue maxing out my 10G
interconnect during replication, so its pretty obvious once its done.
>
> The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
>
> I can then map the rbd on the primary, write more data to it, snap it again, and I
am back in the same position.
>
> ________________________________
> From: "adamb" <adamb(a)medent.com>
> To: "dillaman" <dillaman(a)redhat.com>
> Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 3:11:31 PM
> Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> Sure thing.
>
> root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
> SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
> 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
> 12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
>
>
> root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
> vm-100-disk-1:
> global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
> state: up+replaying
> description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
> service: admin on Bunkcephmon1
> last_update: 2021-01-21 15:06:24
> peer_sites:
> name: ccs
> state: up+stopped
> description: local image is primary
> last_update: 2021-01-21 15:06:23
>
>
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
> /dev/nbd0
> root@Bunkcephtest1:~# blkid /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
>
>
> Primary still looks good.
>
> root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
> root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
> /dev/nbd0
> root@Ccscephtest1:~# blkid /dev/nbd0
> /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d"
TYPE="ext4"
> root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
> /dev/nbd0 /usr2 ext4 rw,relatime 0 0
>
>
>
>
>
>
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 3:01:46 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > I was able to trigger the issue again.
> >
> > - On the primary I created a snap called TestSnapper for disk vm-100-disk-1
> > - Allowed the next RBD-Mirror scheduled snap to complete
> > - At this point the snapshot is showing up on the remote side.
> >
> > root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
> > vm-100-disk-1:
> > global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
> > state: up+replaying
> > description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
> > service: admin on Bunkcephmon1
> > last_update: 2021-01-21 11:46:24
> > peer_sites:
> > name: ccs
> > state: up+stopped
> > description: local image is primary
> > last_update: 2021-01-21 11:46:28
> >
> > root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
> > SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
> > 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
> > 11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
> >
> > Seems like the sync is complete, So I then clone it, map it and attempt to
mount it.
>
> Can you run "snap ls --all" on the non-primary cluster? The
> non-primary snapshot will list its status. On my cluster (with a much
> smaller image):
>
> #
> # CLUSTER 1
> #
> $ rbd --cluster cluster1 create --size 1G mirror/image1
> $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
> Mirroring enabled
> $ rbd --cluster cluster1 device map -t nbd mirror/image1
> /dev/nbd0
> $ mkfs.ext4 /dev/nbd0
> mke2fs 1.45.5 (07-Jan-2020)
> Discarding device blocks: done
> Creating filesystem with 262144 4k blocks and 65536 inodes
> Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
> Superblock backups stored on blocks:
> 32768, 98304, 163840, 229376
>
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (8192 blocks): done
> Writing superblocks and filesystem accounting information: done
> $ blkid /dev/nbd0
> /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
> BLOCK_SIZE="4096" TYPE="ext4"
> $ rbd --cluster cluster1 snap create mirror/image1@fs
> Creating snap: 100% complete...done.
> $ rbd --cluster cluster1 mirror image snapshot mirror/image1
> Snapshot ID: 6
> $ rbd --cluster cluster1 snap ls --all mirror/image1
> SNAPID NAME
> SIZE PROTECTED TIMESTAMP
> NAMESPACE
> 5 fs
> 1 GiB Thu Jan 21 14:50:24 2021
> user
> 6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
> 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
> peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
>
> #
> # CLUSTER 2
> #
>
> $ rbd --cluster cluster2 mirror image status mirror/image1
> image1:
> global_id: f9f692b8-2405-416c-9247-5628e303947a
> state: up+replaying
> description: replaying,
>
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
> service: mirror.0 on cube-1
> last_update: 2021-01-21 14:51:18
> peer_sites:
> name: cluster1
> state: up+stopped
> description: local image is primary
> last_update: 2021-01-21 14:51:27
> $ rbd --cluster cluster2 snap ls --all mirror/image1
> SNAPID NAME
> SIZE PROTECTED TIMESTAMP
> NAMESPACE
> 5 fs
> 1 GiB Thu Jan 21 14:50:52
> 2021 user
> 6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
> 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
> peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
> $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
> mirror/image1@fs mirror/image2
> $ rbd --cluster cluster2 device map -t nbd mirror/image2
> /dev/nbd1
> $ blkid /dev/nbd1
> /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
> BLOCK_SIZE="4096" TYPE="ext4"
> $ mount /dev/nbd1 /mnt/
> $ mount | grep nbd
> /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
>
> > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
> > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
> >
> > On the primary still no issues
> >
> > root@Ccscephtest1:/etc/pve/priv# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
> > root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
> >
> >
> >
> >
> >
> > ________________________________
> > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Thursday, January 21, 2021 9:42:26 AM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
> > >
> > > After the resync finished. I can mount it now.
> > >
> > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id
admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > /dev/nbd0
> > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > >
> > > Makes me a bit nervous how it got into that position and everything
appeared ok.
> >
> > We unfortunately need to create the snapshots that are being synced as
> > a first step, but perhaps there are some extra guardrails we can put
> > on the system to prevent premature usage if the sync status doesn't
> > indicate that it's complete.
> >
> > > ________________________________
> > > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > > To: "adamb" <adamb(a)medent.com>
> > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > > Sent: Thursday, January 21, 2021 9:25:11 AM
> > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > >
> > > On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com>
wrote:
> > > >
> > > > When cloning the snapshot on the remote cluster I can't see my
ext4 filesystem.
> > > >
> > > > Using the same exact snapshot on both sides. Shouldn't this be
consistent?
> > >
> > > Yes. Has the replication process completed ("rbd mirror image status
> > > CephTestPool1/vm-100-disk-0")?
> > >
> > > > Primary Site
> > > > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 |
grep TestSnapper1
> > > > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
> > > >
> > > > root@Ccscephtest1:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE
> > > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > > /dev/nbd0
> > > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> > > >
> > > > Secondary Site
> > > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 |
grep TestSnapper1
> > > > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
> > > >
> > > > root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE
> > > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > > /dev/nbd0
> > > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0,
missing codepage or helper program, or other error.
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: "adamb" <adamb(a)medent.com>
> > > > To: "dillaman" <dillaman(a)redhat.com>
> > > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>,
"ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> > > > Sent: Wednesday, January 20, 2021 3:42:46 PM
> > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > > >
> > > > Awesome information. I new I had to be missing something.
> > > >
> > > > All of my clients will be far newer than mimic so I don't think
that will be an issue.
> > > >
> > > > Added the following to my ceph.conf on both clusters.
> > > >
> > > > rbd_default_clone_format = 2
> > > >
> > > > root@Bunkcephmon2:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE
> > > > root@Bunkcephmon2:~# rbd ls CephTestPool2
> > > > vm-100-disk-0-CLONE
> > > >
> > > > I am sure I will be back with more questions. Hoping to replace our
Nimble storage with Ceph and NVMe.
> > > >
> > > > Appreciate it!
> > > >
> > > > ________________________________
> > > > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > > > To: "adamb" <adamb(a)medent.com>
> > > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>,
"ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> > > > Sent: Wednesday, January 20, 2021 3:28:39 PM
> > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > > >
> > > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com>
wrote:
> > > > >
> > > > > That's what I though as well, specially based on this.
> > > > >
> > > > >
> > > > >
> > > > > Note
> > > > >
> > > > > You may clone a snapshot from one pool to an image in another
pool. For example, you may maintain read-only images and snapshots as templates in one
pool, and writeable clones in another pool.
> > > > >
> > > > > root@Bunkcephmon2:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE
> > > > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1
librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be
protected
> > > > >
> > > > > root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > > rbd: protecting snap failed: (30) Read-only file system
> > > >
> > > > You have two options: (1) protect the snapshot on the primary image
so
> > > > that the protection status replicates or (2) utilize RBD clone v2
> > > > which doesn't require protection but does require Mimic or later
> > > > clients [1].
> > > >
> > > > >
> > > > > From: "Eugen Block" <eblock(a)nde.ag>
> > > > > To: "adamb" <adamb(a)medent.com>
> > > > > Cc: "ceph-users" <ceph-users(a)ceph.io>io>,
"Matt Wilder" <matt.wilder(a)bitmex.com>
> > > > > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image
Uses
> > > > >
> > > > > But you should be able to clone the mirrored snapshot on the
remote
> > > > > cluster even though it’s not protected, IIRC.
> > > > >
> > > > >
> > > > > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> > > > >
> > > > > > Two separate 4 node clusters with 10 OSD's in each
node. Micron 9300
> > > > > > NVMe's are the OSD drives. Heavily based on the
Micron/Supermicro
> > > > > > white papers.
> > > > > >
> > > > > > When I attempt to protect the snapshot on a remote image,
it errors
> > > > > > with read only.
> > > > > >
> > > > > > root@Bunkcephmon2:~# rbd snap protect
> > > > > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > > > rbd: protecting snap failed: (30) Read-only file system
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > >
> > > > [1]
https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
> > > >
> > > > --
> > > > Jason
> > > >
> > >
> > >
> > > --
> > > Jason
> >
> >
> >
> > --
> > Jason
>
>
>
> --
> Jason
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
--
Jason
9:37 p.m.
Went through the process like 4-5 times now and its looking good. I am going to continue
beating on it to make sure. I will report back tomorrow.
Nice catch!
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 28, 2021 12:53:50 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 28, 2021 at 10:31 AM Jason Dillaman <jdillama(a)redhat.com> wrote:
On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> Doing some more testing.
>
> I can demote the rbd image on the primary, promote on the secondary and the image
looks great. I can map it, mount it, and it looks just like it should.
>
> However, the rbd snapshots are still unusable on the secondary even when promoted. I
went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd
snapshot still won't work either. Both snapshots look and work great on the primary.
>
> If I request a resync from the secondary the rbd snapshots start working just like
the primary, but only if I request a resync.
>
> I get the same exact results whether I am using a 15.2.8 or 16.1 client.
Any chance I can get you to retry your test w/ the object-map /
fast-diff disabled on the primary image (the features will be
propagated to the non-primary image on create / next mirror snapshot)?
I was able to recreate the issue on when the object-map is enabled and
I'll hopefully get a fix for that issue up for review today. I just
want to verify that it is the same issue that you are hitting.
Thanks. I've opened a tracker ticket
(https://tracker.ceph.com/issues/49037) for this.
--
Jason
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Tuesday, January 26, 2021 1:51:13 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one
dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1
build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:44:26 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:02:23 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
>
> I have been doing a lot of testing.
>
> The size of the RBD image doesn't have any effect.
>
> I run into the issue once I actually write data to the rbd. The more data I write
out, the larger the chance of reproducing the issue.
>
> I seem to hit the issue of missing the filesystem all together the most, but I have
also had a few instances where some of the data was simply missing.
>
> I monitor the mirror status on the remote cluster until the snapshot is 100% copied
and also make sure all the IO is done. My setup has no issue maxing out my 10G
interconnect during replication, so its pretty obvious once its done.
>
> The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
>
> I can then map the rbd on the primary, write more data to it, snap it again, and I
am back in the same position.
>
> ________________________________
> From: "adamb" <adamb(a)medent.com>
> To: "dillaman" <dillaman(a)redhat.com>
> Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 3:11:31 PM
> Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> Sure thing.
>
> root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
> SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
> 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
> 12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
>
>
> root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
> vm-100-disk-1:
> global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
> state: up+replaying
> description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
> service: admin on Bunkcephmon1
> last_update: 2021-01-21 15:06:24
> peer_sites:
> name: ccs
> state: up+stopped
> description: local image is primary
> last_update: 2021-01-21 15:06:23
>
>
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
> /dev/nbd0
> root@Bunkcephtest1:~# blkid /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
>
>
> Primary still looks good.
>
> root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
> root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
> /dev/nbd0
> root@Ccscephtest1:~# blkid /dev/nbd0
> /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d"
TYPE="ext4"
> root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
> /dev/nbd0 /usr2 ext4 rw,relatime 0 0
>
>
>
>
>
>
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 3:01:46 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > I was able to trigger the issue again.
> >
> > - On the primary I created a snap called TestSnapper for disk vm-100-disk-1
> > - Allowed the next RBD-Mirror scheduled snap to complete
> > - At this point the snapshot is showing up on the remote side.
> >
> > root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
> > vm-100-disk-1:
> > global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
> > state: up+replaying
> > description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
> > service: admin on Bunkcephmon1
> > last_update: 2021-01-21 11:46:24
> > peer_sites:
> > name: ccs
> > state: up+stopped
> > description: local image is primary
> > last_update: 2021-01-21 11:46:28
> >
> > root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
> > SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
> > 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
> > 11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
> >
> > Seems like the sync is complete, So I then clone it, map it and attempt to
mount it.
>
> Can you run "snap ls --all" on the non-primary cluster? The
> non-primary snapshot will list its status. On my cluster (with a much
> smaller image):
>
> #
> # CLUSTER 1
> #
> $ rbd --cluster cluster1 create --size 1G mirror/image1
> $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
> Mirroring enabled
> $ rbd --cluster cluster1 device map -t nbd mirror/image1
> /dev/nbd0
> $ mkfs.ext4 /dev/nbd0
> mke2fs 1.45.5 (07-Jan-2020)
> Discarding device blocks: done
> Creating filesystem with 262144 4k blocks and 65536 inodes
> Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
> Superblock backups stored on blocks:
> 32768, 98304, 163840, 229376
>
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (8192 blocks): done
> Writing superblocks and filesystem accounting information: done
> $ blkid /dev/nbd0
> /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
> BLOCK_SIZE="4096" TYPE="ext4"
> $ rbd --cluster cluster1 snap create mirror/image1@fs
> Creating snap: 100% complete...done.
> $ rbd --cluster cluster1 mirror image snapshot mirror/image1
> Snapshot ID: 6
> $ rbd --cluster cluster1 snap ls --all mirror/image1
> SNAPID NAME
> SIZE PROTECTED TIMESTAMP
> NAMESPACE
> 5 fs
> 1 GiB Thu Jan 21 14:50:24 2021
> user
> 6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
> 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
> peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
>
> #
> # CLUSTER 2
> #
>
> $ rbd --cluster cluster2 mirror image status mirror/image1
> image1:
> global_id: f9f692b8-2405-416c-9247-5628e303947a
> state: up+replaying
> description: replaying,
>
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
> service: mirror.0 on cube-1
> last_update: 2021-01-21 14:51:18
> peer_sites:
> name: cluster1
> state: up+stopped
> description: local image is primary
> last_update: 2021-01-21 14:51:27
> $ rbd --cluster cluster2 snap ls --all mirror/image1
> SNAPID NAME
> SIZE PROTECTED TIMESTAMP
> NAMESPACE
> 5 fs
> 1 GiB Thu Jan 21 14:50:52
> 2021 user
> 6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
> 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
> peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
> $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
> mirror/image1@fs mirror/image2
> $ rbd --cluster cluster2 device map -t nbd mirror/image2
> /dev/nbd1
> $ blkid /dev/nbd1
> /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
> BLOCK_SIZE="4096" TYPE="ext4"
> $ mount /dev/nbd1 /mnt/
> $ mount | grep nbd
> /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
>
> > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
> > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
> >
> > On the primary still no issues
> >
> > root@Ccscephtest1:/etc/pve/priv# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
> > root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
> >
> >
> >
> >
> >
> > ________________________________
> > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Thursday, January 21, 2021 9:42:26 AM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
> > >
> > > After the resync finished. I can mount it now.
> > >
> > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id
admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > /dev/nbd0
> > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > >
> > > Makes me a bit nervous how it got into that position and everything
appeared ok.
> >
> > We unfortunately need to create the snapshots that are being synced as
> > a first step, but perhaps there are some extra guardrails we can put
> > on the system to prevent premature usage if the sync status doesn't
> > indicate that it's complete.
> >
> > > ________________________________
> > > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > > To: "adamb" <adamb(a)medent.com>
> > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > > Sent: Thursday, January 21, 2021 9:25:11 AM
> > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > >
> > > On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com>
wrote:
> > > >
> > > > When cloning the snapshot on the remote cluster I can't see my
ext4 filesystem.
> > > >
> > > > Using the same exact snapshot on both sides. Shouldn't this be
consistent?
> > >
> > > Yes. Has the replication process completed ("rbd mirror image status
> > > CephTestPool1/vm-100-disk-0")?
> > >
> > > > Primary Site
> > > > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 |
grep TestSnapper1
> > > > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
> > > >
> > > > root@Ccscephtest1:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE
> > > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > > /dev/nbd0
> > > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> > > >
> > > > Secondary Site
> > > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 |
grep TestSnapper1
> > > > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
> > > >
> > > > root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE
> > > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > > /dev/nbd0
> > > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0,
missing codepage or helper program, or other error.
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: "adamb" <adamb(a)medent.com>
> > > > To: "dillaman" <dillaman(a)redhat.com>
> > > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>,
"ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> > > > Sent: Wednesday, January 20, 2021 3:42:46 PM
> > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > > >
> > > > Awesome information. I new I had to be missing something.
> > > >
> > > > All of my clients will be far newer than mimic so I don't think
that will be an issue.
> > > >
> > > > Added the following to my ceph.conf on both clusters.
> > > >
> > > > rbd_default_clone_format = 2
> > > >
> > > > root@Bunkcephmon2:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE
> > > > root@Bunkcephmon2:~# rbd ls CephTestPool2
> > > > vm-100-disk-0-CLONE
> > > >
> > > > I am sure I will be back with more questions. Hoping to replace our
Nimble storage with Ceph and NVMe.
> > > >
> > > > Appreciate it!
> > > >
> > > > ________________________________
> > > > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > > > To: "adamb" <adamb(a)medent.com>
> > > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>,
"ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> > > > Sent: Wednesday, January 20, 2021 3:28:39 PM
> > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > > >
> > > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com>
wrote:
> > > > >
> > > > > That's what I though as well, specially based on this.
> > > > >
> > > > >
> > > > >
> > > > > Note
> > > > >
> > > > > You may clone a snapshot from one pool to an image in another
pool. For example, you may maintain read-only images and snapshots as templates in one
pool, and writeable clones in another pool.
> > > > >
> > > > > root@Bunkcephmon2:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE
> > > > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1
librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be
protected
> > > > >
> > > > > root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > > rbd: protecting snap failed: (30) Read-only file system
> > > >
> > > > You have two options: (1) protect the snapshot on the primary image
so
> > > > that the protection status replicates or (2) utilize RBD clone v2
> > > > which doesn't require protection but does require Mimic or later
> > > > clients [1].
> > > >
> > > > >
> > > > > From: "Eugen Block" <eblock(a)nde.ag>
> > > > > To: "adamb" <adamb(a)medent.com>
> > > > > Cc: "ceph-users" <ceph-users(a)ceph.io>io>,
"Matt Wilder" <matt.wilder(a)bitmex.com>
> > > > > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image
Uses
> > > > >
> > > > > But you should be able to clone the mirrored snapshot on the
remote
> > > > > cluster even though it’s not protected, IIRC.
> > > > >
> > > > >
> > > > > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> > > > >
> > > > > > Two separate 4 node clusters with 10 OSD's in each
node. Micron 9300
> > > > > > NVMe's are the OSD drives. Heavily based on the
Micron/Supermicro
> > > > > > white papers.
> > > > > >
> > > > > > When I attempt to protect the snapshot on a remote image,
it errors
> > > > > > with read only.
> > > > > >
> > > > > > root@Bunkcephmon2:~# rbd snap protect
> > > > > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > > > rbd: protecting snap failed: (30) Read-only file system
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > >
> > > > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
> > > >
> > > > --
> > > > Jason
> > > >
> > >
> > >
> > > --
> > > Jason
> >
> >
> >
> > --
> > Jason
>
>
>
> --
> Jason
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
--
Jason
29 Jan
29 Jan
5:01 p.m.
I have been hammering on my setup non stop with only layering, exclusive-lock and
deep-flatten enabled. Still can't reproduce the issue. I think this is the ticket.
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 28, 2021 3:37:15 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Went through the process like 4-5 times now and its looking good. I am going to continue
beating on it to make sure. I will report back tomorrow.
Nice catch!
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Thursday, January 28, 2021 12:53:50 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 28, 2021 at 10:31 AM Jason Dillaman <jdillama(a)redhat.com> wrote:
On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb(a)medent.com> wrote:
>
> Doing some more testing.
>
> I can demote the rbd image on the primary, promote on the secondary and the image
looks great. I can map it, mount it, and it looks just like it should.
>
> However, the rbd snapshots are still unusable on the secondary even when promoted. I
went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd
snapshot still won't work either. Both snapshots look and work great on the primary.
>
> If I request a resync from the secondary the rbd snapshots start working just like
the primary, but only if I request a resync.
>
> I get the same exact results whether I am using a 15.2.8 or 16.1 client.
Any chance I can get you to retry your test w/ the object-map /
fast-diff disabled on the primary image (the features will be
propagated to the non-primary image on create / next mirror snapshot)?
I was able to recreate the issue on when the object-map is enabled and
I'll hopefully get a fix for that issue up for review today. I just
want to verify that it is the same issue that you are hitting.
Thanks. I've opened a tracker ticket
(https://tracker.ceph.com/issues/49037) for this.
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
________________________________
From: "adamb" <adamb(a)medent.com>
To: "dillaman" <dillaman(a)redhat.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Tuesday, January 26, 2021 1:51:13 PM
Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one
dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1
build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:44:26 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 3:02:23 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master
or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb(a)medent.com> wrote:
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
The steps are pretty straight forward.
- Create rbd image of 500G on the primary
- Enable rbd-mirror snapshot on the image
- Map the image on the primary
- Format the block device with ext4
- Mount it and write out 200-300G worth of data (I am using rsync with some local real
data we have)
- Unmap the image from the primary
- Create rdb snapshot
- Create rdb mirror snapshot
- Wait for copy process to complete
- Clone the rdb snapshot on secondary
- Map the image on secondary
- Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version
ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror
ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________
From: "Jason Dillaman" <jdillama(a)redhat.com>
To: "adamb" <adamb(a)medent.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
Sent: Friday, January 22, 2021 2:11:36 PM
Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't
repeat what you are seeing and we do have test cases that really
hammer random IO on primary images, create snapshots, rinse-and-repeat
and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb(a)medent.com> wrote:
>
> I have been doing a lot of testing.
>
> The size of the RBD image doesn't have any effect.
>
> I run into the issue once I actually write data to the rbd. The more data I write
out, the larger the chance of reproducing the issue.
>
> I seem to hit the issue of missing the filesystem all together the most, but I have
also had a few instances where some of the data was simply missing.
>
> I monitor the mirror status on the remote cluster until the snapshot is 100% copied
and also make sure all the IO is done. My setup has no issue maxing out my 10G
interconnect during replication, so its pretty obvious once its done.
>
> The only way I have found to resolve the issue is to call a mirror resync on the
secondary array.
>
> I can then map the rbd on the primary, write more data to it, snap it again, and I
am back in the same position.
>
> ________________________________
> From: "adamb" <adamb(a)medent.com>
> To: "dillaman" <dillaman(a)redhat.com>
> Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 3:11:31 PM
> Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> Sure thing.
>
> root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1
> SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
> 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user
> 12595
.mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd
2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[]
6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
>
>
> root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
> vm-100-disk-1:
> global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
> state: up+replaying
> description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"}
> service: admin on Bunkcephmon1
> last_update: 2021-01-21 15:06:24
> peer_sites:
> name: ccs
> state: up+stopped
> description: local image is primary
> last_update: 2021-01-21 15:06:23
>
>
> root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
> root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
> /dev/nbd0
> root@Bunkcephtest1:~# blkid /dev/nbd0
> root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
>
>
> Primary still looks good.
>
> root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1
CephTestPool1/vm-100-disk-1-CLONE
> root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
> /dev/nbd0
> root@Ccscephtest1:~# blkid /dev/nbd0
> /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d"
TYPE="ext4"
> root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> root@Ccscephtest1:~# cat /proc/mounts | grep nbd0
> /dev/nbd0 /usr2 ext4 rw,relatime 0 0
>
>
>
>
>
>
> From: "Jason Dillaman" <jdillama(a)redhat.com>
> To: "adamb" <adamb(a)medent.com>
> Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> Sent: Thursday, January 21, 2021 3:01:46 PM
> Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
>
> On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb(a)medent.com> wrote:
> >
> > I was able to trigger the issue again.
> >
> > - On the primary I created a snap called TestSnapper for disk vm-100-disk-1
> > - Allowed the next RBD-Mirror scheduled snap to complete
> > - At this point the snapshot is showing up on the remote side.
> >
> > root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1
> > vm-100-disk-1:
> > global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403
> > state: up+replaying
> > description: replaying,
{"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"}
> > service: admin on Bunkcephmon1
> > last_update: 2021-01-21 11:46:24
> > peer_sites:
> > name: ccs
> > state: up+stopped
> > description: local image is primary
> > last_update: 2021-01-21 11:46:28
> >
> > root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1
> > SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE
> > 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user
> > 11573
.mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253
2 TiB Thu Jan 21 11:35:00 2021 mirror (primary
peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
> >
> > Seems like the sync is complete, So I then clone it, map it and attempt to
mount it.
>
> Can you run "snap ls --all" on the non-primary cluster? The
> non-primary snapshot will list its status. On my cluster (with a much
> smaller image):
>
> #
> # CLUSTER 1
> #
> $ rbd --cluster cluster1 create --size 1G mirror/image1
> $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot
> Mirroring enabled
> $ rbd --cluster cluster1 device map -t nbd mirror/image1
> /dev/nbd0
> $ mkfs.ext4 /dev/nbd0
> mke2fs 1.45.5 (07-Jan-2020)
> Discarding device blocks: done
> Creating filesystem with 262144 4k blocks and 65536 inodes
> Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff
> Superblock backups stored on blocks:
> 32768, 98304, 163840, 229376
>
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (8192 blocks): done
> Writing superblocks and filesystem accounting information: done
> $ blkid /dev/nbd0
> /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
> BLOCK_SIZE="4096" TYPE="ext4"
> $ rbd --cluster cluster1 snap create mirror/image1@fs
> Creating snap: 100% complete...done.
> $ rbd --cluster cluster1 mirror image snapshot mirror/image1
> Snapshot ID: 6
> $ rbd --cluster cluster1 snap ls --all mirror/image1
> SNAPID NAME
> SIZE PROTECTED TIMESTAMP
> NAMESPACE
> 5 fs
> 1 GiB Thu Jan 21 14:50:24 2021
> user
> 6
.mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81
> 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary
> peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
>
> #
> # CLUSTER 2
> #
>
> $ rbd --cluster cluster2 mirror image status mirror/image1
> image1:
> global_id: f9f692b8-2405-416c-9247-5628e303947a
> state: up+replaying
> description: replaying,
>
{"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"}
> service: mirror.0 on cube-1
> last_update: 2021-01-21 14:51:18
> peer_sites:
> name: cluster1
> state: up+stopped
> description: local image is primary
> last_update: 2021-01-21 14:51:27
> $ rbd --cluster cluster2 snap ls --all mirror/image1
> SNAPID NAME
> SIZE PROTECTED TIMESTAMP
> NAMESPACE
> 5 fs
> 1 GiB Thu Jan 21 14:50:52
> 2021 user
> 6
.mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686
> 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary
> peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied)
> $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone
> mirror/image1@fs mirror/image2
> $ rbd --cluster cluster2 device map -t nbd mirror/image2
> /dev/nbd1
> $ blkid /dev/nbd1
> /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff"
> BLOCK_SIZE="4096" TYPE="ext4"
> $ mount /dev/nbd1 /mnt/
> $ mount | grep nbd
> /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
>
> > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper
CephTestPool1/vm-100-disk-1-CLONE
> > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin
--keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing
codepage or helper program, or other error.
> >
> > On the primary still no issues
> >
> > root@Ccscephtest1:/etc/pve/priv# rbd clone
CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE
> > root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > /dev/nbd0
> > root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
> >
> >
> >
> >
> >
> > ________________________________
> > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > To: "adamb" <adamb(a)medent.com>
> > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > Sent: Thursday, January 21, 2021 9:42:26 AM
> > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> >
> > On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb(a)medent.com> wrote:
> > >
> > > After the resync finished. I can mount it now.
> > >
> > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1
CephTestPool1/vm-100-disk-0-CLONE
> > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id
admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > /dev/nbd0
> > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > >
> > > Makes me a bit nervous how it got into that position and everything
appeared ok.
> >
> > We unfortunately need to create the snapshots that are being synced as
> > a first step, but perhaps there are some extra guardrails we can put
> > on the system to prevent premature usage if the sync status doesn't
> > indicate that it's complete.
> >
> > > ________________________________
> > > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > > To: "adamb" <adamb(a)medent.com>
> > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Matt Wilder" <matt.wilder(a)bitmex.com>
> > > Sent: Thursday, January 21, 2021 9:25:11 AM
> > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > >
> > > On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb(a)medent.com>
wrote:
> > > >
> > > > When cloning the snapshot on the remote cluster I can't see my
ext4 filesystem.
> > > >
> > > > Using the same exact snapshot on both sides. Shouldn't this be
consistent?
> > >
> > > Yes. Has the replication process completed ("rbd mirror image status
> > > CephTestPool1/vm-100-disk-0")?
> > >
> > > > Primary Site
> > > > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 |
grep TestSnapper1
> > > > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
> > > >
> > > > root@Ccscephtest1:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE
> > > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > > /dev/nbd0
> > > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2
> > > >
> > > > Secondary Site
> > > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 |
grep TestSnapper1
> > > > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
> > > >
> > > > root@Bunkcephtest1:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE
> > > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE
--id admin --keyring /etc/ceph/ceph.client.admin.keyring
> > > > /dev/nbd0
> > > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
> > > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0,
missing codepage or helper program, or other error.
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: "adamb" <adamb(a)medent.com>
> > > > To: "dillaman" <dillaman(a)redhat.com>
> > > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>,
"ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> > > > Sent: Wednesday, January 20, 2021 3:42:46 PM
> > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > > >
> > > > Awesome information. I new I had to be missing something.
> > > >
> > > > All of my clients will be far newer than mimic so I don't think
that will be an issue.
> > > >
> > > > Added the following to my ceph.conf on both clusters.
> > > >
> > > > rbd_default_clone_format = 2
> > > >
> > > > root@Bunkcephmon2:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE
> > > > root@Bunkcephmon2:~# rbd ls CephTestPool2
> > > > vm-100-disk-0-CLONE
> > > >
> > > > I am sure I will be back with more questions. Hoping to replace our
Nimble storage with Ceph and NVMe.
> > > >
> > > > Appreciate it!
> > > >
> > > > ________________________________
> > > > From: "Jason Dillaman" <jdillama(a)redhat.com>
> > > > To: "adamb" <adamb(a)medent.com>
> > > > Cc: "Eugen Block" <eblock(a)nde.ag>ag>,
"ceph-users" <ceph-users(a)ceph.io>io>, "Matt Wilder"
<matt.wilder(a)bitmex.com>
> > > > Sent: Wednesday, January 20, 2021 3:28:39 PM
> > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
> > > >
> > > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb(a)medent.com>
wrote:
> > > > >
> > > > > That's what I though as well, specially based on this.
> > > > >
> > > > >
> > > > >
> > > > > Note
> > > > >
> > > > > You may clone a snapshot from one pool to an image in another
pool. For example, you may maintain read-only images and snapshots as templates in one
pool, and writeable clones in another pool.
> > > > >
> > > > > root@Bunkcephmon2:~# rbd clone
CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE
> > > > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1
librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be
protected
> > > > >
> > > > > root@Bunkcephmon2:~# rbd snap protect
CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > > rbd: protecting snap failed: (30) Read-only file system
> > > >
> > > > You have two options: (1) protect the snapshot on the primary image
so
> > > > that the protection status replicates or (2) utilize RBD clone v2
> > > > which doesn't require protection but does require Mimic or later
> > > > clients [1].
> > > >
> > > > >
> > > > > From: "Eugen Block" <eblock(a)nde.ag>
> > > > > To: "adamb" <adamb(a)medent.com>
> > > > > Cc: "ceph-users" <ceph-users(a)ceph.io>io>,
"Matt Wilder" <matt.wilder(a)bitmex.com>
> > > > > Sent: Wednesday, January 20, 2021 3:00:54 PM
> > > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image
Uses
> > > > >
> > > > > But you should be able to clone the mirrored snapshot on the
remote
> > > > > cluster even though it’s not protected, IIRC.
> > > > >
> > > > >
> > > > > Zitat von Adam Boyhan <adamb(a)medent.com>om>:
> > > > >
> > > > > > Two separate 4 node clusters with 10 OSD's in each
node. Micron 9300
> > > > > > NVMe's are the OSD drives. Heavily based on the
Micron/Supermicro
> > > > > > white papers.
> > > > > >
> > > > > > When I attempt to protect the snapshot on a remote image,
it errors
> > > > > > with read only.
> > > > > >
> > > > > > root@Bunkcephmon2:~# rbd snap protect
> > > > > > CephTestPool1/vm-100-disk-0@TestSnapper1
> > > > > > rbd: protecting snap failed: (30) Read-only file system
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > >
> > > > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
> > > >
> > > > --
> > > > Jason
> > > >
> > >
> > >
> > > --
> > > Jason
> >
> >
> >
> > --
> > Jason
>
>
>
> --
> Jason
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
--
Jason
1177
days inactive
1186
days old
26 comments
5 participants
participants (5)
-
Adam Boyhan -
David Caro -
Eugen Block -
Jason Dillaman -
Matt Wilder