RBD-Mirror Snapshot Backup Image Uses
I have been doing some testing with RBD-Mirror Snapshots to a remote Ceph cluster. Does anyone know if the images on the remote cluster can be utilized in anyway? Would love the ability to clone them, or even readonly would be nice.
Have you tried just using them? (RO, if you do RW things might go crazy, would be nice to try though). You might be able to create a clone too, and I guess worst case just cp/deep cp. I'm interested in your findings btw. I'd be greatful if you share them :) Thanks! On 01/20 14:23, Adam Boyhan wrote:
I have been doing some testing with RBD-Mirror Snapshots to a remote Ceph cluster.
Does anyone know if the images on the remote cluster can be utilized in anyway? Would love the ability to clone them, or even readonly would be nice. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
Can you describe your Ceph deployment? On Wed, Jan 20, 2021 at 11:24 AM Adam Boyhan <adamb@medent.com> wrote:
I have been doing some testing with RBD-Mirror Snapshots to a remote Ceph cluster.
Does anyone know if the images on the remote cluster can be utilized in anyway? Would love the ability to clone them, or even readonly would be nice. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- This e-mail and all information in, attached to, or linked via this e-mail (together the ‘e-mail’) is confidential and may be legally privileged. It is intended solely for the intended addressee(s). Access to, or any onward transmission, of this e-mail by any other person is not authorised. If you are not the intended recipient, you are requested to immediately alert the sender of this e-mail and to immediately delete this e-mail. Any disclosure in any form of all or part of this e-mail, or of any the parties to it, including any copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. This e-mail is not, and is not intended to be, and should not be construed as being, (a) any offer, solicitation, or promotion of any kind; (b) the basis of any investment or other decision(s); (c) any recommendation to buy, sell or transact in any manner any good(s), product(s) or service(s), nor engage in any investment(s) or other transaction(s) or activities; or (d) the provision of, or related to, any advisory service(s) or activities, including regarding any investment, tax, legal, financial, accounting, consulting or any other related service(s).
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers. When I attempt to protect the snapshot on a remote image, it errors with read only. root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC. Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's what I though as well, specially based on this. Note You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC. Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/ -- Jason
Awesome information. I new I had to be missing something. All of my clients will be far newer than mimic so I don't think that will be an issue. Added the following to my ceph.conf on both clusters. rbd_default_clone_format = 2 root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe. Appreciate it! From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/ -- Jason
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem. Using the same exact snapshot on both sides. Shouldn't this be consistent? Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2 Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses Awesome information. I new I had to be missing something. All of my clients will be far newer than mimic so I don't think that will be an issue. Added the following to my ceph.conf on both clusters. rbd_default_clone_format = 2 root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe. Appreciate it! From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/ -- Jason
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
After the resync finished. I can mount it now. root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 Makes me a bit nervous how it got into that position and everything appeared ok. From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
I was able to trigger the issue again. - On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side. root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28 root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770]) Seems like the sync is complete, So I then clone it, map it and attempt to mount it. root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. On the primary still no issues root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2 From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image): # # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f]) # # CLUSTER 2 # $ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason
Sure thing. root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied) root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23 root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. Primary still looks good. root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0 From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image): # # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f]) # # CLUSTER 2 # $ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason
I have been doing a lot of testing. The size of the RBD image doesn't have any effect. I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue. I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing. I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done. The only way I have found to resolve the issue is to call a mirror resync on the secondary array. I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position. From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses Sure thing. root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied) root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23 root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. Primary still looks good. root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0 From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image): # # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f]) # # CLUSTER 2 # $ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet. Thanks! On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
The steps are pretty straight forward. - Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary Just as a reference. All of my nodes are the same. root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable) root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images This is pretty straight forward, I don't know what I could be missing here. From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet. Thanks! On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 NVMe's are the OSD drives. Heavily based on the Micron/Supermicro white papers.
When I attempt to protect the snapshot on a remote image, it errors with read only.
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)? On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
> Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > white papers. > > When I attempt to protect the snapshot on a remote image, it errors > with read only. > > root@Bunkcephmon2:~# rbd snap protect > CephTestPool1/vm-100-disk-0@TestSnapper1 > rbd: protecting snap failed: (30) Read-only file system > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
I will have to do some looking into how that is done on Proxmox, but most definitely. From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:02:23 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)? On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote:
That's what I though as well, specially based on this.
Note
You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool.
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected
root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
From: "Eugen Block" <eblock@nde.ag> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:00:54 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
But you should be able to clone the mirrored snapshot on the remote cluster even though it’s not protected, IIRC.
Zitat von Adam Boyhan <adamb@medent.com>:
> Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > white papers. > > When I attempt to protect the snapshot on a remote image, it errors > with read only. > > root@Bunkcephmon2:~# rbd snap protect > CephTestPool1/vm-100-disk-0@TestSnapper1 > rbd: protecting snap failed: (30) Read-only file system > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb@medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:02:23 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote: > > That's what I though as well, specially based on this. > > > > Note > > You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected > > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 > rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
> > From: "Eugen Block" <eblock@nde.ag> > To: "adamb" <adamb@medent.com> > Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > Sent: Wednesday, January 20, 2021 3:00:54 PM > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > But you should be able to clone the mirrored snapshot on the remote > cluster even though it’s not protected, IIRC. > > > Zitat von Adam Boyhan <adamb@medent.com>: > > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > > white papers. > > > > When I attempt to protect the snapshot on a remote image, it errors > > with read only. > > > > root@Bunkcephmon2:~# rbd snap protect > > CephTestPool1/vm-100-disk-0@TestSnapper1 > > rbd: protecting snap failed: (30) Read-only file system > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
-- Jason
Did some testing with clients running 16.1. I setup two different clients, each one dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1 build. root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) Unfortunately, I am hitting the same exact issues using a pacific client. Would this confirm that its something specific in 15.2.8 on the osd/mon nodes? From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:44:26 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb@medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:02:23 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote: > > That's what I though as well, specially based on this. > > > > Note > > You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected > > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 > rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
> > From: "Eugen Block" <eblock@nde.ag> > To: "adamb" <adamb@medent.com> > Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > Sent: Wednesday, January 20, 2021 3:00:54 PM > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > But you should be able to clone the mirrored snapshot on the remote > cluster even though it’s not protected, IIRC. > > > Zitat von Adam Boyhan <adamb@medent.com>: > > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > > white papers. > > > > When I attempt to protect the snapshot on a remote image, it errors > > with read only. > > > > root@Bunkcephmon2:~# rbd snap protect > > CephTestPool1/vm-100-disk-0@TestSnapper1 > > rbd: protecting snap failed: (30) Read-only file system > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
-- Jason
Doing some more testing. I can demote the rbd image on the primary, promote on the secondary and the image looks great. I can map it, mount it, and it looks just like it should. However, the rbd snapshots are still unusable on the secondary even when promoted. I went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd snapshot still won't work either. Both snapshots look and work great on the primary. If I request a resync from the secondary the rbd snapshots start working just like the primary, but only if I request a resync. I get the same exact results whether I am using a 15.2.8 or 16.1 client. From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Tuesday, January 26, 2021 1:51:13 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses Did some testing with clients running 16.1. I setup two different clients, each one dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1 build. root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) Unfortunately, I am hitting the same exact issues using a pacific client. Would this confirm that its something specific in 15.2.8 on the osd/mon nodes? From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:44:26 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb@medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:02:23 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote:
When cloning the snapshot on the remote cluster I can't see my ext4 filesystem.
Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
Primary Site root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:~# mount /dev/nbd0 /usr2
Secondary Site root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:42:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Awesome information. I new I had to be missing something.
All of my clients will be far newer than mimic so I don't think that will be an issue.
Added the following to my ceph.conf on both clusters.
rbd_default_clone_format = 2
root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE root@Bunkcephmon2:~# rbd ls CephTestPool2 vm-100-disk-0-CLONE
I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe.
Appreciate it!
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Wednesday, January 20, 2021 3:28:39 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote: > > That's what I though as well, specially based on this. > > > > Note > > You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected > > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 > rbd: protecting snap failed: (30) Read-only file system
You have two options: (1) protect the snapshot on the primary image so that the protection status replicates or (2) utilize RBD clone v2 which doesn't require protection but does require Mimic or later clients [1].
> > From: "Eugen Block" <eblock@nde.ag> > To: "adamb" <adamb@medent.com> > Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > Sent: Wednesday, January 20, 2021 3:00:54 PM > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > But you should be able to clone the mirrored snapshot on the remote > cluster even though it’s not protected, IIRC. > > > Zitat von Adam Boyhan <adamb@medent.com>: > > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > > white papers. > > > > When I attempt to protect the snapshot on a remote image, it errors > > with read only. > > > > root@Bunkcephmon2:~# rbd snap protect > > CephTestPool1/vm-100-disk-0@TestSnapper1 > > rbd: protecting snap failed: (30) Read-only file system > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/
-- Jason
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb@medent.com> wrote:
Doing some more testing.
I can demote the rbd image on the primary, promote on the secondary and the image looks great. I can map it, mount it, and it looks just like it should.
However, the rbd snapshots are still unusable on the secondary even when promoted. I went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd snapshot still won't work either. Both snapshots look and work great on the primary.
If I request a resync from the secondary the rbd snapshots start working just like the primary, but only if I request a resync.
I get the same exact results whether I am using a 15.2.8 or 16.1 client.
Thanks. I've opened a tracker ticket (https://tracker.ceph.com/issues/49037) for this.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Tuesday, January 26, 2021 1:51:13 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1 build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:44:26 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb@medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:02:23 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote:
After the resync finished. I can mount it now.
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2
Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:25:11 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote: > > When cloning the snapshot on the remote cluster I can't see my ext4 filesystem. > > Using the same exact snapshot on both sides. Shouldn't this be consistent?
Yes. Has the replication process completed ("rbd mirror image status CephTestPool1/vm-100-disk-0")?
> Primary Site > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user > > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > /dev/nbd0 > root@Ccscephtest1:~# mount /dev/nbd0 /usr2 > > Secondary Site > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > /dev/nbd0 > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. > > > > ________________________________ > From: "adamb" <adamb@medent.com> > To: "dillaman" <dillaman@redhat.com> > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > Sent: Wednesday, January 20, 2021 3:42:46 PM > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > Awesome information. I new I had to be missing something. > > All of my clients will be far newer than mimic so I don't think that will be an issue. > > Added the following to my ceph.conf on both clusters. > > rbd_default_clone_format = 2 > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > root@Bunkcephmon2:~# rbd ls CephTestPool2 > vm-100-disk-0-CLONE > > I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe. > > Appreciate it! > > ________________________________ > From: "Jason Dillaman" <jdillama@redhat.com> > To: "adamb" <adamb@medent.com> > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > Sent: Wednesday, January 20, 2021 3:28:39 PM > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote: > > > > That's what I though as well, specially based on this. > > > > > > > > Note > > > > You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected > > > > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 > > rbd: protecting snap failed: (30) Read-only file system > > You have two options: (1) protect the snapshot on the primary image so > that the protection status replicates or (2) utilize RBD clone v2 > which doesn't require protection but does require Mimic or later > clients [1]. > > > > > From: "Eugen Block" <eblock@nde.ag> > > To: "adamb" <adamb@medent.com> > > Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > Sent: Wednesday, January 20, 2021 3:00:54 PM > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > But you should be able to clone the mirrored snapshot on the remote > > cluster even though it’s not protected, IIRC. > > > > > > Zitat von Adam Boyhan <adamb@medent.com>: > > > > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > > > white papers. > > > > > > When I attempt to protect the snapshot on a remote image, it errors > > > with read only. > > > > > > root@Bunkcephmon2:~# rbd snap protect > > > CephTestPool1/vm-100-disk-0@TestSnapper1 > > > rbd: protecting snap failed: (30) Read-only file system > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/ > > -- > Jason >
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
On Thu, Jan 28, 2021 at 10:31 AM Jason Dillaman <jdillama@redhat.com> wrote:
On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb@medent.com> wrote:
Doing some more testing.
I can demote the rbd image on the primary, promote on the secondary and the image looks great. I can map it, mount it, and it looks just like it should.
However, the rbd snapshots are still unusable on the secondary even when promoted. I went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd snapshot still won't work either. Both snapshots look and work great on the primary.
If I request a resync from the secondary the rbd snapshots start working just like the primary, but only if I request a resync.
I get the same exact results whether I am using a 15.2.8 or 16.1 client.
Any chance I can get you to retry your test w/ the object-map / fast-diff disabled on the primary image (the features will be propagated to the non-primary image on create / next mirror snapshot)? I was able to recreate the issue on when the object-map is enabled and I'll hopefully get a fix for that issue up for review today. I just want to verify that it is the same issue that you are hitting.
Thanks. I've opened a tracker ticket (https://tracker.ceph.com/issues/49037) for this.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Tuesday, January 26, 2021 1:51:13 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1 build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:44:26 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb@medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:02:23 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote: > > After the resync finished. I can mount it now. > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > /dev/nbd0 > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
> ________________________________ > From: "Jason Dillaman" <jdillama@redhat.com> > To: "adamb" <adamb@medent.com> > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > Sent: Thursday, January 21, 2021 9:25:11 AM > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote: > > > > When cloning the snapshot on the remote cluster I can't see my ext4 filesystem. > > > > Using the same exact snapshot on both sides. Shouldn't this be consistent? > > Yes. Has the replication process completed ("rbd mirror image status > CephTestPool1/vm-100-disk-0")? > > > Primary Site > > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user > > > > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > /dev/nbd0 > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2 > > > > Secondary Site > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user > > > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > /dev/nbd0 > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. > > > > > > > > ________________________________ > > From: "adamb" <adamb@medent.com> > > To: "dillaman" <dillaman@redhat.com> > > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > Sent: Wednesday, January 20, 2021 3:42:46 PM > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > Awesome information. I new I had to be missing something. > > > > All of my clients will be far newer than mimic so I don't think that will be an issue. > > > > Added the following to my ceph.conf on both clusters. > > > > rbd_default_clone_format = 2 > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > root@Bunkcephmon2:~# rbd ls CephTestPool2 > > vm-100-disk-0-CLONE > > > > I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe. > > > > Appreciate it! > > > > ________________________________ > > From: "Jason Dillaman" <jdillama@redhat.com> > > To: "adamb" <adamb@medent.com> > > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > Sent: Wednesday, January 20, 2021 3:28:39 PM > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote: > > > > > > That's what I though as well, specially based on this. > > > > > > > > > > > > Note > > > > > > You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. > > > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected > > > > > > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 > > > rbd: protecting snap failed: (30) Read-only file system > > > > You have two options: (1) protect the snapshot on the primary image so > > that the protection status replicates or (2) utilize RBD clone v2 > > which doesn't require protection but does require Mimic or later > > clients [1]. > > > > > > > > From: "Eugen Block" <eblock@nde.ag> > > > To: "adamb" <adamb@medent.com> > > > Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > > Sent: Wednesday, January 20, 2021 3:00:54 PM > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > But you should be able to clone the mirrored snapshot on the remote > > > cluster even though it’s not protected, IIRC. > > > > > > > > > Zitat von Adam Boyhan <adamb@medent.com>: > > > > > > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > > > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > > > > white papers. > > > > > > > > When I attempt to protect the snapshot on a remote image, it errors > > > > with read only. > > > > > > > > root@Bunkcephmon2:~# rbd snap protect > > > > CephTestPool1/vm-100-disk-0@TestSnapper1 > > > > rbd: protecting snap failed: (30) Read-only file system > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/ > > > > -- > > Jason > > > > > -- > Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
Went through the process like 4-5 times now and its looking good. I am going to continue beating on it to make sure. I will report back tomorrow. Nice catch! From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 28, 2021 12:53:50 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Thu, Jan 28, 2021 at 10:31 AM Jason Dillaman <jdillama@redhat.com> wrote:
On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb@medent.com> wrote:
Doing some more testing.
I can demote the rbd image on the primary, promote on the secondary and the image looks great. I can map it, mount it, and it looks just like it should.
However, the rbd snapshots are still unusable on the secondary even when promoted. I went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd snapshot still won't work either. Both snapshots look and work great on the primary.
If I request a resync from the secondary the rbd snapshots start working just like the primary, but only if I request a resync.
I get the same exact results whether I am using a 15.2.8 or 16.1 client.
Any chance I can get you to retry your test w/ the object-map / fast-diff disabled on the primary image (the features will be propagated to the non-primary image on create / next mirror snapshot)? I was able to recreate the issue on when the object-map is enabled and I'll hopefully get a fix for that issue up for review today. I just want to verify that it is the same issue that you are hitting.
Thanks. I've opened a tracker ticket (https://tracker.ceph.com/issues/49037) for this.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Tuesday, January 26, 2021 1:51:13 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1 build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:44:26 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb@medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:02:23 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote: > > After the resync finished. I can mount it now. > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > /dev/nbd0 > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
> ________________________________ > From: "Jason Dillaman" <jdillama@redhat.com> > To: "adamb" <adamb@medent.com> > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > Sent: Thursday, January 21, 2021 9:25:11 AM > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote: > > > > When cloning the snapshot on the remote cluster I can't see my ext4 filesystem. > > > > Using the same exact snapshot on both sides. Shouldn't this be consistent? > > Yes. Has the replication process completed ("rbd mirror image status > CephTestPool1/vm-100-disk-0")? > > > Primary Site > > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user > > > > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > /dev/nbd0 > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2 > > > > Secondary Site > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user > > > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > /dev/nbd0 > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. > > > > > > > > ________________________________ > > From: "adamb" <adamb@medent.com> > > To: "dillaman" <dillaman@redhat.com> > > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > Sent: Wednesday, January 20, 2021 3:42:46 PM > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > Awesome information. I new I had to be missing something. > > > > All of my clients will be far newer than mimic so I don't think that will be an issue. > > > > Added the following to my ceph.conf on both clusters. > > > > rbd_default_clone_format = 2 > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > root@Bunkcephmon2:~# rbd ls CephTestPool2 > > vm-100-disk-0-CLONE > > > > I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe. > > > > Appreciate it! > > > > ________________________________ > > From: "Jason Dillaman" <jdillama@redhat.com> > > To: "adamb" <adamb@medent.com> > > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > Sent: Wednesday, January 20, 2021 3:28:39 PM > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote: > > > > > > That's what I though as well, specially based on this. > > > > > > > > > > > > Note > > > > > > You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. > > > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected > > > > > > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 > > > rbd: protecting snap failed: (30) Read-only file system > > > > You have two options: (1) protect the snapshot on the primary image so > > that the protection status replicates or (2) utilize RBD clone v2 > > which doesn't require protection but does require Mimic or later > > clients [1]. > > > > > > > > From: "Eugen Block" <eblock@nde.ag> > > > To: "adamb" <adamb@medent.com> > > > Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > > Sent: Wednesday, January 20, 2021 3:00:54 PM > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > But you should be able to clone the mirrored snapshot on the remote > > > cluster even though it’s not protected, IIRC. > > > > > > > > > Zitat von Adam Boyhan <adamb@medent.com>: > > > > > > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > > > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > > > > white papers. > > > > > > > > When I attempt to protect the snapshot on a remote image, it errors > > > > with read only. > > > > > > > > root@Bunkcephmon2:~# rbd snap protect > > > > CephTestPool1/vm-100-disk-0@TestSnapper1 > > > > rbd: protecting snap failed: (30) Read-only file system > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/ > > > > -- > > Jason > > > > > -- > Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
I have been hammering on my setup non stop with only layering, exclusive-lock and deep-flatten enabled. Still can't reproduce the issue. I think this is the ticket. From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 28, 2021 3:37:15 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses Went through the process like 4-5 times now and its looking good. I am going to continue beating on it to make sure. I will report back tomorrow. Nice catch! From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 28, 2021 12:53:50 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses On Thu, Jan 28, 2021 at 10:31 AM Jason Dillaman <jdillama@redhat.com> wrote:
On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb@medent.com> wrote:
Doing some more testing.
I can demote the rbd image on the primary, promote on the secondary and the image looks great. I can map it, mount it, and it looks just like it should.
However, the rbd snapshots are still unusable on the secondary even when promoted. I went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd snapshot still won't work either. Both snapshots look and work great on the primary.
If I request a resync from the secondary the rbd snapshots start working just like the primary, but only if I request a resync.
I get the same exact results whether I am using a 15.2.8 or 16.1 client.
Any chance I can get you to retry your test w/ the object-map / fast-diff disabled on the primary image (the features will be propagated to the non-primary image on create / next mirror snapshot)? I was able to recreate the issue on when the object-map is enabled and I'll hopefully get a fix for that issue up for review today. I just want to verify that it is the same issue that you are hitting.
Thanks. I've opened a tracker ticket (https://tracker.ceph.com/issues/49037) for this.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Tuesday, January 26, 2021 1:51:13 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Did some testing with clients running 16.1. I setup two different clients, each one dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1 build.
root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
Unfortunately, I am hitting the same exact issues using a pacific client.
Would this confirm that its something specific in 15.2.8 on the osd/mon nodes?
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:44:26 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb@medent.com> wrote:
I will have to do some looking into how that is done on Proxmox, but most definitely.
Thanks, appreciate it.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 3:02:23 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you can attempt to repeat the process on the latest master or pacific branch clients (no need to upgrade the MONs/OSDs)?
On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@medent.com> wrote:
The steps are pretty straight forward.
- Create rbd image of 500G on the primary - Enable rbd-mirror snapshot on the image - Map the image on the primary - Format the block device with ext4 - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) - Unmap the image from the primary - Create rdb snapshot - Create rdb mirror snapshot - Wait for copy process to complete - Clone the rdb snapshot on secondary - Map the image on secondary - Try to mount on secondary
Just as a reference. All of my nodes are the same.
root@Bunkcephtest1:~# ceph --version ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable)
root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images
This is pretty straight forward, I don't know what I could be missing here.
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Friday, January 22, 2021 2:11:36 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Any chance you could write a small reproducer test script? I can't repeat what you are seeing and we do have test cases that really hammer random IO on primary images, create snapshots, rinse-and-repeat and they haven't turned up anything yet.
Thanks!
On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@medent.com> wrote:
I have been doing a lot of testing.
The size of the RBD image doesn't have any effect.
I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue.
I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing.
I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done.
The only way I have found to resolve the issue is to call a mirror resync on the secondary array.
I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position.
________________________________ From: "adamb" <adamb@medent.com> To: "dillaman" <dillaman@redhat.com> Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:11:31 PM Subject: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
Sure thing.
root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied)
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 15:06:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 15:06:23
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Bunkcephtest1:~# blkid /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
Primary still looks good.
root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE /dev/nbd0 root@Ccscephtest1:~# blkid /dev/nbd0 /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" root@Ccscephtest1:~# mount /dev/nbd0 /usr2 root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 /dev/nbd0 /usr2 ext4 rw,relatime 0 0
From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 3:01:46 PM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@medent.com> wrote:
I was able to trigger the issue again.
- On the primary I created a snap called TestSnapper for disk vm-100-disk-1 - Allowed the next RBD-Mirror scheduled snap to complete - At this point the snapshot is showing up on the remote side.
root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 vm-100-disk-1: global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} service: admin on Bunkcephmon1 last_update: 2021-01-21 11:46:24 peer_sites: name: ccs state: up+stopped description: local image is primary last_update: 2021-01-21 11:46:28
root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770])
Seems like the sync is complete, So I then clone it, map it and attempt to mount it.
Can you run "snap ls --all" on the non-primary cluster? The non-primary snapshot will list its status. On my cluster (with a much smaller image):
# # CLUSTER 1 # $ rbd --cluster cluster1 create --size 1G mirror/image1 $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot Mirroring enabled $ rbd --cluster cluster1 device map -t nbd mirror/image1 /dev/nbd0 $ mkfs.ext4 /dev/nbd0 mke2fs 1.45.5 (07-Jan-2020) Discarding device blocks: done Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff Superblock backups stored on blocks: 32768, 98304, 163840, 229376
Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done $ blkid /dev/nbd0 /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ rbd --cluster cluster1 snap create mirror/image1@fs Creating snap: 100% complete...done. $ rbd --cluster cluster1 mirror image snapshot mirror/image1 Snapshot ID: 6 $ rbd --cluster cluster1 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:24 2021 user 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f])
# # CLUSTER 2 #
$ rbd --cluster cluster2 mirror image status mirror/image1 image1: global_id: f9f692b8-2405-416c-9247-5628e303947a state: up+replaying description: replaying, {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} service: mirror.0 on cube-1 last_update: 2021-01-21 14:51:18 peer_sites: name: cluster1 state: up+stopped description: local image is primary last_update: 2021-01-21 14:51:27 $ rbd --cluster cluster2 snap ls --all mirror/image1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 5 fs 1 GiB Thu Jan 21 14:50:52 2021 user 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone mirror/image1@fs mirror/image2 $ rbd --cluster cluster2 device map -t nbd mirror/image2 /dev/nbd1 $ blkid /dev/nbd1 /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" BLOCK_SIZE="4096" TYPE="ext4" $ mount /dev/nbd1 /mnt/ $ mount | grep nbd /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel)
root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
On the primary still no issues
root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/nbd0 root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2
________________________________ From: "Jason Dillaman" <jdillama@redhat.com> To: "adamb" <adamb@medent.com> Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> Sent: Thursday, January 21, 2021 9:42:26 AM Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses
On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@medent.com> wrote: > > After the resync finished. I can mount it now. > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > /dev/nbd0 > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > Makes me a bit nervous how it got into that position and everything appeared ok.
We unfortunately need to create the snapshots that are being synced as a first step, but perhaps there are some extra guardrails we can put on the system to prevent premature usage if the sync status doesn't indicate that it's complete.
> ________________________________ > From: "Jason Dillaman" <jdillama@redhat.com> > To: "adamb" <adamb@medent.com> > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > Sent: Thursday, January 21, 2021 9:25:11 AM > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@medent.com> wrote: > > > > When cloning the snapshot on the remote cluster I can't see my ext4 filesystem. > > > > Using the same exact snapshot on both sides. Shouldn't this be consistent? > > Yes. Has the replication process completed ("rbd mirror image status > CephTestPool1/vm-100-disk-0")? > > > Primary Site > > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user > > > > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > /dev/nbd0 > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2 > > > > Secondary Site > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user > > > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > /dev/nbd0 > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. > > > > > > > > ________________________________ > > From: "adamb" <adamb@medent.com> > > To: "dillaman" <dillaman@redhat.com> > > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > Sent: Wednesday, January 20, 2021 3:42:46 PM > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > Awesome information. I new I had to be missing something. > > > > All of my clients will be far newer than mimic so I don't think that will be an issue. > > > > Added the following to my ceph.conf on both clusters. > > > > rbd_default_clone_format = 2 > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > root@Bunkcephmon2:~# rbd ls CephTestPool2 > > vm-100-disk-0-CLONE > > > > I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe. > > > > Appreciate it! > > > > ________________________________ > > From: "Jason Dillaman" <jdillama@redhat.com> > > To: "adamb" <adamb@medent.com> > > Cc: "Eugen Block" <eblock@nde.ag>, "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > Sent: Wednesday, January 20, 2021 3:28:39 PM > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@medent.com> wrote: > > > > > > That's what I though as well, specially based on this. > > > > > > > > > > > > Note > > > > > > You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. > > > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected > > > > > > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 > > > rbd: protecting snap failed: (30) Read-only file system > > > > You have two options: (1) protect the snapshot on the primary image so > > that the protection status replicates or (2) utilize RBD clone v2 > > which doesn't require protection but does require Mimic or later > > clients [1]. > > > > > > > > From: "Eugen Block" <eblock@nde.ag> > > > To: "adamb" <adamb@medent.com> > > > Cc: "ceph-users" <ceph-users@ceph.io>, "Matt Wilder" <matt.wilder@bitmex.com> > > > Sent: Wednesday, January 20, 2021 3:00:54 PM > > > Subject: Re: [ceph-users] Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > But you should be able to clone the mirrored snapshot on the remote > > > cluster even though it’s not protected, IIRC. > > > > > > > > > Zitat von Adam Boyhan <adamb@medent.com>: > > > > > > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > > > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > > > > white papers. > > > > > > > > When I attempt to protect the snapshot on a remote image, it errors > > > > with read only. > > > > > > > > root@Bunkcephmon2:~# rbd snap protect > > > > CephTestPool1/vm-100-disk-0@TestSnapper1 > > > > rbd: protecting snap failed: (30) Read-only file system > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/ > > > > -- > > Jason > > > > > -- > Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Adam Boyhan
-
David Caro
-
Eugen Block
-
Jason Dillaman
-
Matt Wilder