Possible bug with rbd export/import?
Hello, I think I've been running into an rbd export/import bug and wanted to see if anybody else had any experience. We're using rbd images for VM drives both with and without custom stripe sizes. When we try to export/import the drive to another ceph cluster, the VM always comes up in a busted state it can't recover from. This happens both when doing this export/import through stdin/stdout and when using a middle machine as a temp space. I remember doing this a few times in previous versions without error, so I'm not sure if this is a regression or I'm doing something different. I'm still testing this to try and track down where the issue is but wanted to post this here to see if anybody else has any experience. Example command: rbd -c /etc/ceph/cluster1.conf export pool/testvm.boot - | rbd -c /etc/ceph/cluster2.conf import - pool/testvm.boot Current cluster is on 14.2.8 and using Ubuntu 18.04 w/ 5.3.0-40-generic. Let me know if I can provide any more details to help track this down. Thanks,
Hi, Are you exporting rbd image while a VM is running upon it ? As far as I know, rbd export is not consistent You should not export an image, but only snapshots: - create a snapshot of the image - export the snapshot (rbd export pool/image@snap - | ..) - drop the snapshot Regards, On 3/10/20 8:31 PM, Matt Dunavant wrote:
Hello,
I think I've been running into an rbd export/import bug and wanted to see if anybody else had any experience.
We're using rbd images for VM drives both with and without custom stripe sizes. When we try to export/import the drive to another ceph cluster, the VM always comes up in a busted state it can't recover from. This happens both when doing this export/import through stdin/stdout and when using a middle machine as a temp space. I remember doing this a few times in previous versions without error, so I'm not sure if this is a regression or I'm doing something different. I'm still testing this to try and track down where the issue is but wanted to post this here to see if anybody else has any experience.
Example command: rbd -c /etc/ceph/cluster1.conf export pool/testvm.boot - | rbd -c /etc/ceph/cluster2.conf import - pool/testvm.boot
Current cluster is on 14.2.8 and using Ubuntu 18.04 w/ 5.3.0-40-generic.
Let me know if I can provide any more details to help track this down.
Thanks, _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 10/03/2020 19:31, Matt Dunavant wrote:
We're using rbd images for VM drives both with and without custom stripe sizes. When we try to export/import the drive to another ceph cluster, the VM always comes up in a busted state it can't recover from.
Don't shoot me for asking but is the VM being exported still started up and in use? Asking since you don't mention using snapshots. Simon
Should have mentioned, the VM is always off. We are not using snapshots either. -Matt
On Wed, Mar 11, 2020 at 9:03 AM Matt Dunavant <mdunavant@convokesystems.com> wrote:
Should have mentioned, the VM is always off. We are not using snapshots either.
Is there a last-known good release of the rbd CLI where it works as expected? If you run "rbd export -c <path to conf> <image> - | sha1sum" against both sets of images after and import, are they the same?
-Matt _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
I'm not sure of the last known good release of the rbd CLI where this worked. I just ran the sha1sum against the images and they always come up as different. Might be worth knowing, this is a volume that's provisioned at 512GB (with much less actually used) but after export, it only shows up as about 56GB.
On Fri, Mar 13, 2020 at 11:17 AM Matt Dunavant <mdunavant@convokesystems.com> wrote:
I'm not sure of the last known good release of the rbd CLI where this worked. I just ran the sha1sum against the images and they always come up as different. Might be worth knowing, this is a volume that's provisioned at 512GB (with much less actually used) but after export, it only shows up as about 56GB.
The resulting image from the "rbd import" only shows up as 56GiB?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Jason Dillaman wrote:
On Fri, Mar 13, 2020 at 11:17 AM Matt Dunavant <mdunavant(a)convokesystems.com> wrote:
I'm not sure of the last known good release of the rbd CLI where this worked. I just ran the sha1sum against the images and they always come up as different. Might be worth knowing, this is a volume that's provisioned at 512GB (with much less actually used) but after export, it only shows up as about 56GB.
The resulting image from the "rbd import" only shows up as 56GiB?
_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
Yeah, it's super odd. The actual content in the 512GB rbd image is probably about 50ish GB but the command isn't killing itself early or throwing any errors. I believe when I run the export to a middleman and then import, the image shows up as the correct size. I'll test that in a bit.
On Fri, Mar 13, 2020 at 11:36 AM Matt Dunavant <mdunavant@convokesystems.com> wrote:
Jason Dillaman wrote:
On Fri, Mar 13, 2020 at 11:17 AM Matt Dunavant <mdunavant(a)convokesystems.com> wrote:
I'm not sure of the last known good release of the rbd CLI where this worked. I just ran the sha1sum against the images and they always come up as different. Might be worth knowing, this is a volume that's provisioned at 512GB (with much less actually used) but after export, it only shows up as about 56GB.
The resulting image from the "rbd import" only shows up as 56GiB?
_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
Yeah, it's super odd. The actual content in the 512GB rbd image is probably about 50ish GB but the command isn't killing itself early or throwing any errors. I believe when I run the export to a middleman and then import, the image shows up as the correct size. I'll test that in a bit.
Couple test cases to try: Does the "rbd export" progress bar get to 100%? If you run "rbd export - > some_file" does it create a 512GiB file? If you run "rbd import" with a "--sparse-size 0" argument, does it change the result?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Jason Dillaman wrote:
On Fri, Mar 13, 2020 at 11:36 AM Matt Dunavant <mdunavant(a)convokesystems.com> wrote:
Jason Dillaman wrote:
On Fri, Mar 13, 2020 at 11:17 AM Matt Dunavant <mdunavant(a)convokesystems.com> wrote:
I'm not sure of the last known good release of the rbd CLI where this
worked. I just
ran the sha1sum against the images and they always come up as different. Might be worth knowing, this is a volume that's provisioned at 512GB (with much less actually used) but after export, it only shows up as about 56GB. The resulting image from the "rbd import" only shows up as 56GiB?
_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
Yeah, it's super odd. The actual content in the 512GB rbd image is probably about 50ish GB but the command isn't killing itself early or throwing any errors. I believe when I run the export to a middleman and then import, the image shows up as the correct size. I'll test that in a bit.
Couple test cases to try:
Does the "rbd export" progress bar get to 100%? If you run "rbd export - > some_file" does it create a 512GiB file? If you run "rbd import" with a "--sparse-size 0" argument, does it change the result?
_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
rbd export progress bar gets to 100% but there's some weird behavior. It'll jump immediately to 14%, wait for a bit, and then slowly climb to 100%
A few results: 1) rbd export of a snapshot results in an incorrectly sized drive but correct sha1sum. 2) rbd export > some_file creates a 512GB file and a correctly sized import and sha1sum, however the VM disk has some sort of corruption and the OS won't properly load. 3) rbd import with --sparse-size0 results in an incorrect sha1sum
On Fri, Mar 13, 2020 at 2:48 PM Matt Dunavant <mdunavant@convokesystems.com> wrote:
Jason Dillaman wrote:
On Fri, Mar 13, 2020 at 11:36 AM Matt Dunavant <mdunavant(a)convokesystems.com> wrote:
Jason Dillaman wrote:
On Fri, Mar 13, 2020 at 11:17 AM Matt Dunavant <mdunavant(a)convokesystems.com> wrote:
I'm not sure of the last known good release of the rbd CLI where this
worked. I just
ran the sha1sum against the images and they always come up as different. Might be worth knowing, this is a volume that's provisioned at 512GB (with much less actually used) but after export, it only shows up as about 56GB. The resulting image from the "rbd import" only shows up as 56GiB?
_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
Yeah, it's super odd. The actual content in the 512GB rbd image is probably about 50ish GB but the command isn't killing itself early or throwing any errors. I believe when I run the export to a middleman and then import, the image shows up as the correct size. I'll test that in a bit.
Couple test cases to try:
Does the "rbd export" progress bar get to 100%? If you run "rbd export - > some_file" does it create a 512GiB file? If you run "rbd import" with a "--sparse-size 0" argument, does it change the result?
_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
rbd export progress bar gets to 100% but there's some weird behavior. It'll jump immediately to 14%, wait for a bit, and then slowly climb to 100%
A few results:
1) rbd export of a snapshot results in an incorrectly sized drive but correct sha1sum.
The "export" image was the wrong size as compared to "rbd info [image]@[snap]"? What are you comparing the sha1sum against?
2) rbd export > some_file creates a 512GB file and a correctly sized import and sha1sum, however the VM disk has some sort of corruption and the OS won't properly load.
The sha1sum of the "some_file" matches or the sha1sum from re-exporting the newly imported image?
3) rbd import with --sparse-size0 results in an incorrect sha1sum _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
On Fri, Mar 13, 2020 at 3:31 PM Jason Dillaman <jdillama@redhat.com> wrote:
On Fri, Mar 13, 2020 at 2:48 PM Matt Dunavant <mdunavant@convokesystems.com> wrote:
Jason Dillaman wrote:
On Fri, Mar 13, 2020 at 11:36 AM Matt Dunavant <mdunavant(a)convokesystems.com> wrote:
Jason Dillaman wrote:
On Fri, Mar 13, 2020 at 11:17 AM Matt Dunavant <mdunavant(a)convokesystems.com> wrote:
I'm not sure of the last known good release of the rbd CLI where this
worked. I just
ran the sha1sum against the images and they always come up as different. Might be worth knowing, this is a volume that's provisioned at 512GB (with much less actually used) but after export, it only shows up as about 56GB. The resulting image from the "rbd import" only shows up as 56GiB?
_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
Yeah, it's super odd. The actual content in the 512GB rbd image is probably about 50ish GB but the command isn't killing itself early or throwing any errors. I believe when I run the export to a middleman and then import, the image shows up as the correct size. I'll test that in a bit.
Couple test cases to try:
Does the "rbd export" progress bar get to 100%? If you run "rbd export - > some_file" does it create a 512GiB file? If you run "rbd import" with a "--sparse-size 0" argument, does it change the result?
_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
rbd export progress bar gets to 100% but there's some weird behavior. It'll jump immediately to 14%, wait for a bit, and then slowly climb to 100%
A few results:
1) rbd export of a snapshot results in an incorrectly sized drive but correct sha1sum.
The "export" image was the wrong size as compared to "rbd info [image]@[snap]"? What are you comparing the sha1sum against?
Another thing to try would be to compare "rbd export --rbd_concurrent_management_ops=1 [image] - > [some_file]" against an "rbd export [image] [some_file]". The only change (since v14.2.3 for rbd import/export) was to speed-up exports to STDOUT by issuing concurrent I/O.
2) rbd export > some_file creates a 512GB file and a correctly sized import and sha1sum, however the VM disk has some sort of corruption and the OS won't properly load.
The sha1sum of the "some_file" matches or the sha1sum from re-exporting the newly imported image?
3) rbd import with --sparse-size0 results in an incorrect sha1sum _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
-- Jason
participants (4)
-
Jack
-
Jason Dillaman
-
Matt Dunavant
-
Simon Ironside