Ceph 18: Unable to delete image after imcomplete migration "image being migrated"
Hi Folks, I'm running Ceph 18 with OpenStack for my lab (and home services) in a 3 node cluster on Ubuntu 22.04. I'm quite new to these platforms. Just learning. This is my build, for what it's worth: https://blog.rhysgoodwin.com/it/openstack-ceph-hyperconverged/ I got myself into some trouble as follows. This is the sequence of events: I don't recall when but at some stage I must have tried an image migration from one pool to another. The source pool/image is infra-pool/sophosbuild I don't know what the target would have been. In any case on my travels, I found the infra-pool/sophosbuild image in the trash: rhys@hcn03:/imagework# rbd trash ls --all infra-pool 65a87bb2472fe sophosbuild I tried to delete it but got the following: rhys@hcn03:/imagework# rbd trash rm infra-pool/65a87bb2472fe 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::ImageState: 0x7f28a804b600 failed to open image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28a2ffd640 -1 librbd::image::RemoveRequest: 0x7f28a8000b90 handle_open_image: error opening image: (30) Read-only file system rbd: remove error: (30) Read-only file systemRemoving image: 0% complete...failed. Next, I tried to restore the image, and this also failed: rhys@hcn03:/imagework:# rbd trash restore infra-pool/65a87bb2472fe librbd::api::Trash: restore: Current trash source 'migration' does not match expected: user,mirroring,unknown (4) Probably stupidly, I followed the steps in this post: https://www.spinics.net/lists/ceph-users/msg72786.html to change offset 07 from 02 (TRASH_IMAGE_SOURCE_MIGRATION) in omap value to 00(TRASH_IMAGE_SOURCE_USER) After this I was able to restore the image successfully. However, I still could not delete it: rhys@hcn03:/imagework:# rbd rm infra-pool/sophosbuild 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::ImageState: 0x564d3f83d680 failed to open image: (30) Read-only file system Removing image: 0% complete...failed.rbd: delete error: (30) Read-only file system I tried to abort the migration with: root@hcn03:/imagework# rbd migration abort infra-pool/sophosbuild This took a few mins but failed at 99% (sorry, terminal scroll back lost) So now I'm stuck, I don't know how to get rid of this image and while everything is otherwise healthy in the cluster, the dashboard is throwing errors when it tries to enumerate the images in that pool. I'm considering migrating the good images off this pool and deleing the pool. But I don't even know if I'll be allowed to delete the pool while this issue is present. Any advice would be much appreciated. Kind regards, Rhys
Apologies, it's Ceph 17.2.6 not 18! I did try to deploy 18 on Ubuntu 22.04 but a bit beyond me until the packages are available. I'll wait to hear any suggestions, otherwise I'll procced with trying to migrate then delete the pool and report back. Ngā mihi Rhys ------- Original Message ------- On Monday, October 9th, 2023 at 9:52 AM, Rhys Goodwin <rhys.goodwin@proton.me> wrote:
Hi Folks,
I'm running Ceph 18 with OpenStack for my lab (and home services) in a 3 node cluster on Ubuntu 22.04. I'm quite new to these platforms. Just learning. This is my build, for what it's worth: https://blog.rhysgoodwin.com/it/openstack-ceph-hyperconverged/
I got myself into some trouble as follows. This is the sequence of events:
I don't recall when but at some stage I must have tried an image migration from one pool to another. The source pool/image is infra-pool/sophosbuild I don't know what the target would have been. In any case on my travels, I found the infra-pool/sophosbuild image in the trash: rhys@hcn03:/imagework# rbd trash ls --all infra-pool 65a87bb2472fe sophosbuild
I tried to delete it but got the following:
rhys@hcn03:/imagework# rbd trash rm infra-pool/65a87bb2472fe 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::ImageState: 0x7f28a804b600 failed to open image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28a2ffd640 -1 librbd::image::RemoveRequest: 0x7f28a8000b90 handle_open_image: error opening image: (30) Read-only file system rbd: remove error: (30) Read-only file systemRemoving image: 0% complete...failed.
Next, I tried to restore the image, and this also failed: rhys@hcn03:/imagework:# rbd trash restore infra-pool/65a87bb2472fe librbd::api::Trash: restore: Current trash source 'migration' does not match expected: user,mirroring,unknown (4)
Probably stupidly, I followed the steps in this post: https://www.spinics.net/lists/ceph-users/msg72786.html to change offset 07 from 02 (TRASH_IMAGE_SOURCE_MIGRATION) in omap value to 00(TRASH_IMAGE_SOURCE_USER)
After this I was able to restore the image successfully. However, I still could not delete it: rhys@hcn03:/imagework:# rbd rm infra-pool/sophosbuild 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::ImageState: 0x564d3f83d680 failed to open image: (30) Read-only file system Removing image: 0% complete...failed.rbd: delete error: (30) Read-only file system
I tried to abort the migration with: root@hcn03:/imagework# rbd migration abort infra-pool/sophosbuild This took a few mins but failed at 99% (sorry, terminal scroll back lost)
So now I'm stuck, I don't know how to get rid of this image and while everything is otherwise healthy in the cluster, the dashboard is throwing errors when it tries to enumerate the images in that pool.
I'm considering migrating the good images off this pool and deleing the pool. But I don't even know if I'll be allowed to delete the pool while this issue is present.
Any advice would be much appreciated.
Kind regards, Rhys
Thanks Eugen. root@hcn03:~# rbd status infra-pool/sophosbuild 2023-10-10T09:44:21.234+0000 7f1675c524c0 -1 librbd::Migration: open_images: failed to open destination image images/65d188c5f5a34: (2) No such file or directory rbd: getting migration status failed: (2) No such file or directory Watchers: none I've checked over the other pools again, but they only contain Openstack images. There are only 42 images in total across all pools. In fact, the "infra-pool" pool only has 3 images, including the faulty one. So migrating/re-creating is not a big deal. It's more just that I'd like to learn more about how to resolve such issues, if possible. Good call on the history. I found this smoking gun with: 'history |grep "rbd migration": rbd migration prepare infra-pool/sophosbuild images/sophosbuild rbd migration execute images/sophosbuild But images/sophosbuild is definitely not there anymore, and not in the trash. It looks like I was missing the commit. Kind regards, Rhys ------- Original Message ------- Eugen Block Wrote: Hi, there are a couple of things I would check before migrating all images. What's the current 'rbd status infra-pool/sophosbuild'? You probably don't have an infinite number of pools so I would also check if any of the other pools contains an image with the same name, just in case you wanted to keep its original name and only change the pool. Even if you don't have the terminal output, maybe you find some of the commands in the history?
Hi, I just re-read the docs on rbd migration [1], haven't done that in a while, and it states the following:
Note that the source image will be moved to the RBD trash to avoid mistaken usage during the migration process
So it was expected that your source image was in the trash during the migration, no need to restore. According to your history you also ran the "execute" command, do you remember if ran successfully as well? Did you "execute" after the prepare command completed? But you also state that the target image isn't there anymore, so it's hard to tell what exactly happened here. I'm not sure how to continue from here, maybe migrating/re-creating is the only way now. [1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/ Zitat von Rhys Goodwin <rhys.goodwin@proton.me>:
Thanks Eugen.
root@hcn03:~# rbd status infra-pool/sophosbuild 2023-10-10T09:44:21.234+0000 7f1675c524c0 -1 librbd::Migration: open_images: failed to open destination image images/65d188c5f5a34: (2) No such file or directory rbd: getting migration status failed: (2) No such file or directory Watchers: none
I've checked over the other pools again, but they only contain Openstack images. There are only 42 images in total across all pools. In fact, the "infra-pool" pool only has 3 images, including the faulty one. So migrating/re-creating is not a big deal. It's more just that I'd like to learn more about how to resolve such issues, if possible.
Good call on the history. I found this smoking gun with: 'history |grep "rbd migration": rbd migration prepare infra-pool/sophosbuild images/sophosbuild rbd migration execute images/sophosbuild
But images/sophosbuild is definitely not there anymore, and not in the trash. It looks like I was missing the commit.
Kind regards, Rhys
------- Original Message -------
Eugen Block Wrote:
Hi, there are a couple of things I would check before migrating all images. What's the current 'rbd status infra-pool/sophosbuild'? You probably don't have an infinite number of pools so I would also check if any of the other pools contains an image with the same name, just in case you wanted to keep its original name and only change the pool. Even if you don't have the terminal output, maybe you find some of the commands in the history? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks again Eugen. Looking at my command history it does look like I did execute the migration but didn't commit it. I wasn't surprised to see it in the trash based on the doc you mentioned, I only tried the restore as a desperate measure to clean up my mess. It doesn't help that I messed around like this, including with hexedit :O. I should have reached out before messing around. I'll proceed with the migrate/re-create and report back. I'm just crossing my fingers that I'll be allowed to delete the pool. It's a lesson to me to take more care of my wee cluster. Cheers, Rhys ------- Original Message ------- On Wednesday, October 11th, 2023 at 7:54 AM, Eugen Block <eblock@nde.ag> wrote:
Hi,
I just re-read the docs on rbd migration [1], haven't done that in a while, and it states the following:
Note that the source image will be moved to the RBD trash to avoid mistaken usage during the migration process
So it was expected that your source image was in the trash during the migration, no need to restore. According to your history you also ran the "execute" command, do you remember if ran successfully as well? Did you "execute" after the prepare command completed? But you also state that the target image isn't there anymore, so it's hard to tell what exactly happened here. I'm not sure how to continue from here, maybe migrating/re-creating is the only way now.
[1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/
Zitat von Rhys Goodwin rhys.goodwin@proton.me:
Thanks Eugen.
root@hcn03:~# rbd status infra-pool/sophosbuild 2023-10-10T09:44:21.234+0000 7f1675c524c0 -1 librbd::Migration: open_images: failed to open destination image images/65d188c5f5a34: (2) No such file or directory rbd: getting migration status failed: (2) No such file or directory Watchers: none
I've checked over the other pools again, but they only contain Openstack images. There are only 42 images in total across all pools. In fact, the "infra-pool" pool only has 3 images, including the faulty one. So migrating/re-creating is not a big deal. It's more just that I'd like to learn more about how to resolve such issues, if possible.
Good call on the history. I found this smoking gun with: 'history |grep "rbd migration": rbd migration prepare infra-pool/sophosbuild images/sophosbuild rbd migration execute images/sophosbuild
But images/sophosbuild is definitely not there anymore, and not in the trash. It looks like I was missing the commit.
Kind regards, Rhys
------- Original Message -------
Eugen Block Wrote:
Hi, there are a couple of things I would check before migrating all images. What's the current 'rbd status infra-pool/sophosbuild'? You probably don't have an infinite number of pools so I would also check if any of the other pools contains an image with the same name, just in case you wanted to keep its original name and only change the pool. Even if you don't have the terminal output, maybe you find some of the commands in the history? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, then I misinterpreted your message and thought you were actually surprised about the trash image. Yeah I don't think messing with hexedit really helped here, but I'm not sure either. Anyway, let us know how it went. Zitat von Rhys Goodwin <rhys.goodwin@proton.me>:
Thanks again Eugen. Looking at my command history it does look like I did execute the migration but didn't commit it. I wasn't surprised to see it in the trash based on the doc you mentioned, I only tried the restore as a desperate measure to clean up my mess. It doesn't help that I messed around like this, including with hexedit :O. I should have reached out before messing around.
I'll proceed with the migrate/re-create and report back. I'm just crossing my fingers that I'll be allowed to delete the pool. It's a lesson to me to take more care of my wee cluster.
Cheers, Rhys
------- Original Message ------- On Wednesday, October 11th, 2023 at 7:54 AM, Eugen Block <eblock@nde.ag> wrote:
Hi,
I just re-read the docs on rbd migration [1], haven't done that in a while, and it states the following:
Note that the source image will be moved to the RBD trash to avoid mistaken usage during the migration process
So it was expected that your source image was in the trash during the migration, no need to restore. According to your history you also ran the "execute" command, do you remember if ran successfully as well? Did you "execute" after the prepare command completed? But you also state that the target image isn't there anymore, so it's hard to tell what exactly happened here. I'm not sure how to continue from here, maybe migrating/re-creating is the only way now.
[1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/
Zitat von Rhys Goodwin rhys.goodwin@proton.me:
Thanks Eugen.
root@hcn03:~# rbd status infra-pool/sophosbuild 2023-10-10T09:44:21.234+0000 7f1675c524c0 -1 librbd::Migration: open_images: failed to open destination image images/65d188c5f5a34: (2) No such file or directory rbd: getting migration status failed: (2) No such file or directory Watchers: none
I've checked over the other pools again, but they only contain Openstack images. There are only 42 images in total across all pools. In fact, the "infra-pool" pool only has 3 images, including the faulty one. So migrating/re-creating is not a big deal. It's more just that I'd like to learn more about how to resolve such issues, if possible.
Good call on the history. I found this smoking gun with: 'history |grep "rbd migration": rbd migration prepare infra-pool/sophosbuild images/sophosbuild rbd migration execute images/sophosbuild
But images/sophosbuild is definitely not there anymore, and not in the trash. It looks like I was missing the commit.
Kind regards, Rhys
------- Original Message -------
Eugen Block Wrote:
Hi, there are a couple of things I would check before migrating all images. What's the current 'rbd status infra-pool/sophosbuild'? You probably don't have an infinite number of pools so I would also check if any of the other pools contains an image with the same name, just in case you wanted to keep its original name and only change the pool. Even if you don't have the terminal output, maybe you find some of the commands in the history? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Eugen. Operation complete: root@hcn03:/imagework# ceph osd pool delete infra-pool infra-pool --yes-i-really-really-mean-it pool 'infra-pool' removed Everything clean and tidy again. Thanks for your help and support. ------- Original Message ------- On Wednesday, October 11th, 2023 at 7:21 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
then I misinterpreted your message and thought you were actually surprised about the trash image. Yeah I don't think messing with hexedit really helped here, but I'm not sure either. Anyway, let us know how it went.
Zitat von Rhys Goodwin rhys.goodwin@proton.me:
Thanks again Eugen. Looking at my command history it does look like I did execute the migration but didn't commit it. I wasn't surprised to see it in the trash based on the doc you mentioned, I only tried the restore as a desperate measure to clean up my mess. It doesn't help that I messed around like this, including with hexedit :O. I should have reached out before messing around.
I'll proceed with the migrate/re-create and report back. I'm just crossing my fingers that I'll be allowed to delete the pool. It's a lesson to me to take more care of my wee cluster.
Cheers, Rhys
------- Original Message ------- On Wednesday, October 11th, 2023 at 7:54 AM, Eugen Block eblock@nde.ag wrote:
Hi,
I just re-read the docs on rbd migration [1], haven't done that in a while, and it states the following:
Note that the source image will be moved to the RBD trash to avoid mistaken usage during the migration process
So it was expected that your source image was in the trash during the migration, no need to restore. According to your history you also ran the "execute" command, do you remember if ran successfully as well? Did you "execute" after the prepare command completed? But you also state that the target image isn't there anymore, so it's hard to tell what exactly happened here. I'm not sure how to continue from here, maybe migrating/re-creating is the only way now.
[1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/
Zitat von Rhys Goodwin rhys.goodwin@proton.me:
Thanks Eugen.
root@hcn03:~# rbd status infra-pool/sophosbuild 2023-10-10T09:44:21.234+0000 7f1675c524c0 -1 librbd::Migration: open_images: failed to open destination image images/65d188c5f5a34: (2) No such file or directory rbd: getting migration status failed: (2) No such file or directory Watchers: none
I've checked over the other pools again, but they only contain Openstack images. There are only 42 images in total across all pools. In fact, the "infra-pool" pool only has 3 images, including the faulty one. So migrating/re-creating is not a big deal. It's more just that I'd like to learn more about how to resolve such issues, if possible.
Good call on the history. I found this smoking gun with: 'history |grep "rbd migration": rbd migration prepare infra-pool/sophosbuild images/sophosbuild rbd migration execute images/sophosbuild
But images/sophosbuild is definitely not there anymore, and not in the trash. It looks like I was missing the commit.
Kind regards, Rhys
------- Original Message -------
Eugen Block Wrote:
Hi, there are a couple of things I would check before migrating all images. What's the current 'rbd status infra-pool/sophosbuild'? You probably don't have an infinite number of pools so I would also check if any of the other pools contains an image with the same name, just in case you wanted to keep its original name and only change the pool. Even if you don't have the terminal output, maybe you find some of the commands in the history? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, there are a couple of things I would check before migrating all images. What's the current 'rbd status infra-pool/sophosbuild'? You probably don't have an infinite number of pools so I would also check if any of the other pools contains an image with the same name, just in case you wanted to keep its original name and only change the pool. Even if you don't have the terminal output, maybe you find some of the commands in the history? Zitat von Rhys Goodwin <rhys.goodwin@proton.me>:
Hi Folks,
I'm running Ceph 18 with OpenStack for my lab (and home services) in a 3 node cluster on Ubuntu 22.04. I'm quite new to these platforms. Just learning. This is my build, for what it's worth: https://blog.rhysgoodwin.com/it/openstack-ceph-hyperconverged/
I got myself into some trouble as follows. This is the sequence of events:
I don't recall when but at some stage I must have tried an image migration from one pool to another. The source pool/image is infra-pool/sophosbuild I don't know what the target would have been. In any case on my travels, I found the infra-pool/sophosbuild image in the trash: rhys@hcn03:/imagework# rbd trash ls --all infra-pool 65a87bb2472fe sophosbuild
I tried to delete it but got the following:
rhys@hcn03:/imagework# rbd trash rm infra-pool/65a87bb2472fe 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::ImageState: 0x7f28a804b600 failed to open image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28a2ffd640 -1 librbd::image::RemoveRequest: 0x7f28a8000b90 handle_open_image: error opening image: (30) Read-only file system rbd: remove error: (30) Read-only file systemRemoving image: 0% complete...failed.
Next, I tried to restore the image, and this also failed: rhys@hcn03:/imagework:# rbd trash restore infra-pool/65a87bb2472fe librbd::api::Trash: restore: Current trash source 'migration' does not match expected: user,mirroring,unknown (4)
Probably stupidly, I followed the steps in this post: https://www.spinics.net/lists/ceph-users/msg72786.html to change offset 07 from 02 (TRASH_IMAGE_SOURCE_MIGRATION) in omap value to 00(TRASH_IMAGE_SOURCE_USER)
After this I was able to restore the image successfully. However, I still could not delete it: rhys@hcn03:/imagework:# rbd rm infra-pool/sophosbuild 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::ImageState: 0x564d3f83d680 failed to open image: (30) Read-only file system Removing image: 0% complete...failed.rbd: delete error: (30) Read-only file system
I tried to abort the migration with: root@hcn03:/imagework# rbd migration abort infra-pool/sophosbuild This took a few mins but failed at 99% (sorry, terminal scroll back lost)
So now I'm stuck, I don't know how to get rid of this image and while everything is otherwise healthy in the cluster, the dashboard is throwing errors when it tries to enumerate the images in that pool.
I'm considering migrating the good images off this pool and deleing the pool. But I don't even know if I'll be allowed to delete the pool while this issue is present.
Any advice would be much appreciated.
Kind regards, Rhys _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Eugen. root@hcn03:~# rbd status infra-pool/sophosbuild 2023-10-10T09:44:21.234+0000 7f1675c524c0 -1 librbd::Migration: open_images: failed to open destination image images/65d188c5f5a34: (2) No such file or directory rbd: getting migration status failed: (2) No such file or directory Watchers: none I've checked over the other pools again, but they are all Openstack images. There are only 42 images in total across all pools. In fact, the infra-pool only has 3 images including the faulty one. So, migrating/re-creating is not a big deal. It's more just that I'd like to learn more about how to resolve such issues, if possible. Good call on the history. I found this smoking gun with: 'history |grep "rbd migration": rbd migration prepare infra-pool/sophosbuild images/sophosbuild rbd migration execute images/sophosbuild But images/sophosbuild is not there anymore, and not in the trash. It looks like I was missing the commit.
participants (3)
-
Eugen Block
-
Rhys Goodwin
-
rhys.goodwin＠proton.me