Force remove undeletable image on RBD pool?
Hi folks, I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`. This pool is a EC based pool and neither the metadata nor the data pool have any data of that image remaining. So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can. Thanks!
Is that image in the trash? `rbd -p pool trash ls` Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data pool have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi there! Nope it is not in the trash pool. Le jeu. 5 juin 2025 à 12:37, Eugen Block <eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data pool have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Gael, You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects): 1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023 2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e 3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm 4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a Cheers, Enrico On 6/10/25 08:47, Gaël THEROND wrote:
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data pool have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command. I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool. If have already tried that, I'm all ears out ;-) Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
On 6/10/25 08:47, Gaël THEROND wrote:
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data pool have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I think you will need to figure out which objects belong to the orphaned image and delete them on the rados level. Here's an example of the rados objects in a fresh pool with just this image in it (note that it's a tiny image to keep the output brief): ---snip--- for i in $(rados -p rbd ls); do echo "Object: $i"; rados -p rbd listomapkeys $i; echo -e "\n";done Object: rbd_object_map.42f317c94344 Object: rbd_directory id_42f317c94344 name_image1 Object: rbd_info Object: rbd_data.42f317c94344.0000000000000000 Object: rbd_trash Object: rbd_header.42f317c94344 access_timestamp create_timestamp features modify_timestamp object_prefix order size snap_seq Object: rbd_id.image1 ---snip--- So you'll have to find out which block_name_prefix belongs to the orphaned image (by mapping the existing images to their rbd_data objects). If you found the correct image name/prefix etc., you'll have to remove the rbd_header object, the rbd_id object, all rbd_data objects (from the EC pool), the rbd_object_map objects and finally, clean up the rbd_directory by removing the corresponding omapkey (rados -p pool rmomapkey rbd_directory <image_id>|<name_>). Be very careful what you're doing, take backups of the omapkeys/vals, just to be safe. Double and triple check your actions so you don't remove the wrong objects. If you have a test environment, I'd recommend to practice there first. Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
On 6/10/25 08:47, Gaël THEROND wrote:
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data pool have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, This is really not nice to be exploring like this. I think this is also the same kind of nightmare when doing failover and trying to map PVC from kube to RBD image. I have a suggestion : implement labels on RBD images ! I don't have a clue if it's already planned somewhere, but it would be really helpful. Regards Le ven. 20 juin 2025 à 20:08, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I think you will need to figure out which objects belong to the orphaned image and delete them on the rados level. Here's an example of the rados objects in a fresh pool with just this image in it (note that it's a tiny image to keep the output brief):
---snip--- for i in $(rados -p rbd ls); do echo "Object: $i"; rados -p rbd listomapkeys $i; echo -e "\n";done Object: rbd_object_map.42f317c94344
Object: rbd_directory id_42f317c94344 name_image1
Object: rbd_info
Object: rbd_data.42f317c94344.0000000000000000
Object: rbd_trash
Object: rbd_header.42f317c94344 access_timestamp create_timestamp features modify_timestamp object_prefix order size snap_seq
Object: rbd_id.image1 ---snip---
So you'll have to find out which block_name_prefix belongs to the orphaned image (by mapping the existing images to their rbd_data objects). If you found the correct image name/prefix etc., you'll have to remove the rbd_header object, the rbd_id object, all rbd_data objects (from the EC pool), the rbd_object_map objects and finally, clean up the rbd_directory by removing the corresponding omapkey (rados -p pool rmomapkey rbd_directory <image_id>|<name_>). Be very careful what you're doing, take backups of the omapkeys/vals, just to be safe. Double and triple check your actions so you don't remove the wrong objects. If you have a test environment, I'd recommend to practice there first.
Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data
On 6/10/25 08:47, Gaël THEROND wrote: pool
have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I agree, it’s not something one likes to do. And in 10 years using Ceph, I have never had to go through this procedure on a production cluster. So the question is, how does that happen? Zitat von Julien Laurenceau <julien.laurenceau@pepitedata.com>:
Hi,
This is really not nice to be exploring like this. I think this is also the same kind of nightmare when doing failover and trying to map PVC from kube to RBD image.
I have a suggestion : implement labels on RBD images ! I don't have a clue if it's already planned somewhere, but it would be really helpful.
Regards
Le ven. 20 juin 2025 à 20:08, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I think you will need to figure out which objects belong to the orphaned image and delete them on the rados level. Here's an example of the rados objects in a fresh pool with just this image in it (note that it's a tiny image to keep the output brief):
---snip--- for i in $(rados -p rbd ls); do echo "Object: $i"; rados -p rbd listomapkeys $i; echo -e "\n";done Object: rbd_object_map.42f317c94344
Object: rbd_directory id_42f317c94344 name_image1
Object: rbd_info
Object: rbd_data.42f317c94344.0000000000000000
Object: rbd_trash
Object: rbd_header.42f317c94344 access_timestamp create_timestamp features modify_timestamp object_prefix order size snap_seq
Object: rbd_id.image1 ---snip---
So you'll have to find out which block_name_prefix belongs to the orphaned image (by mapping the existing images to their rbd_data objects). If you found the correct image name/prefix etc., you'll have to remove the rbd_header object, the rbd_id object, all rbd_data objects (from the EC pool), the rbd_object_map objects and finally, clean up the rbd_directory by removing the corresponding omapkey (rados -p pool rmomapkey rbd_directory <image_id>|<name_>). Be very careful what you're doing, take backups of the omapkeys/vals, just to be safe. Double and triple check your actions so you don't remove the wrong objects. If you have a test environment, I'd recommend to practice there first.
Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
> Hi folks, > > I've a quick question. On one of our pool we found out an image that > doesn't exist anymore physically (This image doesn't exist, have no snap > attached, is not parent of another image) but is still listed when > performing a `rbd -p pool ls`. However, it error with a nice "Error opening > image <image_name>: (2) No such file or directory" when we try to delete it > using `rbd -p pool rm <image_name>`. > > This pool is a EC based pool and neither the metadata nor the data
On 6/10/25 08:47, Gaël THEROND wrote: pool
> have any data of that image remaining. > > So my question is, is there a cli way to force ceph to forget/abandon this > image? A command which isn't involving manually manipulating the various > maps would be preferred but if I had to dig that deep I can. > > Thanks! > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Very simple example: imagine you install ceph-csi-rbd on Kuberentes and you configured the storageClass on Kubernetes with reclaimPolicy=Retain. When the user deletes the PVC, the PV is still there but it is not Bound : on ceph side the rbd image is also not bound. Let's imagine you want to clean orphan images on the ceph side : images that do not belong to any existing PV... that's a nightmare. Orphan images may appear for example on test infrastructure when you spin frequently new kubernetes clusters. If you do not delete all PV before deleting the cluster, then you have to clean the image on ceph directly. Regards Le sam. 21 juin 2025 à 16:12, Eugen Block <eblock@nde.ag> a écrit :
I agree, it’s not something one likes to do. And in 10 years using Ceph, I have never had to go through this procedure on a production cluster. So the question is, how does that happen?
Zitat von Julien Laurenceau <julien.laurenceau@pepitedata.com>:
Hi,
This is really not nice to be exploring like this. I think this is also the same kind of nightmare when doing failover and trying to map PVC from kube to RBD image.
I have a suggestion : implement labels on RBD images ! I don't have a clue if it's already planned somewhere, but it would be really helpful.
Regards
Le ven. 20 juin 2025 à 20:08, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I think you will need to figure out which objects belong to the orphaned image and delete them on the rados level. Here's an example of the rados objects in a fresh pool with just this image in it (note that it's a tiny image to keep the output brief):
---snip--- for i in $(rados -p rbd ls); do echo "Object: $i"; rados -p rbd listomapkeys $i; echo -e "\n";done Object: rbd_object_map.42f317c94344
Object: rbd_directory id_42f317c94344 name_image1
Object: rbd_info
Object: rbd_data.42f317c94344.0000000000000000
Object: rbd_trash
Object: rbd_header.42f317c94344 access_timestamp create_timestamp features modify_timestamp object_prefix order size snap_seq
Object: rbd_id.image1 ---snip---
So you'll have to find out which block_name_prefix belongs to the orphaned image (by mapping the existing images to their rbd_data objects). If you found the correct image name/prefix etc., you'll have to remove the rbd_header object, the rbd_id object, all rbd_data objects (from the EC pool), the rbd_object_map objects and finally, clean up the rbd_directory by removing the corresponding omapkey (rados -p pool rmomapkey rbd_directory <image_id>|<name_>). Be very careful what you're doing, take backups of the omapkeys/vals, just to be safe. Double and triple check your actions so you don't remove the wrong objects. If you have a test environment, I'd recommend to practice there first.
Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
> Is that image in the trash? > > `rbd -p pool trash ls` > > Zitat von Gaël THEROND<gael.therond@bitswalk.com>: > >> Hi folks, >> >> I've a quick question. On one of our pool we found out an image
>> doesn't exist anymore physically (This image doesn't exist, have no snap >> attached, is not parent of another image) but is still listed when >> performing a `rbd -p pool ls`. However, it error with a nice "Error > opening >> image <image_name>: (2) No such file or directory" when we try to delete > it >> using `rbd -p pool rm <image_name>`. >> >> This pool is a EC based pool and neither the metadata nor the data
On 6/10/25 08:47, Gaël THEROND wrote: that pool
>> have any data of that image remaining. >> >> So my question is, is there a cli way to force ceph to forget/abandon > this >> image? A command which isn't involving manually manipulating the various >> maps would be preferred but if I had to dig that deep I can. >> >> Thanks! >> _______________________________________________ >> ceph-users mailing list --ceph-users@ceph.io >> To unsubscribe send an email toceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io > _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, This is reason why you should choose Cinder, k8s orphans is not a problem of Ceph as Backend software k Sent from my iPhone
On 22 Jun 2025, at 17:37, Julien Laurenceau <julien.laurenceau@pepitedata.com> wrote:
Hi,
Very simple example: imagine you install ceph-csi-rbd on Kuberentes and you configured the storageClass on Kubernetes with reclaimPolicy=Retain. When the user deletes the PVC, the PV is still there but it is not Bound : on ceph side the rbd image is also not bound. Let's imagine you want to clean orphan images on the ceph side : images that do not belong to any existing PV... that's a nightmare.
Orphan images may appear for example on test infrastructure when you spin frequently new kubernetes clusters. If you do not delete all PV before deleting the cluster, then you have to clean the image on ceph directly.
Regards
Le sam. 21 juin 2025 à 16:12, Eugen Block <eblock@nde.ag> a écrit :
I agree, it’s not something one likes to do. And in 10 years using Ceph, I have never had to go through this procedure on a production cluster. So the question is, how does that happen?
Zitat von Julien Laurenceau <julien.laurenceau@pepitedata.com>:
Hi,
This is really not nice to be exploring like this. I think this is also the same kind of nightmare when doing failover and trying to map PVC from kube to RBD image.
I have a suggestion : implement labels on RBD images ! I don't have a clue if it's already planned somewhere, but it would be really helpful.
Regards
Le ven. 20 juin 2025 à 20:08, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I think you will need to figure out which objects belong to the orphaned image and delete them on the rados level. Here's an example of the rados objects in a fresh pool with just this image in it (note that it's a tiny image to keep the output brief):
---snip--- for i in $(rados -p rbd ls); do echo "Object: $i"; rados -p rbd listomapkeys $i; echo -e "\n";done Object: rbd_object_map.42f317c94344
Object: rbd_directory id_42f317c94344 name_image1
Object: rbd_info
Object: rbd_data.42f317c94344.0000000000000000
Object: rbd_trash
Object: rbd_header.42f317c94344 access_timestamp create_timestamp features modify_timestamp object_prefix order size snap_seq
Object: rbd_id.image1 ---snip---
So you'll have to find out which block_name_prefix belongs to the orphaned image (by mapping the existing images to their rbd_data objects). If you found the correct image name/prefix etc., you'll have to remove the rbd_header object, the rbd_id object, all rbd_data objects (from the EC pool), the rbd_object_map objects and finally, clean up the rbd_directory by removing the corresponding omapkey (rados -p pool rmomapkey rbd_directory <image_id>|<name_>). Be very careful what you're doing, take backups of the omapkeys/vals, just to be safe. Double and triple check your actions so you don't remove the wrong objects. If you have a test environment, I'd recommend to practice there first.
Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
On 6/10/25 08:47, Gaël THEROND wrote: > Hi there! Nope it is not in the trash pool. > > Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit : > >> Is that image in the trash? >> >> `rbd -p pool trash ls` >> >> Zitat von Gaël THEROND<gael.therond@bitswalk.com>: >> >>> Hi folks, >>> >>> I've a quick question. On one of our pool we found out an image that >>> doesn't exist anymore physically (This image doesn't exist, have no snap >>> attached, is not parent of another image) but is still listed when >>> performing a `rbd -p pool ls`. However, it error with a nice "Error >> opening >>> image <image_name>: (2) No such file or directory" when we try to delete >> it >>> using `rbd -p pool rm <image_name>`. >>> >>> This pool is a EC based pool and neither the metadata nor the data pool >>> have any data of that image remaining. >>> >>> So my question is, is there a cli way to force ceph to forget/abandon >> this >>> image? A command which isn't involving manually manipulating the various >>> maps would be preferred but if I had to dig that deep I can. >>> >>> Thanks! >>> _______________________________________________ >>> ceph-users mailing list --ceph-users@ceph.io >>> To unsubscribe send an email toceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list --ceph-users@ceph.io >> To unsubscribe send an email toceph-users-leave@ceph.io >> > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
hello, when you delete the pvc, you still have a pv. In the pv definition you have the image name. Before cleaning the pv, get the image name and delete it in ceph pool. I manage doozen of k8s cluster in this way with rook and it's working pretty good. I wrote a script to do it once a week, once a month according to the customer needs. oau Le dimanche 22 juin 2025 à 16:34 +0200, Julien Laurenceau a écrit :
Hi,
Very simple example: imagine you install ceph-csi-rbd on Kuberentes and you configured the storageClass on Kubernetes with reclaimPolicy=Retain. When the user deletes the PVC, the PV is still there but it is not Bound : on ceph side the rbd image is also not bound. Let's imagine you want to clean orphan images on the ceph side : images that do not belong to any existing PV... that's a nightmare.
Orphan images may appear for example on test infrastructure when you spin frequently new kubernetes clusters. If you do not delete all PV before deleting the cluster, then you have to clean the image on ceph directly.
Regards
Le sam. 21 juin 2025 à 16:12, Eugen Block <eblock@nde.ag> a écrit :
I agree, it’s not something one likes to do. And in 10 years using Ceph, I have never had to go through this procedure on a production cluster. So the question is, how does that happen?
Zitat von Julien Laurenceau <julien.laurenceau@pepitedata.com>:
Hi,
This is really not nice to be exploring like this. I think this is also the same kind of nightmare when doing failover and trying to map PVC from kube to RBD image.
I have a suggestion : implement labels on RBD images ! I don't have a clue if it's already planned somewhere, but it would be really helpful.
Regards
Le ven. 20 juin 2025 à 20:08, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I think you will need to figure out which objects belong to the orphaned image and delete them on the rados level. Here's an example of the rados objects in a fresh pool with just this image in it (note that it's a tiny image to keep the output brief):
---snip--- for i in $(rados -p rbd ls); do echo "Object: $i"; rados -p rbd listomapkeys $i; echo -e "\n";done Object: rbd_object_map.42f317c94344
Object: rbd_directory id_42f317c94344 name_image1
Object: rbd_info
Object: rbd_data.42f317c94344.0000000000000000
Object: rbd_trash
Object: rbd_header.42f317c94344 access_timestamp create_timestamp features modify_timestamp object_prefix order size snap_seq
Object: rbd_id.image1 ---snip---
So you'll have to find out which block_name_prefix belongs to the orphaned image (by mapping the existing images to their rbd_data objects). If you found the correct image name/prefix etc., you'll have to remove the rbd_header object, the rbd_id object, all rbd_data objects (from the EC pool), the rbd_object_map objects and finally, clean up the rbd_directory by removing the corresponding omapkey (rados -p pool rmomapkey rbd_directory <image_id>|<name_>). Be very careful what you're doing, take backups of the omapkeys/vals, just to be safe. Double and triple check your actions so you don't remove the wrong objects. If you have a test environment, I'd recommend to practice there first.
Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast- diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc- 6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
On 6/10/25 08:47, Gaël THEROND wrote: > Hi there! Nope it is not in the trash pool. > > Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a > écrit : > > > Is that image in the trash? > > > > `rbd -p pool trash ls` > > > > Zitat von Gaël THEROND<gael.therond@bitswalk.com>: > > > > > Hi folks, > > > > > > I've a quick question. On one of our pool we found > > > out an image that > > > doesn't exist anymore physically (This image doesn't > > > exist, have no snap > > > attached, is not parent of another image) but is > > > still listed when > > > performing a `rbd -p pool ls`. However, it error with > > > a nice "Error > > opening > > > image <image_name>: (2) No such file or directory" > > > when we try to delete > > it > > > using `rbd -p pool rm <image_name>`. > > > > > > This pool is a EC based pool and neither the metadata > > > nor the data pool > > > have any data of that image remaining. > > > > > > So my question is, is there a cli way to force ceph > > > to forget/abandon > > this > > > image? A command which isn't involving manually > > > manipulating the various > > > maps would be preferred but if I had to dig that deep > > > I can. > > > > > > Thanks! > > > _______________________________________________ > > > ceph-users mailing list --ceph-users@ceph.io > > > To unsubscribe send an email > > > toceph-users-leave@ceph.io > > > > _______________________________________________ > > ceph-users mailing list --ceph-users@ceph.io > > To unsubscribe send an email toceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have a similar problem with some RBDs being deleted but still appearing in ls. To remove the stuck rbd I do the following: 1. rbd -p <pool> <rbd_name> --debug_ms 1/1 2>&1 | grep header | tail 2. I can see that an OSD is listed in the output on the last line. 3. I restart that OSD 4. Delete the rbd I'm not sure if it is the same thing but it doesn't hurt to try. On Fri, Jun 13, 2025 at 4:34 AM Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data
On 6/10/25 08:47, Gaël THEROND wrote: pool
have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi folks! Thanks a lot for this thread! Awesome interactions. As Enrico suggested, I'll goes with the deep remove by comparing what is existing and what's not, but honestly my goal when I wrote that message was indeed to avoid having to touch data this way. I already knew the answer in terms of how could I do it by suppressing data which are basically orphaned as I already deeply explored the way data are stored and all, but I was really hoping that a more "official" way to do it was implemented. I really think that a specific group of orphaned_data commands such as: orphans plan (look for orphans) orphans plan apply (remove actually found orphans) orphans schedule (recurrent orphans plan scheduling) That would at least give operators a way to cleanup things like index/metadata with missing data, data with no longer existing index/metadata etc but with a bit more confidence than a manual for loop operation that is prone to error from humans. All in all, it's great to see that I'm not the only one concerned with such tasks :-) Thanks a lot for all those answers! PS: I'll do the cleaning manually and while doing that investigate to find out the root cause of this situation as we only have one image with such status over 2Pb of data. PS2: This CEPH cluster do host an Openstack platform that itself host K8s clusters and other various workloads, none of those workloads (k8s included) access the ceph cluster directly, they all pass by the openstack layer one way or another. Le lun. 23 juin 2025 à 02:30, Reid Guyett <reid.guyett@gmail.com> a écrit :
I have a similar problem with some RBDs being deleted but still appearing in ls. To remove the stuck rbd I do the following: 1. rbd -p <pool> <rbd_name> --debug_ms 1/1 2>&1 | grep header | tail 2. I can see that an OSD is listed in the output on the last line. 3. I restart that OSD 4. Delete the rbd
I'm not sure if it is the same thing but it doesn't hurt to try.
On Fri, Jun 13, 2025 at 4:34 AM Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data
On 6/10/25 08:47, Gaël THEROND wrote: pool
have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Enrico Bocchi
-
Eugen Block
-
Gaël THEROND
-
Julien Laurenceau
-
Konstantin Shalygin
-
Olivier AUDRY
-
Reid Guyett