Hi, This is really not nice to be exploring like this. I think this is also the same kind of nightmare when doing failover and trying to map PVC from kube to RBD image. I have a suggestion : implement labels on RBD images ! I don't have a clue if it's already planned somewhere, but it would be really helpful. Regards Le ven. 20 juin 2025 à 20:08, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I think you will need to figure out which objects belong to the orphaned image and delete them on the rados level. Here's an example of the rados objects in a fresh pool with just this image in it (note that it's a tiny image to keep the output brief):
---snip--- for i in $(rados -p rbd ls); do echo "Object: $i"; rados -p rbd listomapkeys $i; echo -e "\n";done Object: rbd_object_map.42f317c94344
Object: rbd_directory id_42f317c94344 name_image1
Object: rbd_info
Object: rbd_data.42f317c94344.0000000000000000
Object: rbd_trash
Object: rbd_header.42f317c94344 access_timestamp create_timestamp features modify_timestamp object_prefix order size snap_seq
Object: rbd_id.image1 ---snip---
So you'll have to find out which block_name_prefix belongs to the orphaned image (by mapping the existing images to their rbd_data objects). If you found the correct image name/prefix etc., you'll have to remove the rbd_header object, the rbd_id object, all rbd_data objects (from the EC pool), the rbd_object_map objects and finally, clean up the rbd_directory by removing the corresponding omapkey (rados -p pool rmomapkey rbd_directory <image_id>|<name_>). Be very careful what you're doing, take backups of the omapkeys/vals, just to be safe. Double and triple check your actions so you don't remove the wrong objects. If you have a test environment, I'd recommend to practice there first.
Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data
On 6/10/25 08:47, Gaël THEROND wrote: pool
have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io