Hi, This is reason why you should choose Cinder, k8s orphans is not a problem of Ceph as Backend software k Sent from my iPhone
On 22 Jun 2025, at 17:37, Julien Laurenceau <julien.laurenceau@pepitedata.com> wrote:
Hi,
Very simple example: imagine you install ceph-csi-rbd on Kuberentes and you configured the storageClass on Kubernetes with reclaimPolicy=Retain. When the user deletes the PVC, the PV is still there but it is not Bound : on ceph side the rbd image is also not bound. Let's imagine you want to clean orphan images on the ceph side : images that do not belong to any existing PV... that's a nightmare.
Orphan images may appear for example on test infrastructure when you spin frequently new kubernetes clusters. If you do not delete all PV before deleting the cluster, then you have to clean the image on ceph directly.
Regards
Le sam. 21 juin 2025 à 16:12, Eugen Block <eblock@nde.ag> a écrit :
I agree, it’s not something one likes to do. And in 10 years using Ceph, I have never had to go through this procedure on a production cluster. So the question is, how does that happen?
Zitat von Julien Laurenceau <julien.laurenceau@pepitedata.com>:
Hi,
This is really not nice to be exploring like this. I think this is also the same kind of nightmare when doing failover and trying to map PVC from kube to RBD image.
I have a suggestion : implement labels on RBD images ! I don't have a clue if it's already planned somewhere, but it would be really helpful.
Regards
Le ven. 20 juin 2025 à 20:08, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I think you will need to figure out which objects belong to the orphaned image and delete them on the rados level. Here's an example of the rados objects in a fresh pool with just this image in it (note that it's a tiny image to keep the output brief):
---snip--- for i in $(rados -p rbd ls); do echo "Object: $i"; rados -p rbd listomapkeys $i; echo -e "\n";done Object: rbd_object_map.42f317c94344
Object: rbd_directory id_42f317c94344 name_image1
Object: rbd_info
Object: rbd_data.42f317c94344.0000000000000000
Object: rbd_trash
Object: rbd_header.42f317c94344 access_timestamp create_timestamp features modify_timestamp object_prefix order size snap_seq
Object: rbd_id.image1 ---snip---
So you'll have to find out which block_name_prefix belongs to the orphaned image (by mapping the existing images to their rbd_data objects). If you found the correct image name/prefix etc., you'll have to remove the rbd_header object, the rbd_id object, all rbd_data objects (from the EC pool), the rbd_object_map objects and finally, clean up the rbd_directory by removing the corresponding omapkey (rados -p pool rmomapkey rbd_directory <image_id>|<name_>). Be very careful what you're doing, take backups of the omapkeys/vals, just to be safe. Double and triple check your actions so you don't remove the wrong objects. If you have a test environment, I'd recommend to practice there first.
Zitat von Gaël THEROND <gael.therond@bitswalk.com>:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
On 6/10/25 08:47, Gaël THEROND wrote: > Hi there! Nope it is not in the trash pool. > > Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit : > >> Is that image in the trash? >> >> `rbd -p pool trash ls` >> >> Zitat von Gaël THEROND<gael.therond@bitswalk.com>: >> >>> Hi folks, >>> >>> I've a quick question. On one of our pool we found out an image that >>> doesn't exist anymore physically (This image doesn't exist, have no snap >>> attached, is not parent of another image) but is still listed when >>> performing a `rbd -p pool ls`. However, it error with a nice "Error >> opening >>> image <image_name>: (2) No such file or directory" when we try to delete >> it >>> using `rbd -p pool rm <image_name>`. >>> >>> This pool is a EC based pool and neither the metadata nor the data pool >>> have any data of that image remaining. >>> >>> So my question is, is there a cli way to force ceph to forget/abandon >> this >>> image? A command which isn't involving manually manipulating the various >>> maps would be preferred but if I had to dig that deep I can. >>> >>> Thanks! >>> _______________________________________________ >>> ceph-users mailing list --ceph-users@ceph.io >>> To unsubscribe send an email toceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list --ceph-users@ceph.io >> To unsubscribe send an email toceph-users-leave@ceph.io >> > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io