Hi folks! Thanks a lot for this thread! Awesome interactions. As Enrico suggested, I'll goes with the deep remove by comparing what is existing and what's not, but honestly my goal when I wrote that message was indeed to avoid having to touch data this way. I already knew the answer in terms of how could I do it by suppressing data which are basically orphaned as I already deeply explored the way data are stored and all, but I was really hoping that a more "official" way to do it was implemented. I really think that a specific group of orphaned_data commands such as: orphans plan (look for orphans) orphans plan apply (remove actually found orphans) orphans schedule (recurrent orphans plan scheduling) That would at least give operators a way to cleanup things like index/metadata with missing data, data with no longer existing index/metadata etc but with a bit more confidence than a manual for loop operation that is prone to error from humans. All in all, it's great to see that I'm not the only one concerned with such tasks :-) Thanks a lot for all those answers! PS: I'll do the cleaning manually and while doing that investigate to find out the root cause of this situation as we only have one image with such status over 2Pb of data. PS2: This CEPH cluster do host an Openstack platform that itself host K8s clusters and other various workloads, none of those workloads (k8s included) access the ceph cluster directly, they all pass by the openstack layer one way or another. Le lun. 23 juin 2025 à 02:30, Reid Guyett <reid.guyett@gmail.com> a écrit :
I have a similar problem with some RBDs being deleted but still appearing in ls. To remove the stuck rbd I do the following: 1. rbd -p <pool> <rbd_name> --debug_ms 1/1 2>&1 | grep header | tail 2. I can see that an OSD is listed in the output on the last line. 3. I restart that OSD 4. Delete the rbd
I'm not sure if it is the same thing but it doesn't hurt to try.
On Fri, Jun 13, 2025 at 4:34 AM Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi Enrico, Thanks a lot for your answer, however I've already tried that and it can't work as no data still exist of this image on the cluster and the image info end up with the same error than previous command.
I think my only remaining solution would be to eliminate this image metadata from the metadata pool and index itself, but I'm not having a clear procedure to do that yet as I can't just rm the image, I would need to identify it within the metadata pool of this EC data pool.
If have already tried that, I'm all ears out ;-)
Le mer. 11 juin 2025 à 09:15, Enrico Bocchi <enrico.bocchi@cern.ch> a écrit :
Hi Gael,
You may want to try this clean-up procedure (it involves some lower-level manual manipulation of rados objects):
1. List // Info the image rbd -p pool ls rbd -p pool info 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rbd image '00587c5a-8d54-40d1-b7fc-6aeb77d48a8a': size 50 TiB in 13107200 objects order 22 (4 MiB objects) snapshot_count: 0 id: 9907dc2254ff2e block_name_prefix: rbd_data.9907dc2254ff2e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Apr 6 08:24:25 2023 access_timestamp: Thu Apr 6 08:24:25 2023 modify_timestamp: Thu Apr 6 08:24:25 2023
2. Remove the header with rados commands rados -p pool rm rbd_id.00587c5a-8d54-40d1-b7fc-6aeb77d48a8a rados -p pool rm rm rbd_header.9907dc2254ff2e
3. Remove all the RBD data rados -p pool ls | grep '^rbd_data.9907dc2254ff2e.' | xargs rados -p pool rm
4. Remove from RBD list rbd -p pool rm 00587c5a-8d54-40d1-b7fc-6aeb77d48a8a
Cheers, Enrico
Hi there! Nope it is not in the trash pool.
Le jeu. 5 juin 2025 à 12:37, Eugen Block<eblock@nde.ag> a écrit :
Is that image in the trash?
`rbd -p pool trash ls`
Zitat von Gaël THEROND<gael.therond@bitswalk.com>:
Hi folks,
I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`.
This pool is a EC based pool and neither the metadata nor the data
On 6/10/25 08:47, Gaël THEROND wrote: pool
have any data of that image remaining.
So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can.
Thanks! _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io