Hi Thomas, A few Ceph versions ago we hit a bug (a "corner case", as they called it) and dropped a 1 PB cluster. After deleting the whole cluster, we found that the disks still held 200 TB of orphaned object data. I asked several times on IRC and got very little information. The only thing that became clear was that, at the time, this was related to RocksDB compaction and similar. We never found out the root cause 100%. It is quite hard to manage. In our case, the orphan search went into an infinite loop and could literally run for weeks producing listings, and then deleting the orphans object by object was not easy at all. On top of that, I believe there was a bug in the garbage collector where it reported the object as deleted but actually left it there. Definitely not a pleasant experience, and I hope none of you go through it. There is nothing more frustrating than these situations in distributed storage systems. I know of several hyperscalers that have run into this too, and their way of "fixing" it at the time was keeping 4 replicas of the data instead of 3. Of course, with infinitely deep pockets, which we didn't have. Good luck, and I hope your search is productive. Regards, MANUEL RIOS FERNANDEZ -----Mensaje original----- De: Thomas Erbesdobler <ceph-users@ceph.io> Enviado el: Monday, October 19, 2026 7:27 PM Para: ceph-users@ceph.io Asunto: [ceph-users] Creating unfound objects on purpose Hi all, do you know about a reliable way to create unfound objects? I recently had a couple unfound objects on a production cluster after a group of osds crashed, and I would like to reproduce this state and experiment with it in my development environment. But therefore I first need to get unfound objects ;-). In all cases I've used ceph 19.2.3 with bluestore OSDs. What I tried so far: Attempt 1: * create a replicated pg with 3 shards, min_size=1 (same as in the production system at the time of the crash) * stop all but one osd, set norecover and nobackfill * write to a part of an object * start a second osd, wait until it boots and peers, stop the first osd * the object is indeed unfound, but I cannot run `ceph pg ... mark_unfound_lost ...` on it, since the osds who might have the object have not been queried * but when I destroy and recreate the first osd, the object becomes available again after recovery and reads the new content ~> recovery must have happened in some way Attempt 2: * same pg as in attempt 1 * stop all but one osd * write a lot to the pg (or rather its pool, that is, to create the data I've actually used the pool for rbd) * start one more osd of the pg and wait until recovery starts, then stop the first osd * the pg goes down, but no unfound objects have been created Basically I tried to follow the sequence sketched in https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#unfo... . So, I guess it is more complicated to create unfound objects, maybe one requires to cause the "correct" osd crashes such that the pg's log somehow gets out of sync with the actual data stored on the osd? Any ideas would be welcome. Regards, Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io