Hi all, do you know about a reliable way to create unfound objects? I recently had a couple unfound objects on a production cluster after a group of osds crashed, and I would like to reproduce this state and experiment with it in my development environment. But therefore I first need to get unfound objects ;-). In all cases I've used ceph 19.2.3 with bluestore OSDs. What I tried so far: Attempt 1: * create a replicated pg with 3 shards, min_size=1 (same as in the production system at the time of the crash) * stop all but one osd, set norecover and nobackfill * write to a part of an object * start a second osd, wait until it boots and peers, stop the first osd * the object is indeed unfound, but I cannot run `ceph pg ... mark_unfound_lost ...` on it, since the osds who might have the object have not been queried * but when I destroy and recreate the first osd, the object becomes available again after recovery and reads the new content ~> recovery must have happened in some way Attempt 2: * same pg as in attempt 1 * stop all but one osd * write a lot to the pg (or rather its pool, that is, to create the data I've actually used the pool for rbd) * start one more osd of the pg and wait until recovery starts, then stop the first osd * the pg goes down, but no unfound objects have been created Basically I tried to follow the sequence sketched in https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#unfo... . So, I guess it is more complicated to create unfound objects, maybe one requires to cause the "correct" osd crashes such that the pg's log somehow gets out of sync with the actual data stored on the osd? Any ideas would be welcome. Regards, Thomas