Hi, I would like to get a better understanding at what happens under the hood when an object is deleted in a Ceph cluster. Is there a (detailed) write up of the deletion process? I would like to be able to answer questions like: - How can we be sure that after the successful completion of for example a "rbd rm $image" command it is impossible for the data to be restored. - How do we know we don't get deleted data returned that was marked as deleted (but never overwritten with new data) Can we be sure? Or are there ways to perform a recovery procedure (as a storage admin with access to all components of the cluster), i.e. gather information about higher level Ceph abstractions (rbd, file in CephFS, RGW bucket) that have existed in the past, gather information about what objects where in use, do CRUSH calculation (and consult upmaps) to obtain OSDs involved, Query RocksDB / Bluestore to obtain information about LBA on disk, read the data and in this way reconstruct the object(s). Pathological example: rbd rm $image (successful deletion) ceph pause immediately after that Do the recovery procedure noted above How likely is it that we would be able to recovery the data? I understand it's hard to put a number on it, so pointers to where (parts) of this process is described will help me getting a better understanding. I'm also aware that it highly depends on how busy a cluster would be, what type of media involved, if the discard option of Ceph and disks is in use, and many other factors. Thanks, Stefan