Pathological example:
rbd rm $image (successful deletion) ceph pause immediately after that Do the recovery procedure noted above
How likely is it that we would be able to recovery the data?
Like most filesystems, pretty likely at a certain granularity. In the above case, the RBD trash system may even be in play. I’m confident that users can’t get at data, but to protect against thieves, liquidators etc., dmcrypt is your friend - just never get rid of the mon SSDs, or at the very least issue crypto erase operations on them before letting go.
Yes, I agree with the above statement. If you have a client using this RBD and needs to make sure evil ceph admins can't get to the data, they should definitely encrypt the partition where this RBD gets mounted. Then it is just a matter of forgetting the key to this and delete the RBD image as you please. If the secret you are hoping to protect is like a x509 cert key, it will be around 1-2k at most, so anyone being able to read the raw disks 5 seconds after you do "ceph pause" will then find this ------ BEGIN KEY -------- and be able to steal this super important key because it will be lying around somewhere on three or more OSDs. If you as a client on the other hand LUKS-encrypt the RBD mount, then wipe the key (or reset key to a random value you don't know afterwards) and then delete the RBD image, the encrypted data will possibly be able to be found by people with root access to OSD hosts - or the physical boxes, but they will not be able to read the secret data you wanted to protect. I'm somewhat of the idea that if someone asked me to successfully piece together a 40G RBD image that was deleted some time ago on a busy cluster, I would probably not be able to, not even to save my kids lives. But others theoretically might, and auditors seldom care for non-numeric values like "JJs kids lives" so if you have a real case, just go with local disk or partition encryption and be done with it. The perf hit is not so bad anymore and needing to trust someone you might not trust and convince an auditor you can trust them is just not worth the hassle. All major OSs have encrypted file system options so this part is solved already, and if the data is important enough for these kinds of questions you can take the burden of setting it up. -- May the most significant bit of your life be positive.