How does Ceph delete objects / data on disk?
Hi, I would like to get a better understanding at what happens under the hood when an object is deleted in a Ceph cluster. Is there a (detailed) write up of the deletion process? I would like to be able to answer questions like: - How can we be sure that after the successful completion of for example a "rbd rm $image" command it is impossible for the data to be restored. - How do we know we don't get deleted data returned that was marked as deleted (but never overwritten with new data) Can we be sure? Or are there ways to perform a recovery procedure (as a storage admin with access to all components of the cluster), i.e. gather information about higher level Ceph abstractions (rbd, file in CephFS, RGW bucket) that have existed in the past, gather information about what objects where in use, do CRUSH calculation (and consult upmaps) to obtain OSDs involved, Query RocksDB / Bluestore to obtain information about LBA on disk, read the data and in this way reconstruct the object(s). Pathological example: rbd rm $image (successful deletion) ceph pause immediately after that Do the recovery procedure noted above How likely is it that we would be able to recovery the data? I understand it's hard to put a number on it, so pointers to where (parts) of this process is described will help me getting a better understanding. I'm also aware that it highly depends on how busy a cluster would be, what type of media involved, if the discard option of Ceph and disks is in use, and many other factors. Thanks, Stefan
I would like to get a better understanding at what happens under the hood when an object is deleted in a Ceph cluster.
Is there a (detailed) write up of the deletion process?
I suspect that such would quickly become obsolete once written but would be good to know.
I would like to be able to answer questions like:
- How can we be sure that after the successful completion of for example a "rbd rm $image" command it is impossible for the data to be restored.
RBD images themselves are thin-provisioned usually, which takes care of that angle.
- How do we know we don't get deleted data returned that was marked as deleted (but never overwritten with new data)
Can we be sure? Or are there ways to perform a recovery procedure (as a storage admin with access to all components of the cluster), i.e. gather information about higher level Ceph abstractions (rbd, file in CephFS, RGW bucket) that have existed in the past, gather information about what objects where in use, do CRUSH calculation (and consult upmaps) to obtain OSDs involved, Query RocksDB / Bluestore to obtain information about LBA on disk, read the data and in this way reconstruct the object(s).
Or at a smaller granularity, dd the OSD devices. I once had to effectively do that in college to get an art student’s paper back after a certain crappy piece of software MINCEd her floppy.
Pathological example:
rbd rm $image (successful deletion) ceph pause immediately after that Do the recovery procedure noted above
How likely is it that we would be able to recovery the data?
Like most filesystems, pretty likely at a certain granularity. In the above case, the RBD trash system may even be in play. I’m confident that users can’t get at data, but to protect against thieves, liquidators etc., dmcrypt is your friend - just never get rid of the mon SSDs, or at the very least issue crypto erase operations on them before letting go. Were Ceph to zero data on delete, that would present substantially more IOPs to the devices, and in the case of SSDs erode endurance. Certain industries might pay the price nonetheless. I must note as an aside that dding /dev/zero to an SSD most likely does not overwrite data.
I understand it's hard to put a number on it, so pointers to where (parts) of this process is described will help me getting a better understanding. I'm also aware that it highly depends on how busy a cluster would be, what type of media involved, if the discard option of Ceph and disks is in use, and many other factors.
Thanks,
Stefan
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Pathological example:
rbd rm $image (successful deletion) ceph pause immediately after that Do the recovery procedure noted above
How likely is it that we would be able to recovery the data?
Like most filesystems, pretty likely at a certain granularity. In the above case, the RBD trash system may even be in play. I’m confident that users can’t get at data, but to protect against thieves, liquidators etc., dmcrypt is your friend - just never get rid of the mon SSDs, or at the very least issue crypto erase operations on them before letting go.
Yes, I agree with the above statement. If you have a client using this RBD and needs to make sure evil ceph admins can't get to the data, they should definitely encrypt the partition where this RBD gets mounted. Then it is just a matter of forgetting the key to this and delete the RBD image as you please. If the secret you are hoping to protect is like a x509 cert key, it will be around 1-2k at most, so anyone being able to read the raw disks 5 seconds after you do "ceph pause" will then find this ------ BEGIN KEY -------- and be able to steal this super important key because it will be lying around somewhere on three or more OSDs. If you as a client on the other hand LUKS-encrypt the RBD mount, then wipe the key (or reset key to a random value you don't know afterwards) and then delete the RBD image, the encrypted data will possibly be able to be found by people with root access to OSD hosts - or the physical boxes, but they will not be able to read the secret data you wanted to protect. I'm somewhat of the idea that if someone asked me to successfully piece together a 40G RBD image that was deleted some time ago on a busy cluster, I would probably not be able to, not even to save my kids lives. But others theoretically might, and auditors seldom care for non-numeric values like "JJs kids lives" so if you have a real case, just go with local disk or partition encryption and be done with it. The perf hit is not so bad anymore and needing to trust someone you might not trust and convince an auditor you can trust them is just not worth the hassle. All major OSs have encrypted file system options so this part is solved already, and if the data is important enough for these kinds of questions you can take the burden of setting it up. -- May the most significant bit of your life be positive.
As others have suggested, it's not in any sense a "secure erase" where data is overwritten. At the RADOS level, when an OSD receives a delete operation, it twiddles the metadata so the object isn't there any more, and BlueStore marks the space as unallocated, etc. You could go in and get it with disk forensics; I don't think we have any meaningful way of finding deleted objects with ceph-objectstore-tool or such (though I could be wrong?) but I don't think those are meaningfully different from a security perspective. There is absolutely no way for a RADOS client to retrieve deleted data using RADOS ops once a delete is performed (barring snapshots etc being in play). RBD and CephFS have their own layers on top of this: rbd has a trash bin where objects sit for a while before being purged; files unlinked in CephFS get moved into a purge queue once there are no more active references to them, and that purge queue is processed by the mds to issue the actual object deletes. RGW has a similar purging concept but I don't know the details of it (it's quite sophisticated due to the semantics involved and the rules around S3 object versioning and how they're implemented.) As Janne suggested, use client-side encryption if you're worried about it. -Greg On Wed, May 28, 2025 at 6:43 AM Janne Johansson <icepic.dz@gmail.com> wrote:
Pathological example:
rbd rm $image (successful deletion) ceph pause immediately after that Do the recovery procedure noted above
How likely is it that we would be able to recovery the data?
Like most filesystems, pretty likely at a certain granularity. In the above case, the RBD trash system may even be in play. I’m confident that users can’t get at data, but to protect against thieves, liquidators etc., dmcrypt is your friend - just never get rid of the mon SSDs, or at the very least issue crypto erase operations on them before letting go.
Yes, I agree with the above statement. If you have a client using this RBD and needs to make sure evil ceph admins can't get to the data, they should definitely encrypt the partition where this RBD gets mounted. Then it is just a matter of forgetting the key to this and delete the RBD image as you please.
If the secret you are hoping to protect is like a x509 cert key, it will be around 1-2k at most, so anyone being able to read the raw disks 5 seconds after you do "ceph pause" will then find this ------ BEGIN KEY -------- and be able to steal this super important key because it will be lying around somewhere on three or more OSDs. If you as a client on the other hand LUKS-encrypt the RBD mount, then wipe the key (or reset key to a random value you don't know afterwards) and then delete the RBD image, the encrypted data will possibly be able to be found by people with root access to OSD hosts - or the physical boxes, but they will not be able to read the secret data you wanted to protect.
I'm somewhat of the idea that if someone asked me to successfully piece together a 40G RBD image that was deleted some time ago on a busy cluster, I would probably not be able to, not even to save my kids lives. But others theoretically might, and auditors seldom care for non-numeric values like "JJs kids lives" so if you have a real case, just go with local disk or partition encryption and be done with it. The perf hit is not so bad anymore and needing to trust someone you might not trust and convince an auditor you can trust them is just not worth the hassle. All major OSs have encrypted file system options so this part is solved already, and if the data is important enough for these kinds of questions you can take the burden of setting it up.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Gregory Farnum
-
Janne Johansson
-
Stefan Kooman