From my initial testing it looks like 14.2.4 fully supports the deduplication mentioned here: https://docs.ceph.com/docs/master/dev/deduplication/ However, I'm not sure where the struct object_manifest script goes in relation to foo and foo-chunk, and I'm not aware of what the offsets/caspool should be. If this still isn't fully implemented how does the dedup tool work? If I remove a file but it exists elsewhere on the volume, will it be purged or would the tool need to run again to clear the data?
So since nobody who's actually working on it has chimed in: While there is some deduplication functionality built into the system, AFAIK it's not something considered for users at this point. It's under ongoing development, doesn't have performance data, and isn't plumbed through into a lot of the system. Last I heard about it was a discussion about how to handle the reference counting used to delete data when it's no longer needed. (See the thread "refcounting chunks vs snapshots" and https://github.com/ceph/ceph/pull/29283) -Greg On Wed, Oct 2, 2019 at 4:48 PM The Zombie Hunter <thezombiehunter@gmail.com> wrote:
From my initial testing it looks like 14.2.4 fully supports the deduplication mentioned here:
https://docs.ceph.com/docs/master/dev/deduplication/
However, I'm not sure where the struct object_manifest script goes in relation to foo and foo-chunk, and I'm not aware of what the offsets/caspool should be.
If this still isn't fully implemented how does the dedup tool work? If I remove a file but it exists elsewhere on the volume, will it be purged or would the tool need to run again to clear the data? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Oct 9, 2019 at 12:56 PM Gregory Farnum <gfarnum@redhat.com> wrote:
So since nobody who's actually working on it has chimed in: While there is some deduplication functionality built into the system, AFAIK it's not something considered for users at this point. It's under ongoing development, doesn't have performance data, and isn't plumbed through into a lot of the system. Last I heard about it was a discussion about how to handle the reference counting used to delete data when it's no longer needed. (See the thread "refcounting chunks vs snapshots" and https://github.com/ceph/ceph/pull/29283) -Greg
I was holding off, as this is really not a part of Ceph, but we have done some preliminary testing of VDO as a client on top of RBD, and it looked fine. We also run borg on top of RBD, which is also stable and able to handle large volumes of data. Hope this helps. -- Alex Gorbachev Intelligent Systems Services Inc.
On Wed, Oct 2, 2019 at 4:48 PM The Zombie Hunter <thezombiehunter@gmail.com> wrote:
From my initial testing it looks like 14.2.4 fully supports the
deduplication mentioned here:
https://docs.ceph.com/docs/master/dev/deduplication/
However, I'm not sure where the struct object_manifest script goes in
relation to foo and foo-chunk, and I'm not aware of what the offsets/caspool should be.
If this still isn't fully implemented how does the dedup tool work? If I
remove a file but it exists elsewhere on the volume, will it be purged or would the tool need to run again to clear the data?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Alex Gorbachev
-
Gregory Farnum
-
The Zombie Hunter