Hi,
On Sunday, February 21st, 2021 at 12:39, Loïc Dachary <loic(a)dachary.org> wrote:
For the record, here is a summary of the key takeaways
from this conversation (so far):
- Ambry[0] is a perfect match and I'll keep exploring it[1].
- To keep billions of small objects manageable, they must be packed together.
- Immutable & never deleted objects can be grouped together for the purpose of
packing them without a central database. For this to work the id of the group to which an
object belongs is included in the object ID (e.g. SHA256 + UUID of the group). That's
what Ambry does.
What about using OMAP for the purpose of "grouping together" small objects? You
would store tiny objects as OMAP key/value pairs on a Ceph object. Internally, they're
stored in RocksDB so the min allocation size isn't an issue, and retrieving individual
objects should be fairly quick.
Has anyone tried this? How does RocksDB cope with terabytes of data and hundreds of
millions of key-value pairs? Is recovery faster for OMAP compared to the equivalent number
of RADOS objects?
Cheers,
--
Ben