Bonjour, In the context Software Heritage (a noble mission to preserve all source code)[0], artifacts have an average size of ~3KB and there are billions of them. They never change and are never deleted. To save space it would make sense to write them, one after the other, in an every growing RBD volume (more than 100TB). An index, located somewhere else, would record the offset and size of the artifacts in the volume. I wonder if someone already implemented this idea with success? And if not... does anyone see a reason why it would be a bad idea? Cheers [0] https://docs.softwareheritage.org/ -- Loïc Dachary, Artisan Logiciel Libre _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Dear Loïc , I do not have direct experience with this many files, but it resonates for me with deduplication, such as borg (https://www.borgbackup.org/) or a similar implementation in the latest Proxmox Backup Server ( https://pbs.proxmox.com/wiki/index.php/Main_Page). I think you would need a filesystem for either, so not sure how well this would integrate directly with RBD, but maybe cephfs is an option? I typically run zfs on top of rbd, and use only zfs compression, and then put borg on top of zfs. There is overhead, but this is a very flexible setup, operationally. All the best in your endeavor! -- Alex Gorbachev ISS/Storcium On Sat, Jan 30, 2021 at 10:01 AM Loïc Dachary <loic(a)dachary.org> wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi Alex, Using borg would indeed make sense to copy the replicate the rbd content in case rbd-mirror is not an option, nice idea :-) Interestingly there is no need for a proper file system: the files are immutable and never deleted. They are indexed by the SHA256 of their content and a map where the key is the SHA256 and the value is the offset,size in the rbd image would be enough. Cheers On 01/02/2021 03:27, Alex Gorbachev wrote: for need directly There of make -- Loïc Dachary, Artisan Logiciel Libre

Bonjour, In the context Software Heritage (a noble mission to preserve all source code)[0], artifacts have an average size of ~3KB and there are billions of them. They never change and are never deleted. To save space it would make sense to write them, one after the other, in an every growing RBD volume (more than 100TB). An index, located somewhere else, would record the offset and size of the artifacts in the volume. I wonder if someone already implemented this idea with success? And if not... does anyone see a reason why it would be a bad idea? Cheers [0] https://docs.softwareheritage.org/ -- Loïc Dachary, Artisan Logiciel Libre _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi Dan, On 01/02/2021 21:13, Dan van der Ster wrote: Good idea! I'll look for a cluster with 100TB of free space and post my findings. Yes. It is inspired by https://static.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf which suggests an ad-hoc implementation to pack immutable objects together. But I think RBD already provides the underlying logic, even though it is not specialized for this use case. RGW also packs small objects together and would be a good candidate. But it provides more flexibility to modify/delete objects and I assume it will be slower to write N objects with RGW than to write them sequentially on an RBD image. But I did not try and maybe I should. To be continued. -- Loïc Dachary, Artisan Logiciel Libre _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Packing's obviously a good idea for storing these kinds of artifacts in Ceph, and hacking through the existing librbd might indeed be easier than building something up from raw RADOS, especially if you want to use stuff like rbd-mirror. My main concern would just be as Dan points out, that we don't test rbd with extremely large images and we know deleting that image will take a looooong time — I don't know of other issues off the top of my head, and in the worst case you could always fall back to manipulating it with raw librados if there is an issue.

But you might also check in on the status of Danny Al-Gaaf's rados email project. Email and these artifacts seemingly have a lot in common.

-Greg On Mon, Feb 1, 2021 at 12:52 PM Loïc Dachary <loic(a)dachary.org> wrote: > Hi Dan, > > On 01/02/2021 21:13, Dan van der Ster wrote: >> Hi Loïc, >> >> We've never managed 100TB+ in a single RBD volume. I can't think of >> anything, but perhaps there are some unknown limitations when they get so >> big. >> It should be easy enough to use rbd bench to create and fill a massive test >> image to validate everything works well at that size. > Good idea! I'll look for a cluster with 100TB of free space and post my findings. >> Also, I assume you'll be doing the IO from just one client? Multiple >> readers/writers to a single volume could get complicated. > Yes. >> Otherwise, yes RBD sounds very convenient for what you need. > It is inspired by https://static.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf which suggests an ad-hoc implementation to pack immutable objects together. But I think RBD already provides the underlying logic, even though it is not specialized for this use case. RGW also packs small objects together and would be a good candidate. But it provides more flexibility to modify/delete objects and I assume it will be slower to write N objects with RGW than to write them sequentially on an RBD image. But I did not try and maybe I should. > > To be continued. >> Cheers, Dan >> >> >> On Sat, Jan 30, 2021, 4:01 PM Loïc Dachary <loic(a)dachary.org> wrote: >> >>> Bonjour, >>> >>> In the context Software Heritage (a noble mission to preserve all source >>> code)[0], artifacts have an average size of ~3KB and there are billions of >>> them. They never change and are never deleted. To save space it would make >>> sense to write them, one after the other, in an every growing RBD volume >>> (more than 100TB). An index, located somewhere else, would record the >>> offset and size of the artifacts in the volume. >>> >>> I wonder if someone already implemented this idea with success? And if >>> not... does anyone see a reason why it would be a bad idea? >>> >>> Cheers >>> >>> [0] https://docs.softwareheritage.org/ >>> >>> -- >>> Loïc Dachary, Artisan Logiciel Libre >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- > Loïc Dachary, Artisan Logiciel Libre > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Just my 2 cents: You could use the first byte of the SHA sum to identify the image, e.g. using a fixed number of 256 images. Or some flexible approach similar to the way filestore used to store rados objects.

Hi, On 2/3/21 9:41 AM, Loïc Dachary wrote: using a fixed number of 256 images. Or some flexible approach similar to the way filestore used to store rados objects. If you want to further reduce the index size, you can just store the offset, and the first 4? 8? bytes at that offset define the size of the following artifacts. That's similar to the way Pascal used to store strings in the good ol' times. You might also want to think about using a complete header which also includes the artifact's name etc. This will allow you to rebuild the index if it becomes corrupted. The storage overhead should be insignificant Your index will become a simple mapping of SHA sum -> offset, and you might also be able to use optimized implementations. Regards, Burkhard _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io