If it were me, I would do something along the lines of: - Bundle larger blocks of code into pixz <https://github.com/vasi/pixz> (essentially indexed tar files, allowing random access) and store them in RadosGW. - Build a small frontend that fetches (with caching) them and provides the file contents via whatever your UI is. On Wed, Feb 3, 2021 at 12:55 AM Burkhard Linke < Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
Hi,
Just my 2 cents:
You could use the first byte of the SHA sum to identify the image, e.g. using a fixed number of 256 images. Or some flexible approach similar to
On 2/3/21 9:41 AM, Loïc Dachary wrote: the way filestore used to store rados objects.
A friend suggested the same to save space. Good idea.
If you want to further reduce the index size, you can just store the offset, and the first 4? 8? bytes at that offset define the size of the following artifacts. That's similar to the way Pascal used to store strings in the good ol' times. You might also want to think about using a complete header which also includes the artifact's name etc. This will allow you to rebuild the index if it becomes corrupted. The storage overhead should be insignificant
Your index will become a simple mapping of SHA sum -> offset, and you might also be able to use optimized implementations.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- This e-mail and all information in, attached to, or linked via this e-mail (together the ‘e-mail’) is confidential and may be legally privileged. It is intended solely for the intended addressee(s). Access to, or any onward transmission, of this e-mail by any other person is not authorised. If you are not the intended recipient, you are requested to immediately alert the sender of this e-mail and to immediately delete this e-mail. Any disclosure in any form of all or part of this e-mail, or of any the parties to it, including any copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. This e-mail is not, and is not intended to be, and should not be construed as being, (a) any offer, solicitation, or promotion of any kind; (b) the basis of any investment or other decision(s); (c) any recommendation to buy, sell or transact in any manner any good(s), product(s) or service(s), nor engage in any investment(s) or other transaction(s) or activities; or (d) the provision of, or related to, any advisory service(s) or activities, including regarding any investment, tax, legal, financial, accounting, consulting or any other related service(s).