If it were me, I would do something along the lines of:
- Bundle larger blocks of code into pixz
<https://github.com/vasi/pixz> (essentially
indexed tar files, allowing random access) and store them in RadosGW.
- Build a small frontend that fetches (with caching) them and provides the
file contents via whatever your UI is.
On Wed, Feb 3, 2021 at 12:55 AM Burkhard Linke <
Burkhard.Linke(a)computational.bio.uni-giessen.de> wrote:
Hi,
On 2/3/21 9:41 AM, Loïc Dachary wrote:
> Just my 2 cents:
>
> You could use the first byte of the SHA sum to identify the image, e.g.
using
a fixed number of 256 images. Or some flexible approach similar to
the way filestore used to store rados objects.
A friend suggested the same to save space. Good
idea.
If you want to further reduce the index size, you can just store the
offset, and the first 4? 8? bytes at that offset define the size of the
following artifacts. That's similar to the way Pascal used to store
strings in the good ol' times. You might also want to think about using
a complete header which also includes the artifact's name etc. This will
allow you to rebuild the index if it becomes corrupted. The storage
overhead should be insignificant
Your index will become a simple mapping of SHA sum -> offset, and you
might also be able to use optimized implementations.
Regards,
Burkhard
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
This e-mail and all information in, attached to, or linked via this
e-mail (together the ‘e-mail’) is confidential and may be legally
privileged. It is intended solely for the intended addressee(s). Access to,
or any onward transmission, of this e-mail by any other person is not
authorised. If you are not the intended recipient, you are requested to
immediately alert the sender of this e-mail and to immediately delete this
e-mail. Any disclosure in any form of all or part of this e-mail, or of any
the parties to it, including any copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be
unlawful.
This e-mail is not, and is not intended to be, and should
not be construed as being, (a) any offer, solicitation, or promotion of any
kind; (b) the basis of any investment or other decision(s); (c) any
recommendation to buy, sell or transact in any manner any good(s),
product(s) or service(s), nor engage in any investment(s) or other
transaction(s) or activities; or (d) the provision of, or related to, any
advisory service(s) or activities, including regarding any investment, tax,
legal, financial, accounting, consulting or any other related service(s).