17 Feb
2021
17 Feb
'21
10:06 p.m.
Bonjour,
TL;DR: Is it more advisable to work on Ceph internals to make it friendly to this
particular workload or write something similar to EOS[0] (i.e Rocksdb + Xrootd + RBD)?
This is a followup of two previous mails[1] sent while researching this topic. In a
nutshell, the Software Heritage project[1] currently has ~750TB and 10 billions objects,
75% of which have a size smaller than 16KB and 50% have a size smaller than 4KB. But they
only account for ~5% of the 750TB: 25% of the objects have a size > 16KB and total
~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of actual
storage. (if you're interested in the details see [2]).
Let say those 10 billions objects are stored in a single 4+2 erasure coded pool with
bluestore compression set for objects that have a size > 32KB and the smallest
allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected 350TB
but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is
because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to
repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use
no less than 4KB because that's the smallest allocation size for bluestore. That's
4 * 4KB = 16KB even when all that is needed is 1 byte.
It was suggested[6] to have two different pools: one with a 4+2 erasure pool and
compression for all objects with a size > 32KB that are expected to compress to 16KB.
And another with 3 replicas for the smaller objects to reduce space amplification to a
minimum without compromising on durability. A client looking for the object could make two
simultaneous requests to the two pools. They would get 404 from one of them and the object
from the other.
Another workaround, is best described in the "Finding a needle in Haystack:
Facebook’s photo storage"[9] paper and essentially boils down to using a database to
store a map between the object name and its location. That does not scale out (writing the
database index is the bottleneck) but it's simple enough and is successfully
implemented in EOS[0] with >200PB worth of data and in seaweedfs[10], another promising
object store software based on the same idea.
Instead of working around the problem, maybe Ceph could be modified to make better use of
the immutability of these objects[7], a hint that is apparently only used to figure out
how to best compress it and for checksum calculation[8]. I honestly have not clue how
difficult it would be. All I know is that it's not easy otherwise it would have been
done already: there seem to be a general need for efficiently (space wise and performance
wise) storing large quantities of objects smaller than 4KB.
Is it more advisable to:
* work on Ceph internals to make it friendly to this particular workload or,
* write another implementation of "Finding a needle in Haystack: Facebook’s photo
storage"[9] based on RBD[11]?
I'm currently leaning toward working on Ceph internals but there are pros and cons to
both approaches[12]. And since all this is still very new to me, there also is the
possibility that I'm missing something. Maybe it's *super* difficult to improve
Ceph in this way. I should try to figure that out sooner rather than later.
I realize it's a lot to take in and unless you're facing the exact same problem
there is very little chance you read that far :-) But if you did... I'm *really*
interested to hear what yout think. In any case I'll report back to this thread once a
decision has been made.
Cheers
[0] https://eos-web.web.cern.ch/eos-web/
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
[2] https://forge.softwareheritage.org/T3054
[3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
[4] https://forge.softwareheritage.org/T3052#58864
[5] https://forge.softwareheritage.org/T3052#58917
[6] https://forge.softwareheritage.org/T3052#58876
[7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
[8] https://forge.softwareheritage.org/T3055
[9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
[10] https://github.com/chrislusf/seaweedfs/wiki/Components
[11] https://forge.softwareheritage.org/T3049
[12] https://forge.softwareheritage.org/T3054#58977
--
Loïc Dachary, Artisan Logiciel Libre
17 Feb
17 Feb
10:34 p.m.
I'm not much of a programmer, but as soon as I hear "immutable
objects" I think "content-addressed". I don't know if you have many
duplicate objects in this set, but content-addressing gives you
object-level dedup for free. Do you have to preserve some meaningful
object names from the original dataset, or just do you just need some
kind of ID?
On Wed, Feb 17, 2021 at 11:37 AM Loïc Dachary <loic(a)dachary.org> wrote:
>
> Bonjour,
>
> TL;DR: Is it more advisable to work on Ceph internals to make it friendly to this
particular workload or write something similar to EOS[0] (i.e Rocksdb + Xrootd + RBD)?
>
> This is a followup of two previous mails[1] sent while researching this topic. In a
nutshell, the Software Heritage project[1] currently has ~750TB and 10 billions objects,
75% of which have a size smaller than 16KB and 50% have a size smaller than 4KB. But they
only account for ~5% of the 750TB: 25% of the objects have a size > 16KB and total
~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of actual
storage. (if you're interested in the details see [2]).
>
> Let say those 10 billions objects are stored in a single 4+2 erasure coded pool with
bluestore compression set for objects that have a size > 32KB and the smallest
allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected 350TB
but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is
because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to
repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use
no less than 4KB because that's the smallest allocation size for bluestore. That's
4 * 4KB = 16KB even when all that is needed is 1 byte.
>
> It was suggested[6] to have two different pools: one with a 4+2 erasure pool and
compression for all objects with a size > 32KB that are expected to compress to 16KB.
And another with 3 replicas for the smaller objects to reduce space amplification to a
minimum without compromising on durability. A client looking for the object could make two
simultaneous requests to the two pools. They would get 404 from one of them and the object
from the other.
>
> Another workaround, is best described in the "Finding a needle in Haystack:
Facebook’s photo storage"[9] paper and essentially boils down to using a database to
store a map between the object name and its location. That does not scale out (writing the
database index is the bottleneck) but it's simple enough and is successfully
implemented in EOS[0] with >200PB worth of data and in seaweedfs[10], another promising
object store software based on the same idea.
>
> Instead of working around the problem, maybe Ceph could be modified to make better
use of the immutability of these objects[7], a hint that is apparently only used to figure
out how to best compress it and for checksum calculation[8]. I honestly have not clue how
difficult it would be. All I know is that it's not easy otherwise it would have been
done already: there seem to be a general need for efficiently (space wise and performance
wise) storing large quantities of objects smaller than 4KB.
>
> Is it more advisable to:
>
> * work on Ceph internals to make it friendly to this particular workload or,
> * write another implementation of "Finding a needle in Haystack: Facebook’s
photo storage"[9] based on RBD[11]?
>
> I'm currently leaning toward working on Ceph internals but there are pros and
cons to both approaches[12]. And since all this is still very new to me, there also is the
possibility that I'm missing something. Maybe it's *super* difficult to improve
Ceph in this way. I should try to figure that out sooner rather than later.
>
> I realize it's a lot to take in and unless you're facing the exact same
problem there is very little chance you read that far :-) But if you did... I'm
*really* interested to hear what yout think. In any case I'll report back to this
thread once a decision has been made.
>
> Cheers
>
> [0] https://eos-web.web.cern.ch/eos-web/
> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
> [2] https://forge.softwareheritage.org/T3054
> [3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
> [4] https://forge.softwareheritage.org/T3052#58864
> [5] https://forge.softwareheritage.org/T3052#58917
> [6] https://forge.softwareheritage.org/T3052#58876
> [7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
> [8] https://forge.softwareheritage.org/T3055
> [9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
> [10] https://github.com/chrislusf/seaweedfs/wiki/Components
> [11] https://forge.softwareheritage.org/T3049
> [12] https://forge.softwareheritage.org/T3054#58977
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:43 p.m.
Hi Nathan,
Good thinking :-) The names of the objects are indeed the SHA256 of their content, which
provides deduplication.
Cheers
On 17/02/2021 18:04, Nathan Fish wrote:
I'm not much of a programmer, but as soon as I
hear "immutable
objects" I think "content-addressed". I don't know if you have many
duplicate objects in this set, but content-addressing gives you
object-level dedup for free. Do you have to preserve some meaningful
object names from the original dataset, or just do you just need some
kind of ID?
On Wed, Feb 17, 2021 at 11:37 AM Loïc Dachary <loic(a)dachary.org> wrote:
> Bonjour,
>
> TL;DR: Is it more advisable to work on Ceph internals to make it friendly to this
particular workload or write something similar to EOS[0] (i.e Rocksdb + Xrootd + RBD)?
>
> This is a followup of two previous mails[1] sent while researching this topic. In a
nutshell, the Software Heritage project[1] currently has ~750TB and 10 billions objects,
75% of which have a size smaller than 16KB and 50% have a size smaller than 4KB. But they
only account for ~5% of the 750TB: 25% of the objects have a size > 16KB and total
~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of actual
storage. (if you're interested in the details see [2]).
>
> Let say those 10 billions objects are stored in a single 4+2 erasure coded pool with
bluestore compression set for objects that have a size > 32KB and the smallest
allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected 350TB
but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is
because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to
repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use
no less than 4KB because that's the smallest allocation size for bluestore. That's
4 * 4KB = 16KB even when all that is needed is 1 byte.
>
> It was suggested[6] to have two different pools: one with a 4+2 erasure pool and
compression for all objects with a size > 32KB that are expected to compress to 16KB.
And another with 3 replicas for the smaller objects to reduce space amplification to a
minimum without compromising on durability. A client looking for the object could make two
simultaneous requests to the two pools. They would get 404 from one of them and the object
from the other.
>
> Another workaround, is best described in the "Finding a needle in Haystack:
Facebook’s photo storage"[9] paper and essentially boils down to using a database to
store a map between the object name and its location. That does not scale out (writing the
database index is the bottleneck) but it's simple enough and is successfully
implemented in EOS[0] with >200PB worth of data and in seaweedfs[10], another promising
object store software based on the same idea.
>
> Instead of working around the problem, maybe Ceph could be modified to make better
use of the immutability of these objects[7], a hint that is apparently only used to figure
out how to best compress it and for checksum calculation[8]. I honestly have not clue how
difficult it would be. All I know is that it's not easy otherwise it would have been
done already: there seem to be a general need for efficiently (space wise and performance
wise) storing large quantities of objects smaller than 4KB.
>
> Is it more advisable to:
>
> * work on Ceph internals to make it friendly to this particular workload or,
> * write another implementation of "Finding a needle in Haystack: Facebook’s
photo storage"[9] based on RBD[11]?
>
> I'm currently leaning toward working on Ceph internals but there are pros and
cons to both approaches[12]. And since all this is still very new to me, there also is the
possibility that I'm missing something. Maybe it's *super* difficult to improve
Ceph in this way. I should try to figure that out sooner rather than later.
>
> I realize it's a lot to take in and unless you're facing the exact same
problem there is very little chance you read that far :-) But if you did... I'm
*really* interested to hear what yout think. In any case I'll report back to this
thread once a decision has been made.
>
> Cheers
>
> [0] https://eos-web.web.cern.ch/eos-web/
> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
> [2] https://forge.softwareheritage.org/T3054
> [3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
> [4] https://forge.softwareheritage.org/T3052#58864
> [5] https://forge.softwareheritage.org/T3052#58917
> [6] https://forge.softwareheritage.org/T3052#58876
> [7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
> [8] https://forge.softwareheritage.org/T3055
> [9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
> [10] https://github.com/chrislusf/seaweedfs/wiki/Components
> [11] https://forge.softwareheritage.org/T3049
> [12] https://forge.softwareheritage.org/T3054#58977
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Loïc Dachary, Artisan Logiciel Libre
10:57 p.m.
Why not put all the data to a zfs pool with 3-4 levels deep directory
structure each directory named with 2 byte in range 00-FF?
Four levels deep, you get 255^4=4B folders with 3-4 objects per folder
or three levels deep you get 255^3=16M folders with ~1000 objects
each.
On Wed, Feb 17, 2021 at 8:14 PM Loïc Dachary <loic(a)dachary.org> wrote:
>
> Hi Nathan,
>
> Good thinking :-) The names of the objects are indeed the SHA256 of their content,
which provides deduplication.
>
> Cheers
>
> On 17/02/2021 18:04, Nathan Fish wrote:
> > I'm not much of a programmer, but as soon as I hear "immutable
> > objects" I think "content-addressed". I don't know if you
have many
> > duplicate objects in this set, but content-addressing gives you
> > object-level dedup for free. Do you have to preserve some meaningful
> > object names from the original dataset, or just do you just need some
> > kind of ID?
> >
> > On Wed, Feb 17, 2021 at 11:37 AM Loïc Dachary <loic(a)dachary.org> wrote:
> >> Bonjour,
> >>
> >> TL;DR: Is it more advisable to work on Ceph internals to make it friendly to
this particular workload or write something similar to EOS[0] (i.e Rocksdb + Xrootd +
RBD)?
> >>
> >> This is a followup of two previous mails[1] sent while researching this
topic. In a nutshell, the Software Heritage project[1] currently has ~750TB and 10
billions objects, 75% of which have a size smaller than 16KB and 50% have a size smaller
than 4KB. But they only account for ~5% of the 750TB: 25% of the objects have a size >
16KB and total ~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of
actual storage. (if you're interested in the details see [2]).
> >>
> >> Let say those 10 billions objects are stored in a single 4+2 erasure coded
pool with bluestore compression set for objects that have a size > 32KB and the
smallest allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected
350TB but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is
because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to
repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use
no less than 4KB because that's the smallest allocation size for bluestore. That's
4 * 4KB = 16KB even when all that is needed is 1 byte.
> >>
> >> It was suggested[6] to have two different pools: one with a 4+2 erasure pool
and compression for all objects with a size > 32KB that are expected to compress to
16KB. And another with 3 replicas for the smaller objects to reduce space amplification to
a minimum without compromising on durability. A client looking for the object could make
two simultaneous requests to the two pools. They would get 404 from one of them and the
object from the other.
> >>
> >> Another workaround, is best described in the "Finding a needle in
Haystack: Facebook’s photo storage"[9] paper and essentially boils down to using a
database to store a map between the object name and its location. That does not scale out
(writing the database index is the bottleneck) but it's simple enough and is
successfully implemented in EOS[0] with >200PB worth of data and in seaweedfs[10],
another promising object store software based on the same idea.
> >>
> >> Instead of working around the problem, maybe Ceph could be modified to make
better use of the immutability of these objects[7], a hint that is apparently only used to
figure out how to best compress it and for checksum calculation[8]. I honestly have not
clue how difficult it would be. All I know is that it's not easy otherwise it would
have been done already: there seem to be a general need for efficiently (space wise and
performance wise) storing large quantities of objects smaller than 4KB.
> >>
> >> Is it more advisable to:
> >>
> >> * work on Ceph internals to make it friendly to this particular workload
or,
> >> * write another implementation of "Finding a needle in Haystack:
Facebook’s photo storage"[9] based on RBD[11]?
> >>
> >> I'm currently leaning toward working on Ceph internals but there are
pros and cons to both approaches[12]. And since all this is still very new to me, there
also is the possibility that I'm missing something. Maybe it's *super* difficult
to improve Ceph in this way. I should try to figure that out sooner rather than later.
> >>
> >> I realize it's a lot to take in and unless you're facing the exact
same problem there is very little chance you read that far :-) But if you did... I'm
*really* interested to hear what yout think. In any case I'll report back to this
thread once a decision has been made.
> >>
> >> Cheers
> >>
> >> [0] https://eos-web.web.cern.ch/eos-web/
> >> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
> >> [2] https://forge.softwareheritage.org/T3054
> >> [3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
> >> [4] https://forge.softwareheritage.org/T3052#58864
> >> [5] https://forge.softwareheritage.org/T3052#58917
> >> [6] https://forge.softwareheritage.org/T3052#58876
> >> [7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
> >> [8] https://forge.softwareheritage.org/T3055
> >> [9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
> >> [10] https://github.com/chrislusf/seaweedfs/wiki/Components
> >> [11] https://forge.softwareheritage.org/T3049
> >> [12] https://forge.softwareheritage.org/T3054#58977
> >>
> >> --
> >> Loïc Dachary, Artisan Logiciel Libre
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
18 Feb
18 Feb
1:45 a.m.
On 17/02/2021 18:27, Serkan Çoban wrote:
Why not put all the data to a zfs pool with 3-4 levels
deep directory
structure each directory named with 2 byte in range 00-FF?
Four levels deep, you get 255^4=4B folders with 3-4 objects per folder
or three levels deep you get 255^3=16M folders with ~1000 objects
each.
It is more or less the current setup :-) I should have mentioned that there
currently are ~750TB and 10 billions objects. But it's growing by 50TB every month and
it will keep growing indefinitely. Reason why a solution that scales out is desirable.
On Wed, Feb 17, 2021 at 8:14 PM Loïc Dachary <loic(a)dachary.org> wrote:
> Hi Nathan,
>
> Good thinking :-) The names of the objects are indeed the SHA256 of their content,
which provides deduplication.
>
> Cheers
>
> On 17/02/2021 18:04, Nathan Fish wrote:
>> I'm not much of a programmer, but as soon as I hear "immutable
>> objects" I think "content-addressed". I don't know if you have
many
>> duplicate objects in this set, but content-addressing gives you
>> object-level dedup for free. Do you have to preserve some meaningful
>> object names from the original dataset, or just do you just need some
>> kind of ID?
>>
>> On Wed, Feb 17, 2021 at 11:37 AM Loïc Dachary <loic(a)dachary.org> wrote:
>>> Bonjour,
>>>
>>> TL;DR: Is it more advisable to work on Ceph internals to make it friendly to
this particular workload or write something similar to EOS[0] (i.e Rocksdb + Xrootd +
RBD)?
>>>
>>> This is a followup of two previous mails[1] sent while researching this
topic. In a nutshell, the Software Heritage project[1] currently has ~750TB and 10
billions objects, 75% of which have a size smaller than 16KB and 50% have a size smaller
than 4KB. But they only account for ~5% of the 750TB: 25% of the objects have a size >
16KB and total ~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of
actual storage. (if you're interested in the details see [2]).
>>>
>>> Let say those 10 billions objects are stored in a single 4+2 erasure coded
pool with bluestore compression set for objects that have a size > 32KB and the
smallest allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected
350TB but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is
because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to
repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use
no less than 4KB because that's the smallest allocation size for bluestore. That's
4 * 4KB = 16KB even when all that is needed is 1 byte.
>>>
>>> It was suggested[6] to have two different pools: one with a 4+2 erasure pool
and compression for all objects with a size > 32KB that are expected to compress to
16KB. And another with 3 replicas for the smaller objects to reduce space amplification to
a minimum without compromising on durability. A client looking for the object could make
two simultaneous requests to the two pools. They would get 404 from one of them and the
object from the other.
>>>
>>> Another workaround, is best described in the "Finding a needle in
Haystack: Facebook’s photo storage"[9] paper and essentially boils down to using a
database to store a map between the object name and its location. That does not scale out
(writing the database index is the bottleneck) but it's simple enough and is
successfully implemented in EOS[0] with >200PB worth of data and in seaweedfs[10],
another promising object store software based on the same idea.
>>>
>>> Instead of working around the problem, maybe Ceph could be modified to make
better use of the immutability of these objects[7], a hint that is apparently only used to
figure out how to best compress it and for checksum calculation[8]. I honestly have not
clue how difficult it would be. All I know is that it's not easy otherwise it would
have been done already: there seem to be a general need for efficiently (space wise and
performance wise) storing large quantities of objects smaller than 4KB.
>>>
>>> Is it more advisable to:
>>>
>>> * work on Ceph internals to make it friendly to this particular workload
or,
>>> * write another implementation of "Finding a needle in Haystack:
Facebook’s photo storage"[9] based on RBD[11]?
>>>
>>> I'm currently leaning toward working on Ceph internals but there are pros
and cons to both approaches[12]. And since all this is still very new to me, there also is
the possibility that I'm missing something. Maybe it's *super* difficult to
improve Ceph in this way. I should try to figure that out sooner rather than later.
>>>
>>> I realize it's a lot to take in and unless you're facing the exact
same problem there is very little chance you read that far :-) But if you did... I'm
*really* interested to hear what yout think. In any case I'll report back to this
thread once a decision has been made.
>>>
>>> Cheers
>>>
>>> [0] https://eos-web.web.cern.ch/eos-web/
>>> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
>>> [2] https://forge.softwareheritage.org/T3054
>>> [3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
>>> [4] https://forge.softwareheritage.org/T3052#58864
>>> [5] https://forge.softwareheritage.org/T3052#58917
>>> [6] https://forge.softwareheritage.org/T3052#58876
>>> [7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
>>> [8] https://forge.softwareheritage.org/T3055
>>> [9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
>>> [10] https://github.com/chrislusf/seaweedfs/wiki/Components
>>> [11] https://forge.softwareheritage.org/T3049
>>> [12] https://forge.softwareheritage.org/T3054#58977
>>>
>>> --
>>> Loïc Dachary, Artisan Logiciel Libre
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Loïc Dachary, Artisan Logiciel Libre
9:07 a.m.
I still prefer the simplest solution. There are 4U servers with 110 x
20TB disks on the market.
After raid you get 1.5PiB per server. This is 30 months of data.
2 such servers will hold 5 years of data with minimal problems.
If you need backup; then buy 2 more sets and just send zfs snapshot
diffs to this set.
On Wed, Feb 17, 2021 at 11:15 PM Loïc Dachary <loic(a)dachary.org> wrote:
>
>
>
> On 17/02/2021 18:27, Serkan Çoban wrote:
> > Why not put all the data to a zfs pool with 3-4 levels deep directory
> > structure each directory named with 2 byte in range 00-FF?
> > Four levels deep, you get 255^4=4B folders with 3-4 objects per folder
> > or three levels deep you get 255^3=16M folders with ~1000 objects
> > each.
> It is more or less the current setup :-) I should have mentioned that there currently
are ~750TB and 10 billions objects. But it's growing by 50TB every month and it will
keep growing indefinitely. Reason why a solution that scales out is desirable.
> >
> > On Wed, Feb 17, 2021 at 8:14 PM Loïc Dachary <loic(a)dachary.org> wrote:
> >> Hi Nathan,
> >>
> >> Good thinking :-) The names of the objects are indeed the SHA256 of their
content, which provides deduplication.
> >>
> >> Cheers
> >>
> >> On 17/02/2021 18:04, Nathan Fish wrote:
> >>> I'm not much of a programmer, but as soon as I hear "immutable
> >>> objects" I think "content-addressed". I don't know if
you have many
> >>> duplicate objects in this set, but content-addressing gives you
> >>> object-level dedup for free. Do you have to preserve some meaningful
> >>> object names from the original dataset, or just do you just need some
> >>> kind of ID?
> >>>
> >>> On Wed, Feb 17, 2021 at 11:37 AM Loïc Dachary <loic(a)dachary.org>
wrote:
> >>>> Bonjour,
> >>>>
> >>>> TL;DR: Is it more advisable to work on Ceph internals to make it
friendly to this particular workload or write something similar to EOS[0] (i.e Rocksdb +
Xrootd + RBD)?
> >>>>
> >>>> This is a followup of two previous mails[1] sent while researching
this topic. In a nutshell, the Software Heritage project[1] currently has ~750TB and 10
billions objects, 75% of which have a size smaller than 16KB and 50% have a size smaller
than 4KB. But they only account for ~5% of the 750TB: 25% of the objects have a size >
16KB and total ~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of
actual storage. (if you're interested in the details see [2]).
> >>>>
> >>>> Let say those 10 billions objects are stored in a single 4+2 erasure
coded pool with bluestore compression set for objects that have a size > 32KB and the
smallest allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected
350TB but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is
because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to
repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use
no less than 4KB because that's the smallest allocation size for bluestore. That's
4 * 4KB = 16KB even when all that is needed is 1 byte.
> >>>>
> >>>> It was suggested[6] to have two different pools: one with a 4+2
erasure pool and compression for all objects with a size > 32KB that are expected to
compress to 16KB. And another with 3 replicas for the smaller objects to reduce space
amplification to a minimum without compromising on durability. A client looking for the
object could make two simultaneous requests to the two pools. They would get 404 from one
of them and the object from the other.
> >>>>
> >>>> Another workaround, is best described in the "Finding a needle
in Haystack: Facebook’s photo storage"[9] paper and essentially boils down to using a
database to store a map between the object name and its location. That does not scale out
(writing the database index is the bottleneck) but it's simple enough and is
successfully implemented in EOS[0] with >200PB worth of data and in seaweedfs[10],
another promising object store software based on the same idea.
> >>>>
> >>>> Instead of working around the problem, maybe Ceph could be modified
to make better use of the immutability of these objects[7], a hint that is apparently only
used to figure out how to best compress it and for checksum calculation[8]. I honestly
have not clue how difficult it would be. All I know is that it's not easy otherwise it
would have been done already: there seem to be a general need for efficiently (space wise
and performance wise) storing large quantities of objects smaller than 4KB.
> >>>>
> >>>> Is it more advisable to:
> >>>>
> >>>> * work on Ceph internals to make it friendly to this particular
workload or,
> >>>> * write another implementation of "Finding a needle in
Haystack: Facebook’s photo storage"[9] based on RBD[11]?
> >>>>
> >>>> I'm currently leaning toward working on Ceph internals but there
are pros and cons to both approaches[12]. And since all this is still very new to me,
there also is the possibility that I'm missing something. Maybe it's *super*
difficult to improve Ceph in this way. I should try to figure that out sooner rather than
later.
> >>>>
> >>>> I realize it's a lot to take in and unless you're facing the
exact same problem there is very little chance you read that far :-) But if you did...
I'm *really* interested to hear what yout think. In any case I'll report back to
this thread once a decision has been made.
> >>>>
> >>>> Cheers
> >>>>
> >>>> [0] https://eos-web.web.cern.ch/eos-web/
> >>>> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
> >>>> [2] https://forge.softwareheritage.org/T3054
> >>>> [3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
> >>>> [4] https://forge.softwareheritage.org/T3052#58864
> >>>> [5] https://forge.softwareheritage.org/T3052#58917
> >>>> [6] https://forge.softwareheritage.org/T3052#58876
> >>>> [7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
> >>>> [8] https://forge.softwareheritage.org/T3055
> >>>> [9]
https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
> >>>> [10] https://github.com/chrislusf/seaweedfs/wiki/Components
> >>>> [11] https://forge.softwareheritage.org/T3049
> >>>> [12] https://forge.softwareheritage.org/T3054#58977
> >>>>
> >>>> --
> >>>> Loïc Dachary, Artisan Logiciel Libre
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ceph-users mailing list -- ceph-users(a)ceph.io
> >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >> --
> >> Loïc Dachary, Artisan Logiciel Libre
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
20 Feb
20 Feb
6:01 p.m.
I did not know it was possible to buy such a machine, very impressive. How much does that
cost? I thought 18TB was the current maximum size for HDD :-P
On 18/02/2021 04:37, Serkan Çoban wrote:
I still prefer the simplest solution. There are 4U
servers with 110 x
20TB disks on the market.
After raid you get 1.5PiB per server. This is 30 months of data.
2 such servers will hold 5 years of data with minimal problems.
If you need backup; then buy 2 more sets and just send zfs snapshot
diffs to this set.
On Wed, Feb 17, 2021 at 11:15 PM Loïc Dachary <loic(a)dachary.org> wrote:
>
>
> On 17/02/2021 18:27, Serkan Çoban wrote:
>> Why not put all the data to a zfs pool with 3-4 levels deep directory
>> structure each directory named with 2 byte in range 00-FF?
>> Four levels deep, you get 255^4=4B folders with 3-4 objects per folder
>> or three levels deep you get 255^3=16M folders with ~1000 objects
>> each.
> It is more or less the current setup :-) I should have mentioned that there currently
are ~750TB and 10 billions objects. But it's growing by 50TB every month and it will
keep growing indefinitely. Reason why a solution that scales out is desirable.
>> On Wed, Feb 17, 2021 at 8:14 PM Loïc Dachary <loic(a)dachary.org> wrote:
>>> Hi Nathan,
>>>
>>> Good thinking :-) The names of the objects are indeed the SHA256 of their
content, which provides deduplication.
>>>
>>> Cheers
>>>
>>> On 17/02/2021 18:04, Nathan Fish wrote:
>>>> I'm not much of a programmer, but as soon as I hear "immutable
>>>> objects" I think "content-addressed". I don't know if
you have many
>>>> duplicate objects in this set, but content-addressing gives you
>>>> object-level dedup for free. Do you have to preserve some meaningful
>>>> object names from the original dataset, or just do you just need some
>>>> kind of ID?
>>>>
>>>> On Wed, Feb 17, 2021 at 11:37 AM Loïc Dachary <loic(a)dachary.org>
wrote:
>>>>> Bonjour,
>>>>>
>>>>> TL;DR: Is it more advisable to work on Ceph internals to make it
friendly to this particular workload or write something similar to EOS[0] (i.e Rocksdb +
Xrootd + RBD)?
>>>>>
>>>>> This is a followup of two previous mails[1] sent while researching
this topic. In a nutshell, the Software Heritage project[1] currently has ~750TB and 10
billions objects, 75% of which have a size smaller than 16KB and 50% have a size smaller
than 4KB. But they only account for ~5% of the 750TB: 25% of the objects have a size >
16KB and total ~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of
actual storage. (if you're interested in the details see [2]).
>>>>>
>>>>> Let say those 10 billions objects are stored in a single 4+2 erasure
coded pool with bluestore compression set for objects that have a size > 32KB and the
smallest allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected
350TB but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is
because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to
repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use
no less than 4KB because that's the smallest allocation size for bluestore. That's
4 * 4KB = 16KB even when all that is needed is 1 byte.
>>>>>
>>>>> It was suggested[6] to have two different pools: one with a 4+2
erasure pool and compression for all objects with a size > 32KB that are expected to
compress to 16KB. And another with 3 replicas for the smaller objects to reduce space
amplification to a minimum without compromising on durability. A client looking for the
object could make two simultaneous requests to the two pools. They would get 404 from one
of them and the object from the other.
>>>>>
>>>>> Another workaround, is best described in the "Finding a needle
in Haystack: Facebook’s photo storage"[9] paper and essentially boils down to using a
database to store a map between the object name and its location. That does not scale out
(writing the database index is the bottleneck) but it's simple enough and is
successfully implemented in EOS[0] with >200PB worth of data and in seaweedfs[10],
another promising object store software based on the same idea.
>>>>>
>>>>> Instead of working around the problem, maybe Ceph could be modified
to make better use of the immutability of these objects[7], a hint that is apparently only
used to figure out how to best compress it and for checksum calculation[8]. I honestly
have not clue how difficult it would be. All I know is that it's not easy otherwise it
would have been done already: there seem to be a general need for efficiently (space wise
and performance wise) storing large quantities of objects smaller than 4KB.
>>>>>
>>>>> Is it more advisable to:
>>>>>
>>>>> * work on Ceph internals to make it friendly to this particular
workload or,
>>>>> * write another implementation of "Finding a needle in
Haystack: Facebook’s photo storage"[9] based on RBD[11]?
>>>>>
>>>>> I'm currently leaning toward working on Ceph internals but there
are pros and cons to both approaches[12]. And since all this is still very new to me,
there also is the possibility that I'm missing something. Maybe it's *super*
difficult to improve Ceph in this way. I should try to figure that out sooner rather than
later.
>>>>>
>>>>> I realize it's a lot to take in and unless you're facing the
exact same problem there is very little chance you read that far :-) But if you did...
I'm *really* interested to hear what yout think. In any case I'll report back to
this thread once a decision has been made.
>>>>>
>>>>> Cheers
>>>>>
>>>>> [0] https://eos-web.web.cern.ch/eos-web/
>>>>> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
>>>>> [2] https://forge.softwareheritage.org/T3054
>>>>> [3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
>>>>> [4] https://forge.softwareheritage.org/T3052#58864
>>>>> [5] https://forge.softwareheritage.org/T3052#58917
>>>>> [6] https://forge.softwareheritage.org/T3052#58876
>>>>> [7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
>>>>> [8] https://forge.softwareheritage.org/T3055
>>>>> [9]
https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
>>>>> [10] https://github.com/chrislusf/seaweedfs/wiki/Components
>>>>> [11] https://forge.softwareheritage.org/T3049
>>>>> [12] https://forge.softwareheritage.org/T3054#58977
>>>>>
>>>>> --
>>>>> Loïc Dachary, Artisan Logiciel Libre
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> --
>>> Loïc Dachary, Artisan Logiciel Libre
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
--
Loïc Dachary, Artisan Logiciel Libre
10:30 p.m.
You pay $300-$350 per HDD when you buy in large quantities. So such a
server costs $50k-$60k.
There are 20TB drives but they are SMR(1), they are ok for archival
use cases, ceph does not recommend SMR drives.
We use 90 disk 4u servers(2), I remember HPE showed us 110 drive
servers but can't find them now.
Seagate also has 4u 106 drive JBOD enclosure(3)
1-
https://www.westerndigital.com/products/data-center-platforms/ultrastar-dc-…
2-
https://www.delltechnologies.com/no-no/collaterals/unauth/data-sheets/produ…
3-
https://www.seagate.com/files/www-content/datasheets/pdfs/exos-e-4u106-DS19…
On Sat, Feb 20, 2021 at 3:31 PM Loïc Dachary <loic(a)dachary.org> wrote:
>
> I did not know it was possible to buy such a machine, very impressive. How much does
that cost? I thought 18TB was the current maximum size for HDD :-P
>
> On 18/02/2021 04:37, Serkan Çoban wrote:
> > I still prefer the simplest solution. There are 4U servers with 110 x
> > 20TB disks on the market.
> > After raid you get 1.5PiB per server. This is 30 months of data.
> > 2 such servers will hold 5 years of data with minimal problems.
> > If you need backup; then buy 2 more sets and just send zfs snapshot
> > diffs to this set.
> >
> >
> > On Wed, Feb 17, 2021 at 11:15 PM Loïc Dachary <loic(a)dachary.org> wrote:
> >>
> >>
> >> On 17/02/2021 18:27, Serkan Çoban wrote:
> >>> Why not put all the data to a zfs pool with 3-4 levels deep directory
> >>> structure each directory named with 2 byte in range 00-FF?
> >>> Four levels deep, you get 255^4=4B folders with 3-4 objects per folder
> >>> or three levels deep you get 255^3=16M folders with ~1000 objects
> >>> each.
> >> It is more or less the current setup :-) I should have mentioned that there
currently are ~750TB and 10 billions objects. But it's growing by 50TB every month and
it will keep growing indefinitely. Reason why a solution that scales out is desirable.
> >>> On Wed, Feb 17, 2021 at 8:14 PM Loïc Dachary <loic(a)dachary.org>
wrote:
> >>>> Hi Nathan,
> >>>>
> >>>> Good thinking :-) The names of the objects are indeed the SHA256 of
their content, which provides deduplication.
> >>>>
> >>>> Cheers
> >>>>
> >>>> On 17/02/2021 18:04, Nathan Fish wrote:
> >>>>> I'm not much of a programmer, but as soon as I hear
"immutable
> >>>>> objects" I think "content-addressed". I don't
know if you have many
> >>>>> duplicate objects in this set, but content-addressing gives you
> >>>>> object-level dedup for free. Do you have to preserve some
meaningful
> >>>>> object names from the original dataset, or just do you just need
some
> >>>>> kind of ID?
> >>>>>
> >>>>> On Wed, Feb 17, 2021 at 11:37 AM Loïc Dachary
<loic(a)dachary.org> wrote:
> >>>>>> Bonjour,
> >>>>>>
> >>>>>> TL;DR: Is it more advisable to work on Ceph internals to
make it friendly to this particular workload or write something similar to EOS[0] (i.e
Rocksdb + Xrootd + RBD)?
> >>>>>>
> >>>>>> This is a followup of two previous mails[1] sent while
researching this topic. In a nutshell, the Software Heritage project[1] currently has
~750TB and 10 billions objects, 75% of which have a size smaller than 16KB and 50% have a
size smaller than 4KB. But they only account for ~5% of the 750TB: 25% of the objects have
a size > 16KB and total ~700TB. The objects can be compressed by ~50% and 750TB only
needs 350TB of actual storage. (if you're interested in the details see [2]).
> >>>>>>
> >>>>>> Let say those 10 billions objects are stored in a single 4+2
erasure coded pool with bluestore compression set for objects that have a size > 32KB
and the smallest allocation size for bluestore set to 4KB[3]. The 750TB won't use the
expected 350TB but about 30% more, i.e. ~450TB (see [4] for the maths). This space
amplification is because storing a 1 byte object uses the same space as storing a 16KB
object (see [5] to repeat the experience at home). In a 4+2 erasure coded pool, each of
the 6 chunks will use no less than 4KB because that's the smallest allocation size for
bluestore. That's 4 * 4KB = 16KB even when all that is needed is 1 byte.
> >>>>>>
> >>>>>> It was suggested[6] to have two different pools: one with a
4+2 erasure pool and compression for all objects with a size > 32KB that are expected
to compress to 16KB. And another with 3 replicas for the smaller objects to reduce space
amplification to a minimum without compromising on durability. A client looking for the
object could make two simultaneous requests to the two pools. They would get 404 from one
of them and the object from the other.
> >>>>>>
> >>>>>> Another workaround, is best described in the "Finding a
needle in Haystack: Facebook’s photo storage"[9] paper and essentially boils down to
using a database to store a map between the object name and its location. That does not
scale out (writing the database index is the bottleneck) but it's simple enough and is
successfully implemented in EOS[0] with >200PB worth of data and in seaweedfs[10],
another promising object store software based on the same idea.
> >>>>>>
> >>>>>> Instead of working around the problem, maybe Ceph could be
modified to make better use of the immutability of these objects[7], a hint that is
apparently only used to figure out how to best compress it and for checksum
calculation[8]. I honestly have not clue how difficult it would be. All I know is that
it's not easy otherwise it would have been done already: there seem to be a general
need for efficiently (space wise and performance wise) storing large quantities of objects
smaller than 4KB.
> >>>>>>
> >>>>>> Is it more advisable to:
> >>>>>>
> >>>>>> * work on Ceph internals to make it friendly to this
particular workload or,
> >>>>>> * write another implementation of "Finding a needle
in Haystack: Facebook’s photo storage"[9] based on RBD[11]?
> >>>>>>
> >>>>>> I'm currently leaning toward working on Ceph internals
but there are pros and cons to both approaches[12]. And since all this is still very new
to me, there also is the possibility that I'm missing something. Maybe it's
*super* difficult to improve Ceph in this way. I should try to figure that out sooner
rather than later.
> >>>>>>
> >>>>>> I realize it's a lot to take in and unless you're
facing the exact same problem there is very little chance you read that far :-) But if you
did... I'm *really* interested to hear what yout think. In any case I'll report
back to this thread once a decision has been made.
> >>>>>>
> >>>>>> Cheers
> >>>>>>
> >>>>>> [0] https://eos-web.web.cern.ch/eos-web/
> >>>>>> [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
> >>>>>> [2] https://forge.softwareheritage.org/T3054
> >>>>>> [3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
> >>>>>> [4] https://forge.softwareheritage.org/T3052#58864
> >>>>>> [5] https://forge.softwareheritage.org/T3052#58917
> >>>>>> [6] https://forge.softwareheritage.org/T3052#58876
> >>>>>> [7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
> >>>>>> [8] https://forge.softwareheritage.org/T3055
> >>>>>> [9]
https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
> >>>>>> [10] https://github.com/chrislusf/seaweedfs/wiki/Components
> >>>>>> [11] https://forge.softwareheritage.org/T3049
> >>>>>> [12] https://forge.softwareheritage.org/T3054#58977
> >>>>>>
> >>>>>> --
> >>>>>> Loïc Dachary, Artisan Logiciel Libre
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
> >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >>>> --
> >>>> Loïc Dachary, Artisan Logiciel Libre
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ceph-users mailing list -- ceph-users(a)ceph.io
> >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >> --
> >> Loïc Dachary, Artisan Logiciel Libre
> >>
> >>
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
18 Feb
18 Feb
5:05 a.m.
On Wed, Feb 17, 2021 at 05:36:53PM +0100, Loïc Dachary wrote:
Bonjour,
TL;DR: Is it more advisable to work on Ceph internals to make it
friendly to this particular workload or write something similar to
EOS[0] (i.e Rocksdb + Xrootd + RBD)?
CERN's EOSPPC instance, which is one of
the biggest from what I can
find, was up around 3.5B files in 2019; and you're proposing running 10B
files, so I don't know how EOS will handle that. Maybe Dan can chime in
on the scalability there.
Please do keep on this important work! I've tried to do something
similar at a much smaller scale for Gentoo Linux's historical collection
of source code media (distfiles), but am significantly further behind
your effort.
Let say those 10 billions objects are stored in a
single 4+2 erasure
coded pool with bluestore compression set for objects that have a size
32KB and the smallest allocation size for bluestore set to 4KB[3].
The 750TB won't use the expected 350TB but about 30% more, i.e.
~450TB (see [4] for the maths). This space amplification is because
storing a 1 byte object uses the same space as storing a 16KB object
(see [5] to repeat the experience at home). In a 4+2 erasure coded
pool, each of the 6 chunks will use no less than 4KB because that's
the smallest allocation size for bluestore. That's 4 * 4KB = 16KB
even when all that is needed is 1 byte.
I think you have an error here: with 4KB
allocation size in 4+2 pool,
any object sized (0,16K] will take _6_ chunks: 20KB of storage.
Any object sized (16K,32K] will take _12_ chunks: 40K of storage.
I'd attack this from another side entirely:
- how aggressively do you want to pack objects overall? e.g. if you have
a few thousand objects in the 4-5K range, do you want zero bytes
wasted between objects?
- how aggressively do you want to dudup objects that share common data,
esp if it's not aligned on some common byte margins?
- what are the data portability requirements to move/extract data from
this system at a later point?
- how complex of an index are you willing to maintain to
reconstruct/access data?
- What requirements are there about the ordering and accessibility of
the packs? How related do the pack objects need to be? e.g. are the
packed as they arrive in time order, to build up successive packs of
size, or are there many packs and you append the "correct" pack for a
given object?
I'm normally distinctly in the camp that object storage systems should
natively expose all objects, but that also doesn't account for your
immutability/append-only nature.
I see your discussion at https://forge.softwareheritage.org/T3054#58977
as well, about the "full scale out" vs "scale up metadata & scale out
data" parts.
To brainstorm parts of an idea, I'm wondering about Git's
still-in-development partial clone work, with the caveat that you intend
to NEVER checkout the entire repository at the same time.
Ideally, using some manner of fuse filesystem (similar to Git Virtual
Filesystem) w/ an index-only clone, naive clients could access the
object they wanted, which would be fetched on demand from the git server
which has mostly git packs and a few sparse objects that are waiting for
packing.
The write path on ingest clients would involve sending back the new
data, and git background processes on some regular interval packing the
loose objects into new packfiles.
Running this on top of CephFS for now means that you get the ability to
move it to future storage systems more easily than any custom RBD/EOS
development you might do: bring up enough space, sync the files over,
profit.
Git handles the deduplication, compression, access methods, and
generates large pack files, which Ceph can store more optimally than the
plethora of tiny objects.
Overall, this isn't great, but there aren't a lot of alternatives as
your great research has noted.
Being able to take a backup of the Git-on-CephFS is also made a lot
easier since it's a filesystem: "just" write out the 350TB to 20x LTO-9
tapes
Thinking back to older systems, like SGI's hierarchal storage modules
for XFS, the packing overhead starts to become significant for your
objects: some of the underlying mechanisms in the XFS HSM DMAPI, if they
ended up packing immutable objects to tape still had tar & tar-like
headers (at least 512 bytes per object), your 10B objects would take at
least 4TB of extra space (before compression).
It was suggested[6] to have two different pools: one
with a 4+2 erasure pool and compression for all objects with a size > 32KB that are
expected to compress to 16KB. And another with 3 replicas for the smaller objects to
reduce space amplification to a minimum without compromising on durability. A client
looking for the object could make two simultaneous requests to the two pools. They would
get 404 from one of them and the object from the other.
Another workaround, is best described in the "Finding a needle in Haystack:
Facebook’s photo storage"[9] paper and essentially boils down to using a database to
store a map between the object name and its location. That does not scale out (writing the
database index is the bottleneck) but it's simple enough and is successfully
implemented in EOS[0] with >200PB worth of data and in seaweedfs[10], another promising
object store software based on the same idea.
Instead of working around the problem, maybe Ceph could be modified to make better use of
the immutability of these objects[7], a hint that is apparently only used to figure out
how to best compress it and for checksum calculation[8]. I honestly have not clue how
difficult it would be. All I know is that it's not easy otherwise it would have been
done already: there seem to be a general need for efficiently (space wise and performance
wise) storing large quantities of objects smaller than 4KB.
Is it more advisable to:
* work on Ceph internals to make it friendly to this particular workload or,
* write another implementation of "Finding a needle in Haystack: Facebook’s photo
storage"[9] based on RBD[11]?
I'm currently leaning toward working on Ceph internals but there are pros and cons to
both approaches[12]. And since all this is still very new to me, there also is the
possibility that I'm missing something. Maybe it's *super* difficult to improve
Ceph in this way. I should try to figure that out sooner rather than later.
I realize it's a lot to take in and unless you're facing the exact same problem
there is very little chance you read that far :-) But if you did... I'm *really*
interested to hear what yout think. In any case I'll report back to this thread once a
decision has been made.
Cheers
[0] https://eos-web.web.cern.ch/eos-web/
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
[2] https://forge.softwareheritage.org/T3054
[3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
[4] https://forge.softwareheritage.org/T3052#58864
[5] https://forge.softwareheritage.org/T3052#58917
[6] https://forge.softwareheritage.org/T3052#58876
[7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
[8] https://forge.softwareheritage.org/T3055
[9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
[10] https://github.com/chrislusf/seaweedfs/wiki/Components
[11] https://forge.softwareheritage.org/T3049
[12] https://forge.softwareheritage.org/T3054#58977
--
Loïc Dachary, Artisan Logiciel Libre
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2(a)gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
10:25 p.m.
On Thu, Feb 18, 2021 at 12:36 AM Robin H. Johnson <robbat2(a)gentoo.org> wrote:
On Wed, Feb 17, 2021 at 05:36:53PM +0100, Loïc Dachary wrote:
The EOS namespace is now QuarkDB https://github.com/gbitzes/QuarkDB
But even with a clever namespace I don't think it is practical to
manage a system with 10B tiny files.
Enumerating them for a consistency check or migrating between hosts or
recovering from failures is going to be painful.
Pack them...
-- Dan
>
> Please do keep on this important work! I've tried to do something
> similar at a much smaller scale for Gentoo Linux's historical collection
> of source code media (distfiles), but am significantly further behind
> your effort.
>
> > Let say those 10 billions objects are stored in a single 4+2 erasure
> > coded pool with bluestore compression set for objects that have a size
> > 32KB and the smallest allocation size for bluestore set to 4KB[3].
> > The 750TB won't use the expected 350TB but about 30% more, i.e.
> > ~450TB (see [4] for the maths). This space amplification is because
> > storing a 1 byte object uses the same space as storing a 16KB object
> > (see [5] to repeat the experience at home). In a 4+2 erasure coded
> > pool, each of the 6 chunks will use no less than 4KB because that's
> > the smallest allocation size for bluestore. That's 4 * 4KB = 16KB
> > even when all that is needed is 1 byte.
> I think you have an error here: with 4KB allocation size in 4+2 pool,
> any object sized (0,16K] will take _6_ chunks: 20KB of storage.
> Any object sized (16K,32K] will take _12_ chunks: 40K of storage.
>
> I'd attack this from another side entirely:
> - how aggressively do you want to pack objects overall? e.g. if you have
> a few thousand objects in the 4-5K range, do you want zero bytes
> wasted between objects?
> - how aggressively do you want to dudup objects that share common data,
> esp if it's not aligned on some common byte margins?
> - what are the data portability requirements to move/extract data from
> this system at a later point?
> - how complex of an index are you willing to maintain to
> reconstruct/access data?
> - What requirements are there about the ordering and accessibility of
> the packs? How related do the pack objects need to be? e.g. are the
> packed as they arrive in time order, to build up successive packs of
> size, or are there many packs and you append the "correct" pack for a
> given object?
>
> I'm normally distinctly in the camp that object storage systems should
> natively expose all objects, but that also doesn't account for your
> immutability/append-only nature.
>
> I see your discussion at https://forge.softwareheritage.org/T3054#58977
> as well, about the "full scale out" vs "scale up metadata & scale
out
> data" parts.
>
> To brainstorm parts of an idea, I'm wondering about Git's
> still-in-development partial clone work, with the caveat that you intend
> to NEVER checkout the entire repository at the same time.
>
> Ideally, using some manner of fuse filesystem (similar to Git Virtual
> Filesystem) w/ an index-only clone, naive clients could access the
> object they wanted, which would be fetched on demand from the git server
> which has mostly git packs and a few sparse objects that are waiting for
> packing.
>
> The write path on ingest clients would involve sending back the new
> data, and git background processes on some regular interval packing the
> loose objects into new packfiles.
>
> Running this on top of CephFS for now means that you get the ability to
> move it to future storage systems more easily than any custom RBD/EOS
> development you might do: bring up enough space, sync the files over,
> profit.
>
> Git handles the deduplication, compression, access methods, and
> generates large pack files, which Ceph can store more optimally than the
> plethora of tiny objects.
>
> Overall, this isn't great, but there aren't a lot of alternatives as
> your great research has noted.
>
> Being able to take a backup of the Git-on-CephFS is also made a lot
> easier since it's a filesystem: "just" write out the 350TB to 20x
LTO-9
> tapes
>
> Thinking back to older systems, like SGI's hierarchal storage modules
> for XFS, the packing overhead starts to become significant for your
> objects: some of the underlying mechanisms in the XFS HSM DMAPI, if they
> ended up packing immutable objects to tape still had tar & tar-like
> headers (at least 512 bytes per object), your 10B objects would take at
> least 4TB of extra space (before compression).
>
>
> > It was suggested[6] to have two different pools: one with a 4+2 erasure pool and
compression for all objects with a size > 32KB that are expected to compress to 16KB.
And another with 3 replicas for the smaller objects to reduce space amplification to a
minimum without compromising on durability. A client looking for the object could make two
simultaneous requests to the two pools. They would get 404 from one of them and the object
from the other.
> >
> > Another workaround, is best described in the "Finding a needle in Haystack:
Facebook’s photo storage"[9] paper and essentially boils down to using a database to
store a map between the object name and its location. That does not scale out (writing the
database index is the bottleneck) but it's simple enough and is successfully
implemented in EOS[0] with >200PB worth of data and in seaweedfs[10], another promising
object store software based on the same idea.
> >
> > Instead of working around the problem, maybe Ceph could be modified to make
better use of the immutability of these objects[7], a hint that is apparently only used to
figure out how to best compress it and for checksum calculation[8]. I honestly have not
clue how difficult it would be. All I know is that it's not easy otherwise it would
have been done already: there seem to be a general need for efficiently (space wise and
performance wise) storing large quantities of objects smaller than 4KB.
> >
> > Is it more advisable to:
> >
> > * work on Ceph internals to make it friendly to this particular workload or,
> > * write another implementation of "Finding a needle in Haystack:
Facebook’s photo storage"[9] based on RBD[11]?
> >
> > I'm currently leaning toward working on Ceph internals but there are pros
and cons to both approaches[12]. And since all this is still very new to me, there also is
the possibility that I'm missing something. Maybe it's *super* difficult to
improve Ceph in this way. I should try to figure that out sooner rather than later.
> >
> > I realize it's a lot to take in and unless you're facing the exact same
problem there is very little chance you read that far :-) But if you did... I'm
*really* interested to hear what yout think. In any case I'll report back to this
thread once a decision has been made.
> >
> > Cheers
> >
> > [0] https://eos-web.web.cern.ch/eos-web/
> > [1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
> > [2] https://forge.softwareheritage.org/T3054
> > [3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
> > [4] https://forge.softwareheritage.org/T3052#58864
> > [5] https://forge.softwareheritage.org/T3052#58917
> > [6] https://forge.softwareheritage.org/T3052#58876
> > [7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
> > [8] https://forge.softwareheritage.org/T3055
> > [9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
> > [10] https://github.com/chrislusf/seaweedfs/wiki/Components
> > [11] https://forge.softwareheritage.org/T3049
> > [12] https://forge.softwareheritage.org/T3054#58977
> >
> > --
> > Loïc Dachary, Artisan Logiciel Libre
> >
> >
>
>
>
>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> --
> Robin Hugh Johnson
> Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
> E-Mail : robbat2(a)gentoo.org
> GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
> GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
Bonjour,
TL;DR: Is it more advisable to work on Ceph internals to make it
friendly to this particular workload or write something similar to
EOS[0] (i.e Rocksdb + Xrootd + RBD)?
CERN's EOSPPC instance, which is one of
the biggest from what I can
find, was up around 3.5B files in 2019; and you're proposing running 10B
files, so I don't know how EOS will handle that. Maybe Dan can chime in
on the scalability there.
20 Feb
20 Feb
9:11 p.m.
On 18/02/2021 17:55, Dan van der Ster wrote:
--
Loïc Dachary, Artisan Logiciel Libre
On Thu, Feb 18, 2021 at 12:36 AM Robin H. Johnson
<robbat2(a)gentoo.org> wrote:
Thanks for the update and the wise advice Dan :-)
On Wed, Feb 17, 2021 at 05:36:53PM +0100, Loïc
Dachary wrote:
The EOS namespace is now QuarkDB
https://github.com/gbitzes/QuarkDB
But even with a clever namespace I don't think it is practical to
manage a system with 10B tiny files.
Enumerating them for a consistency check or migrating between hosts or
recovering from failures is going to be painful.
Pack them...
-- Dan Bonjour,
TL;DR: Is it more advisable to work on Ceph internals to make it
friendly to this particular workload or write something similar to
EOS[0] (i.e Rocksdb + Xrootd + RBD)?
CERN's EOSPPC instance, which is one of
the biggest from what I can
find, was up around 3.5B files in 2019; and you're proposing running 10B
files, so I don't know how EOS will handle that. Maybe Dan can chime in
on the scalability there. Please do keep on this important work! I've
tried to do something
similar at a much smaller scale for Gentoo Linux's historical collection
of source code media (distfiles), but am significantly further behind
your effort.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Let say those 10 billions objects are stored in a
single 4+2 erasure
coded pool with bluestore compression set for objects that have a size
32KB and the smallest allocation size for bluestore set to 4KB[3].
The 750TB won't use the expected 350TB but about 30% more, i.e.
~450TB (see [4] for the maths). This space amplification is because
storing a 1 byte object uses the same space as storing a 16KB object
(see [5] to repeat the experience at home). In a 4+2 erasure coded
pool, each of the 6 chunks will use no less than 4KB because that's
the smallest allocation size for bluestore. That's 4 * 4KB = 16KB
even when all that is needed is 1 byte.
I think you have an error here: with 4KB
allocation size in 4+2 pool,
any object sized (0,16K] will take _6_ chunks: 20KB of storage.
Any object sized (16K,32K] will take _12_ chunks: 40K of storage.
I'd attack this from another side entirely:
- how aggressively do you want to pack objects overall? e.g. if you have
a few thousand objects in the 4-5K range, do you want zero bytes
wasted between objects?
- how aggressively do you want to dudup objects that share common data,
esp if it's not aligned on some common byte margins?
- what are the data portability requirements to move/extract data from
this system at a later point?
- how complex of an index are you willing to maintain to
reconstruct/access data?
- What requirements are there about the ordering and accessibility of
the packs? How related do the pack objects need to be? e.g. are the
packed as they arrive in time order, to build up successive packs of
size, or are there many packs and you append the "correct" pack for a
given object?
I'm normally distinctly in the camp that object storage systems should
natively expose all objects, but that also doesn't account for your
immutability/append-only nature.
I see your discussion at https://forge.softwareheritage.org/T3054#58977
as well, about the "full scale out" vs "scale up metadata & scale out
data" parts.
To brainstorm parts of an idea, I'm wondering about Git's
still-in-development partial clone work, with the caveat that you intend
to NEVER checkout the entire repository at the same time.
Ideally, using some manner of fuse filesystem (similar to Git Virtual
Filesystem) w/ an index-only clone, naive clients could access the
object they wanted, which would be fetched on demand from the git server
which has mostly git packs and a few sparse objects that are waiting for
packing.
The write path on ingest clients would involve sending back the new
data, and git background processes on some regular interval packing the
loose objects into new packfiles.
Running this on top of CephFS for now means that you get the ability to
move it to future storage systems more easily than any custom RBD/EOS
development you might do: bring up enough space, sync the files over,
profit.
Git handles the deduplication, compression, access methods, and
generates large pack files, which Ceph can store more optimally than the
plethora of tiny objects.
Overall, this isn't great, but there aren't a lot of alternatives as
your great research has noted.
Being able to take a backup of the Git-on-CephFS is also made a lot
easier since it's a filesystem: "just" write out the 350TB to 20x LTO-9
tapes
Thinking back to older systems, like SGI's hierarchal storage modules
for XFS, the packing overhead starts to become significant for your
objects: some of the underlying mechanisms in the XFS HSM DMAPI, if they
ended up packing immutable objects to tape still had tar & tar-like
headers (at least 512 bytes per object), your 10B objects would take at
least 4TB of extra space (before compression).
It was suggested[6] to have two different pools:
one with a 4+2 erasure pool and compression for all objects with a size > 32KB that are
expected to compress to 16KB. And another with 3 replicas for the smaller objects to
reduce space amplification to a minimum without compromising on durability. A client
looking for the object could make two simultaneous requests to the two pools. They would
get 404 from one of them and the object from the other.
Another workaround, is best described in the "Finding a needle in Haystack:
Facebook’s photo storage"[9] paper and essentially boils down to using a database to
store a map between the object name and its location. That does not scale out (writing the
database index is the bottleneck) but it's simple enough and is successfully
implemented in EOS[0] with >200PB worth of data and in seaweedfs[10], another promising
object store software based on the same idea.
Instead of working around the problem, maybe Ceph could be modified to make better use of
the immutability of these objects[7], a hint that is apparently only used to figure out
how to best compress it and for checksum calculation[8]. I honestly have not clue how
difficult it would be. All I know is that it's not easy otherwise it would have been
done already: there seem to be a general need for efficiently (space wise and performance
wise) storing large quantities of objects smaller than 4KB.
Is it more advisable to:
* work on Ceph internals to make it friendly to this particular workload or,
* write another implementation of "Finding a needle in Haystack: Facebook’s photo
storage"[9] based on RBD[11]?
I'm currently leaning toward working on Ceph internals but there are pros and cons to
both approaches[12]. And since all this is still very new to me, there also is the
possibility that I'm missing something. Maybe it's *super* difficult to improve
Ceph in this way. I should try to figure that out sooner rather than later.
I realize it's a lot to take in and unless you're facing the exact same problem
there is very little chance you read that far :-) But if you did... I'm *really*
interested to hear what yout think. In any case I'll report back to this thread once a
decision has been made.
Cheers
[0] https://eos-web.web.cern.ch/eos-web/
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
[2] https://forge.softwareheritage.org/T3054
[3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
[4] https://forge.softwareheritage.org/T3052#58864
[5] https://forge.softwareheritage.org/T3052#58917
[6] https://forge.softwareheritage.org/T3052#58876
[7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
[8] https://forge.softwareheritage.org/T3055
[9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
[10] https://github.com/chrislusf/seaweedfs/wiki/Components
[11] https://forge.softwareheritage.org/T3049
[12] https://forge.softwareheritage.org/T3054#58977
--
Loïc Dachary, Artisan Logiciel Libre
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2(a)gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:59 p.m.
Hi Robin,
On 18/02/2021 00:35, Robin H. Johnson wrote:
I should have
mentioned that my calculations were ignoring the replication overhead (parity chunks or
copies). Good catch :-)
On Wed, Feb 17, 2021 at 05:36:53PM +0100, Loïc Dachary
wrote:
This is an essential piece of information I was missing.
It also makes sense that there are much larger objects in the context of the CERN.
Bonjour,
TL;DR: Is it more advisable to work on Ceph internals to make it
friendly to this particular workload or write something similar to
EOS[0] (i.e Rocksdb + Xrootd + RBD)?
CERN's EOSPPC instance, which is one of
the biggest from what I can
find, was up around 3.5B files in 2019; and you're proposing running 10B
files, so I don't know how EOS will handle that. Maybe Dan can chime in
on the scalability there.
Please do keep on this important work! I've tried to do something
similar at a much smaller scale for Gentoo Linux's historical collection
of source code media (distfiles), but am significantly further behind
your effort.
Thanks for the encouragements! These are very preliminary stages but
I'm enthusiastic about what will follow because I'll have the opportunity to work
on it until a solution is implemented and deployed.
Let say those 10 billions objects are stored in a
single 4+2 erasure
coded pool with bluestore compression set for objects that have a size
32KB and the smallest allocation size for bluestore set to 4KB[3].
The 750TB won't use the expected 350TB but about 30% more, i.e.
~450TB (see [4] for the maths). This space amplification is because
storing a 1 byte object uses the same space as storing a 16KB object
(see [5] to repeat the experience at home). In a 4+2 erasure coded
pool, each of the 6 chunks will use no less than 4KB because that's
the smallest allocation size for bluestore. That's 4 * 4KB = 16KB
even when all that is needed is 1 byte.
I think you have an error here: with 4KB
allocation size in 4+2 pool,
any object sized (0,16K] will take _6_ chunks: 20KB of storage.
Any object sized (16K,32K] will take _12_ chunks: 40K of storage.
I'd attack this from another side entirely:
- how aggressively do you want to pack objects overall? e.g. if you have
a few thousand objects in the 4-5K range, do you want zero bytes
wasted between objects?
50% of the objects have a size <4KB, that is
~5billions currently and growing. *But* they account for only 1% of the total size. So
maybe not very agressively but not passively either.
- how aggressively do you want to dudup objects that
share common data,
esp if it's not aligned on some common byte margins?
Objects/files are
addressed by the SHA256 of their content and that takes care of deduplication.
- what are the data portability requirements to
move/extract data from
this system at a later point?
The data portability is ensured by using Free
Software only and open standards where possible. And by distributing the software in a way
that can be conveniently installed by a third party. Does that answer your question? The
durability of the software/format couple used to store data is something I'm not
worried about but may I should.
- how complex of an index are you willing to maintain
to
reconstruct/access data?
I don't envision the index being more complex than
SHA256 => content (roughly).
- What requirements are there about the ordering and
accessibility of
the packs? How related do the pack objects need to be? e.g. are the
packed as they arrive in time order, to build up successive packs of
size, or are there many packs and you append the "correct" pack for a
given object?
There are no ordering requirements.
I'm normally distinctly in the camp that object storage systems should
natively expose all objects, but that also doesn't account for your
immutability/append-only nature.
I see your discussion at https://forge.softwareheritage.org/T3054#58977
as well, about the "full scale out" vs "scale up metadata & scale out
data" parts.
To brainstorm parts of an idea, I'm wondering about Git's
still-in-development partial clone work,
[snip] I did not know about "partial
clone" and will explore this in https://forge.softwareheritage.org/T3065. Although it
is probably not a good fit for a 2021 solution, it sounds like a great source of
inspiration.
Thinking back to older systems, like SGI's
hierarchal storage modules
for XFS, the packing overhead starts to become significant for your
objects: some of the underlying mechanisms in the XFS HSM DMAPI, if they
ended up packing immutable objects to tape still had tar & tar-like
headers (at least 512 bytes per object), your 10B objects would take at
least 4TB of extra space (before compression).
I'm tempted to overlook lessons
from the past. In part because I'm afraid I'll loose myself :-) In part because I
assume the world changed a lot since. If however you think (have a hunch) that it might be
useful, I'll give it a try.
Thanks for the great feedback!
--
Loïc Dachary, Artisan Logiciel Libre
21 Feb
21 Feb
5:09 p.m.
Bonjour,
For the record, here is a summary of the key takeaways from this conversation (so far):
* Ambry[0] is a perfect match and I'll keep exploring it[1].
* To keep billions of small objects manageable, they must be packed together.
* Immutable & never deleted objects can be grouped together for the purpose of packing
them without a central database. For this to work the id of the group to which an object
belongs is included in the object ID (e.g. SHA256 + UUID of the group). That's what
Ambry does.
To be continued.
[0] https://github.com/linkedin/ambry/wiki
[1] https://forge.softwareheritage.org/T3064
On 17/02/2021 17:36, Loïc Dachary wrote:
Bonjour,
TL;DR: Is it more advisable to work on Ceph internals to make it friendly to this
particular workload or write something similar to EOS[0] (i.e Rocksdb + Xrootd + RBD)?
This is a followup of two previous mails[1] sent while researching this topic. In a
nutshell, the Software Heritage project[1] currently has ~750TB and 10 billions objects,
75% of which have a size smaller than 16KB and 50% have a size smaller than 4KB. But they
only account for ~5% of the 750TB: 25% of the objects have a size > 16KB and total
~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of actual
storage. (if you're interested in the details see [2]).
Let say those 10 billions objects are stored in a single 4+2 erasure coded pool with
bluestore compression set for objects that have a size > 32KB and the smallest
allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected 350TB
but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is
because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to
repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use
no less than 4KB because that's the smallest allocation size for bluestore. That's
4 * 4KB = 16KB even when all that is needed is 1 byte.
It was suggested[6] to have two different pools: one with a 4+2 erasure pool and
compression for all objects with a size > 32KB that are expected to compress to 16KB.
And another with 3 replicas for the smaller objects to reduce space amplification to a
minimum without compromising on durability. A client looking for the object could make two
simultaneous requests to the two pools. They would get 404 from one of them and the object
from the other.
Another workaround, is best described in the "Finding a needle in Haystack:
Facebook’s photo storage"[9] paper and essentially boils down to using a database to
store a map between the object name and its location. That does not scale out (writing the
database index is the bottleneck) but it's simple enough and is successfully
implemented in EOS[0] with >200PB worth of data and in seaweedfs[10], another promising
object store software based on the same idea.
Instead of working around the problem, maybe Ceph could be modified to make better use of
the immutability of these objects[7], a hint that is apparently only used to figure out
how to best compress it and for checksum calculation[8]. I honestly have not clue how
difficult it would be. All I know is that it's not easy otherwise it would have been
done already: there seem to be a general need for efficiently (space wise and performance
wise) storing large quantities of objects smaller than 4KB.
Is it more advisable to:
* work on Ceph internals to make it friendly to this particular workload or,
* write another implementation of "Finding a needle in Haystack: Facebook’s photo
storage"[9] based on RBD[11]?
I'm currently leaning toward working on Ceph internals but there are pros and cons to
both approaches[12]. And since all this is still very new to me, there also is the
possibility that I'm missing something. Maybe it's *super* difficult to improve
Ceph in this way. I should try to figure that out sooner rather than later.
I realize it's a lot to take in and unless you're facing the exact same problem
there is very little chance you read that far :-) But if you did... I'm *really*
interested to hear what yout think. In any case I'll report back to this thread once a
decision has been made.
Cheers
[0] https://eos-web.web.cern.ch/eos-web/
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF…
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX…
[2] https://forge.softwareheritage.org/T3054
[3]
https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/…
[4] https://forge.softwareheritage.org/T3052#58864
[5] https://forge.softwareheritage.org/T3052#58917
[6] https://forge.softwareheritage.org/T3052#58876
[7]
https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN…
[8] https://forge.softwareheritage.org/T3055
[9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
[10] https://github.com/chrislusf/seaweedfs/wiki/Components
[11] https://forge.softwareheritage.org/T3049
[12] https://forge.softwareheritage.org/T3054#58977
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Loïc Dachary, Artisan Logiciel Libre
22 Feb
22 Feb
7:41 p.m.
Hi,
On Sunday, February 21st, 2021 at 12:39, Loïc Dachary <loic(a)dachary.org> wrote:
For the record, here is a summary of the key takeaways
from this conversation (so far):
- Ambry[0] is a perfect match and I'll keep exploring it[1].
- To keep billions of small objects manageable, they must be packed together.
- Immutable & never deleted objects can be grouped together for the purpose of
packing them without a central database. For this to work the id of the group to which an
object belongs is included in the object ID (e.g. SHA256 + UUID of the group). That's
what Ambry does.
What about using OMAP for the purpose of "grouping together" small objects? You
would store tiny objects as OMAP key/value pairs on a Ceph object. Internally, they're
stored in RocksDB so the min allocation size isn't an issue, and retrieving individual
objects should be fairly quick.
Has anyone tried this? How does RocksDB cope with terabytes of data and hundreds of
millions of key-value pairs? Is recovery faster for OMAP compared to the equivalent number
of RADOS objects?
Cheers,
--
Ben
24 Feb
24 Feb
11:42 a.m.
OMAP with keys works as database-like replication, new keys/updates comes to acting set as
data stream, not a full object
k
Sent from my iPhone
On 22 Feb 2021, at 17:13, Benoît Knecht
<bknecht(a)protonmail.ch> wrote:
Is recovery faster for OMAP compared to the equivalent number of RADOS objects?
1155
days inactive
1162
days old
14 comments
7 participants
participants (7)
-
Benoît Knecht -
Dan van der Ster -
Konstantin Shalygin -
Loïc Dachary -
Nathan Fish -
Robin H. Johnson -
Serkan Çoban