Hi, Le 04/02/2021 à 08:41, Loïc Dachary a écrit : If I understand correctly I assume that you are missing a "not" here. Reading the exchanges I believe you are focused on the reading speed and space efficiency. Did you consider the writing speed with such a scheme ? Depending on how you store the index, you could block on each write and would have to consider Ceph latency (ie: if your writer fails recovering can be tricky without having waited for writes to update your index). With your 100TB target and 3kb artifact size a 1ms latency and blocking writes translate to a whole year spent writing. If you manage to get to a 0.1ms latency (not sure if this is achievable with Ceph yet) you end with a month. Depending on how you plan to populate the store this could be a problem. You'll have to consider if the artifact writing rate limit can become a bottleneck during normal use too. You can probably design a scheme supporting storing multiple values in a single write but it seems to add complexity which might come with unwanted performance problems and space use itself. I'm not familiar with space efficiency on modern Ceph versions (still using filestore on Hammer...), do you have a ballpark estimation of the costs of storing artifacts as simple objects ? Unless you already worked out the whole design that would be my first concern : it could end up being an inefficiency worth the trade-off for simplicity. I'm unfamiliar with the gateway and how well and easily it can scale so my first impulse was to bypass RGW to use the librados interface directly. You can definitely begin with a RGW solution as it is a bit easier to implement and switch to librados later if RGW ever becomes a bottleneck. If you need speed either writing or reading, both RGW and librados would work : you can have as many clients managing objects in parallel without any lock on writes on your end to manage. This is a very simple storage design and simplicity can't be overrated :-) The only potential downside (in addition to space inefficiency) that I can see would be walking the list of objects. This is doable but with billions of them this could be very slow. Not sure if it could become a need given your use case though. For reference, I just found the results of a test with a moderately comparable test set : https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond. I didn't finish reading it yet but the volume seems comparable to your use case although with 64kB objects. Note : I've seen questions about 100TB RBDs in the thread. We use such beasts in two clusters : they work fine but are a pain when deleting or downsizing them. During one downsize on the slowest cluster we had to pause the operation manually (SIGSTOP to the rbd process) during periods of high loads and let it continue after. This took about a week (but the cluster was admittedly underpowered for its use at the time). Best regards, -- Lionel Bouton

Hi, On 04/02/2021 07:41, Loïc Dachary wrote: RGW addition is pretty quick up to fairly large buckets; and if you're not expecting to want to list the bucket contents often, then RGW might well be a good option for your object store with small files. Or at least, using some of the RGW code (I think there's a librgw) to re-use a bunch of its code for your use case; this feels more natural to me than using RBD for this. Regards, Matthew [pleased software heritage are still looking at Ceph :) ] -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.