[ceph-users] Re: A practical approach to efficiently store 100 billions small objects in Ceph

-----Original Message----- From: Loïc Dachary &lt;loic(a)dachary.org&gt; Sent: 10 March 2021 15:54 To: Ceph Users &lt;ceph-users(a)ceph.io&gt; Cc: swh-devel(a)inria.fr Subject: [ceph-users] A practical approach to efficiently store 100 billions small objects in Ceph Bonjour, In the past weeks a few mailing list threads[0][1][2] explored the problem of storing billions of small objects in Ceph. There was great feedback (I learned at lot) and it turns out the solution is a rather simple aggregation of the ideas that were suggested during these discussions. It is described in detail here:    https://wiki.softwareheritage.org/wiki/A_practical_approach_to_effici ently_store_100_billions_small_objects_in_Ceph The next step will be to write and run benchmarks[3]. Although I'm convinced it is a good solution, I've been wrong before and confirmation is required :-) Many thanks to all the participants in the discussions on behalf of the Software Heritage project[4]. Cheers [0] Storing 20 billions of immutable objects in Ceph, 75% <16KB https://lists.ceph.io/hyperkitty/list/ceph- users(a)ceph.io/thread/JSG2TXKNXPXEKZOJZGYF2ZPTQHOB4LHJ/ [1] Small RGW objects and RADOS 64KB minimun size https://lists.ceph.io/hyperkitty/list/ceph- users(a)ceph.io/thread/AEMW6O7WVJFMUIX7QGI2KM7HKDSTNIYT/ [2] Using RBD to pack billions of small files https://lists.ceph.io/hyperkitty/list/ceph- users(a)ceph.io/thread/RHQ5ZCHJISXIXOJSH3TU7DLYVYHRGTAT/ [3] https://forge.softwareheritage.org/T3054 [4] https://www.softwareheritage.org/ -- Loïc Dachary, Artisan Logiciel Libre