Workload that delete 100 M object daily via lifecycle
Hi Experts, We plan to setup a Ceph Object to support a S3 workload, that will need to delete 100M file daily via lifecycle. Appreciate your suggestion and setting to handle this kind of scenario Best Regards, Ha
Hi, You may want to check out this doc: https://docs.ceph.com/en/quincy/radosgw/config-ref/#lifecycle-settings As I understand it in short: - if there are thousands of buckets, we should increase the rgw_lc_max_worker. - if there are a few buckets that have hundreds of thousands of objects, we should decrease the rgw_lc_max_wp_worker.
On Mon, 17 Jul 2023 19:19:34 +0700, Ha Nguyen Van <hanv@nttsolution.com> said:
[...] S3 workload, that will need to delete 100M file daily [...]
So many people seem to think that distributed (or even local) filesystems (and in particular their metadata servers) can sustain the same workload as high volume transactional DBMSes. PS 100m deletions per day given 86,400 second per day means on average (what about peaks?) around 1,200 committed deletions per second (across the traditional 3 metadata OSDs) sustained on the metadata server, that may not leave a lot of time for file creation, writing or reading. :-)
Index pool distributed over a large number of NVMe OSDs? Multiple, dedicated RGW instances that only run LC?
On Jul 18, 2023, at 12:08, Peter Grandi <pg@ceph.list.sabi.co.UK> wrote:
On Mon, 17 Jul 2023 19:19:34 +0700, Ha Nguyen Van <hanv@nttsolution.com> said:
[...] S3 workload, that will need to delete 100M file daily [...]
So many people seem to think that distributed (or even local) filesystems (and in particular their metadata servers) can sustain the same workload as high volume transactional DBMSes.
PS 100m deletions per day given 86,400 second per day means on average (what about peaks?) around 1,200 committed deletions per second (across the traditional 3 metadata OSDs) sustained on the metadata server, that may not leave a lot of time for file creation, writing or reading. :-) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[...] S3 workload, that will need to delete 100M file daily [...]
[...] average (what about peaks?) around 1,200 committed deletions per second (across the traditional 3 metadata OSDs) sustained, that may not leave a lot of time for file creation, writing or reading. :-)[...]
[...] So many people seem to think that distributed (or even local) filesystems (and in particular their metadata servers) can sustain the same workload as high volume transactional DBMSes. [...]
Index pool distributed over a large number of NVMe OSDs? Multiple, dedicated RGW instances that only run LC?
As long as that guarantees a total maximum network+write latency of well below 800µs across all of them that might result in a committed rate of a deletion every 800µs (and there are no peaks and the metadata server only does deletions and does not do creations or opens or any "maintenance" operations like checks and backups). :-) Sometimes I suggest somewhat seriously entirely RAM based metadata OSDs, which given a suitable environment may be feasible. But I still wonder why "So many people seem to think ... can sustain the same workload as high volume transactional DBMSes" :-).
Index pool on Aerospike? Building OSDs on PRAM might be a lot less work than trying to ensure consistency on backing storage while still servicing out of RAM and not syncing every transaction.
On Jul 18, 2023, at 14:31, Peter Grandi <pg@ceph.list.sabi.co.UK> wrote:
[...] S3 workload, that will need to delete 100M file daily [...]
[...] average (what about peaks?) around 1,200 committed deletions per second (across the traditional 3 metadata OSDs) sustained, that may not leave a lot of time for file creation, writing or reading. :-)[...]
[...] So many people seem to think that distributed (or even local) filesystems (and in particular their metadata servers) can sustain the same workload as high volume transactional DBMSes. [...]
Index pool distributed over a large number of NVMe OSDs? Multiple, dedicated RGW instances that only run LC?
As long as that guarantees a total maximum network+write latency of well below 800µs across all of them that might result in a committed rate of a deletion every 800µs (and there are no peaks and the metadata server only does deletions and does not do creations or opens or any "maintenance" operations like checks and backups). :-)
Sometimes I suggest somewhat seriously entirely RAM based metadata OSDs, which given a suitable environment may be feasible. But I still wonder why "So many people seem to think ... can sustain the same workload as high volume transactional DBMSes" :-). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Ha Nguyen Van
-
Huy Nguyen
-
pg＠ceph.list.sabi.co.UK