Delete objects on large bucket very slow
Dear Ceph team, I builde a object storage using ceph, a cluster with 5 nodes (9 sas10k 2TB (for data)+ 1 nvme (for metadata) in one node , using ceph version 14.2.3, nautilus (stable). My cluster storage over 100M (minlion) objects (files) with file's size ~ 50KB, and I want to delete them to free capacity and improve performance. When I delete object, speed delete object very slow, about ~ 35-36 objs/s at 100M (minlion) objects point, i using command: s3cmd del -r s3://mybucket/2019 to delete. Can you help me: - How does ceph's deleting object operator work? (it mean about how delete object flow in ceph works?) - how to improve speed of delete object operator ( objects/s) - how to delete mult - bestpractice for this usecase - other recommend.... Thank you very much. ----------------------------- Br, Dương Tuấn Dũng 0986153686
--max-concurrent-ios helps with deletion speed, there's also a memory leak during deletion which is being fixed here: https://github.com/ceph/ceph/pull/30174 Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Sep 12, 2019 at 6:25 AM tuan dung <dungdt1903@gmail.com> wrote:
Dear Ceph team, I builde a object storage using ceph, a cluster with 5 nodes (9 sas10k 2TB (for data)+ 1 nvme (for metadata) in one node , using ceph version 14.2.3, nautilus (stable). My cluster storage over 100M (minlion) objects (files) with file's size ~ 50KB, and I want to delete them to free capacity and improve performance. When I delete object, speed delete object very slow, about ~ 35-36 objs/s at 100M (minlion) objects point, i using command: s3cmd del -r s3://mybucket/2019 to delete. Can you help me: - How does ceph's deleting object operator work? (it mean about how delete object flow in ceph works?) - how to improve speed of delete object operator ( objects/s) - how to delete mult - bestpractice for this usecase - other recommend.... Thank you very much. ----------------------------- Br, Dương Tuấn Dũng 0986153686 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Paul, thank for your answer. Can you explain clearly for me about: - How does ceph's deleting object operator work? (it mean about how delete object flow in ceph works?) - using s3cmd tool which install in client to delete object, how to improve speed of delete object operator (objects/s) with it, current speed is 32-36 objs/s at point 90M-100M object, using 1 client, running 1 process delete ( s3cmd del -r s3://mybucket/2019) thank you very much ----------------------------- Br, Dương Tuấn Dũng 0986153686 On Thu, Sep 12, 2019 at 7:46 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
--max-concurrent-ios helps with deletion speed, there's also a memory leak during deletion which is being fixed here: https://github.com/ceph/ceph/pull/30174
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
On Thu, Sep 12, 2019 at 6:25 AM tuan dung <dungdt1903@gmail.com> wrote:
Dear Ceph team, I builde a object storage using ceph, a cluster with 5 nodes (9 sas10k
2TB (for data)+ 1 nvme (for metadata) in one node , using ceph version 14.2.3, nautilus (stable). My cluster storage over 100M (minlion) objects (files) with file's size ~ 50KB, and I want to delete them to free capacity and improve performance. When I delete object, speed delete object very slow, about ~ 35-36 objs/s at 100M (minlion) objects point, i using command: s3cmd del -r s3://mybucket/2019 to delete. Can you help me:
- How does ceph's deleting object operator work? (it mean about how delete object flow in ceph works?) - how to improve speed of delete object operator ( objects/s) - how to delete mult - bestpractice for this usecase - other recommend.... Thank you very much. ----------------------------- Br, Dương Tuấn Dũng 0986153686 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 9/13/19 10:46 AM, tuan dung wrote:
thank for your answer. Can you explain clearly for me about: - How does ceph's deleting object operator work? (it mean about how delete object flow in ceph works?) - using s3cmd tool which install in client to delete object, how to improve speed of delete object operator (objects/s) with it, current speed is 32-36 objs/s at point 90M-100M object, using 1 client, running 1 process delete ( s3cmd del -r s3://mybucket/2019)
This is option for radosgw-admin tool. k
Anyone help me explain: How does ceph delete an object? how flow of operator detete work? ----------------------------- Br, Dương Tuấn Dũng 0986153686 On Fri, Sep 13, 2019 at 10:51 AM Konstantin Shalygin <k0ste@k0ste.ru> wrote:
On 9/13/19 10:46 AM, tuan dung wrote:
thank for your answer. Can you explain clearly for me about: - How does ceph's deleting object operator work? (it mean about how delete object flow in ceph works?) - using s3cmd tool which install in client to delete object, how to improve speed of delete object operator (objects/s) with it, current speed is 32-36 objs/s at point 90M-100M object, using 1 client, running 1 process delete ( s3cmd del -r s3://mybucket/2019)
This is option for radosgw-admin tool.
k
participants (3)
-
Konstantin Shalygin
-
Paul Emmerich
-
tuan dung