CompleteMultipartUpload takes a long time to finish
Hello, For some time now I’m struggling with the time it takes to CompleteMultipartUpload on one of my rgw clusters. I have a customer with ~8M objects in one bucket uploading quite a large files. From 100GB to like 800GB. I’ve noticed when they are uploading ~200GB files that the requests started timeouting on a LB we have infront of the rgw. When I’ve started going through the logs I’ve noticed that the CompleteMultipartUpload request took like 700s to finish. Which seemed ok-ish, but the number seem quite large. However, when they started uploading 750GB files the time to complete the multipart upload ended around 2500s -> more than 40minutes which seems like a way to much. Do you have a similar experience? Is there anything we can do to improve this? How much time does the CompleteMultipartUpload takes on your clusters? The cluster is running on version 17.2.6. Regards, Ondrej
On Mon, Feb 05, 2024 at 07:51:34PM +0100, Ondřej Kukla wrote:
Hello,
For some time now I’m struggling with the time it takes to CompleteMultipartUpload on one of my rgw clusters.
I have a customer with ~8M objects in one bucket uploading quite a large files. From 100GB to like 800GB.
I’ve noticed when they are uploading ~200GB files that the requests started timeouting on a LB we have infront of the rgw.
When I’ve started going through the logs I’ve noticed that the CompleteMultipartUpload request took like 700s to finish. Which seemed ok-ish, but the number seem quite large.
However, when they started uploading 750GB files the time to complete the multipart upload ended around 2500s -> more than 40minutes which seems like a way to much.
Do you have a similar experience? Is there anything we can do to improve this? How much time does the CompleteMultipartUpload takes on your clusters?
The cluster is running on version 17.2.6. How many incomplete MPUs and MPU parts exist in that bucket?
Is it 8M objects based on ListObjects, or the number of objects reported by radosgw-admin bucket stats? If there are LOTS of incomplete objects, that can cause extreme cases in the listing used by both ListObjects & CompleteMultipartUpload. This is esp. likely if they the huge files used many tiny parts (some tooling defaults to 5MB parts). Easy way to test this is ask them to do a single 200GB upload to a brand new bucket (with no objects). If *that* case is fast, then it's something about the index entries in the existing bucket; likely a high proportion of incomplete parts. CompleteMultipartUpload for 50GB => single-digit seconds. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
I will try to check this case. From a first check I can see that the client has multiple (12) multipart uploads for the same key, started at the exactly same time. But the objects are uploaded to only one “upload id” others are empty so the number of Parts is equal to the number of parts of the running upload. For the part size, client is uploading 500MiB parts so for the 750GiB it’s ~1500 parts. Ondrej
On 6. 2. 2024, at 7:53, Robin H. Johnson <robbat2@gentoo.org> wrote:
On Mon, Feb 05, 2024 at 07:51:34PM +0100, Ondřej Kukla wrote:
Hello,
For some time now I’m struggling with the time it takes to CompleteMultipartUpload on one of my rgw clusters.
I have a customer with ~8M objects in one bucket uploading quite a large files. From 100GB to like 800GB.
I’ve noticed when they are uploading ~200GB files that the requests started timeouting on a LB we have infront of the rgw.
When I’ve started going through the logs I’ve noticed that the CompleteMultipartUpload request took like 700s to finish. Which seemed ok-ish, but the number seem quite large.
However, when they started uploading 750GB files the time to complete the multipart upload ended around 2500s -> more than 40minutes which seems like a way to much.
Do you have a similar experience? Is there anything we can do to improve this? How much time does the CompleteMultipartUpload takes on your clusters?
The cluster is running on version 17.2.6. How many incomplete MPUs and MPU parts exist in that bucket?
Is it 8M objects based on ListObjects, or the number of objects reported by radosgw-admin bucket stats?
If there are LOTS of incomplete objects, that can cause extreme cases in the listing used by both ListObjects & CompleteMultipartUpload.
This is esp. likely if they the huge files used many tiny parts (some tooling defaults to 5MB parts).
Easy way to test this is ask them to do a single 200GB upload to a brand new bucket (with no objects).
If *that* case is fast, then it's something about the index entries in the existing bucket; likely a high proportion of incomplete parts.
CompleteMultipartUpload for 50GB => single-digit seconds.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org <mailto:robbat2@gentoo.org> GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
participants (2)
-
Ondřej Kukla
-
Robin H. Johnson