RadosGW latency on chuked uploads
Hello, I have strange issues with radosgw: When trying to PUT object with “transfer-encoding: chunked”, I can see high request latencies. When trying to PUT the same object as non-chunked – latency is much lower, and also request/s performance is better. Perhaps anyone had the same issue?
On Tue, Jun 09, 2020 at 12:59:10PM +0300, Tadas wrote:
Hello,
I have strange issues with radosgw: When trying to PUT object with “transfer-encoding: chunked”, I can see high request latencies. When trying to PUT the same object as non-chunked – latency is much lower, and also request/s performance is better. Perhaps anyone had the same issue? What is your latency to the RGW?
There's one downside to chunked encoding that I observed with CivetWeb when I implemented chunked transfer encoding for the Bucket Listing. Specifically, CivetWeb did not stuff the socket with all available content, and instead only trickled out entries, waiting for each TCP window ACK before the next segment was sent. If the Bucket Listing took a long time to complete within RGW, the time to the first results was hugely improved, but the time for the full response MAY be worse if the latency was high, due to having more back & forth in TCP ACKs. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Hello, we face like 75-100 ms while doing 600 chunked PUT's. while 40-45ms while doing 1k normal PUT's. (Even amount of PUT's lowers on chunked PUT way) We tried civetweb and beast. Nothing changes. -----Original Message----- From: Robin H. Johnson Sent: Tuesday, June 9, 2020 8:51 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: RadosGW latency on chuked uploads _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Jun 09, 2020 at 09:07:49PM +0300, Tadas wrote:
Hello, we face like 75-100 ms while doing 600 chunked PUT's. while 40-45ms while doing 1k normal PUT's. (Even amount of PUT's lowers on chunked PUT way)
We tried civetweb and beast. Nothing changes. How close is your test running to the RGWs?
Does it get noticeably worse if you inject artificial latency into the network? E.g. https://bencane.com/2012/07/16/tc-adding-simulated-network-latency-to-your-l... If you can run the test without SSL, then tcpdump should let you see if your client is trying to stuff the max amount of data into the pipeline or waiting for an ACK each time. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Tests are running in the same machine radosgw is located. All data goes directly to radosgw via localhost. No ssl is used. -----Original Message----- From: Robin H. Johnson Sent: Tuesday, June 9, 2020 9:38 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: RadosGW latency on chuked uploads _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Robin H. Johnson
-
Tadas