RGW seems to not clean up after some requests
Hi everyone We have faced some RGW outages recently, with the RGW returning HTTP 503. First for a few, then for most, then all requests - in the course of 1-2 hours. This seems to have started since we have updated from 15.2.4 to 15.2.5. The line that accompanies these outages in the log is the following: s3:list_bucket Scheduling request failed with -2218 It first pops up a few times here and there, until it eventually applies to all requests. It seems to indicate that the throttler has reached the limit of open connections. As we run a pair of HAProxy instances in front of RGW, which limit the number of connections to the two RGW instances to 400, this limit should never be reached. We do use RGW metadata sync between the instances, which could account for some extra connections, but if I look at open TCP connections between the instances I can count no more than 20 at any given time. I also noticed that some connections in the RGW log seem to never complete. That is, I can find a ‘starting new request’ line, but no associated ‘req done’ or ‘beast’ line. I don’t think there are any hung connections around, as they are killed by HAProxy after a short timeout. Looking at the code, it seems as if the throttler in use (SimpleThrottler), eventually reaches the maximum count of 1024 connections (outstanding_requests), and never recovers. I believe that the request_complete function is not called in all cases, but I am not familiar with the Ceph codebase, so I am not sure. See https://github.com/ceph/ceph/blob/cc17681b478594aa39dd80437256a54e388432f0/s... <https://github.com/ceph/ceph/blob/cc17681b478594aa39dd80437256a54e388432f0/src/rgw/rgw_dmclock_async_scheduler.h#L166-L214> Does anyone see the same phenomenon? Could this be a bug in the request handling of RGW, or am I wrong in my assumptions? For now we’re just restarting our RGWs regularly, which seems to keep the problem at bay. Thanks for any hints. Denis
Denis Krienbühl <denis@href.ch> writes:
Hi everyone
We have faced some RGW outages recently, with the RGW returning HTTP 503. First for a few, then for most, then all requests - in the course of 1-2 hours. This seems to have started since we have updated from 15.2.4 to 15.2.5.
The line that accompanies these outages in the log is the following:
s3:list_bucket Scheduling request failed with -2218 There isn't much in terms of code changes in the scheduler from v15.2.4->5. Does the perf dump (`ceph daemon perf dump <client.rgw-name> `) on RGW socket show any throttle counts?
It first pops up a few times here and there, until it eventually applies to all requests. It seems to indicate that the throttler has reached the limit of open connections.
As we run a pair of HAProxy instances in front of RGW, which limit the number of connections to the two RGW instances to 400, this limit should never be reached. We do use RGW metadata sync between the instances, which could account for some extra connections, but if I look at open TCP connections between the instances I can count no more than 20 at any given time.
I also noticed that some connections in the RGW log seem to never complete. That is, I can find a ‘starting new request’ line, but no associated ‘req done’ or ‘beast’ line.
I don’t think there are any hung connections around, as they are killed by HAProxy after a short timeout.
Looking at the code, it seems as if the throttler in use (SimpleThrottler), eventually reaches the maximum count of 1024 connections (outstanding_requests), and never recovers. I believe that the request_complete function is not called in all cases, but I am not familiar with the Ceph codebase, so I am not sure.
See https://github.com/ceph/ceph/blob/cc17681b478594aa39dd80437256a54e388432f0/s... <https://github.com/ceph/ceph/blob/cc17681b478594aa39dd80437256a54e388432f0/src/rgw/rgw_dmclock_async_scheduler.h#L166-L214>
Does anyone see the same phenomenon? Could this be a bug in the request handling of RGW, or am I wrong in my assumptions?
For now we’re just restarting our RGWs regularly, which seems to keep the problem at bay.
Thanks for any hints.
Denis _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Abhishek
Hi Abhishek
On 2 Nov 2020, at 14:54, Abhishek Lekshmanan <abhishek@suse.com> wrote:
There isn't much in terms of code changes in the scheduler from v15.2.4->5. Does the perf dump (`ceph daemon perf dump <client.rgw-name> `) on RGW socket show any throttle counts?
I know, I was wondering if this somehow might have an influence, but I’m likely wrong: https://github.com/ceph/ceph/commit/c43f71056322e1a149a444735bf65d80fec7a7ae <https://github.com/ceph/ceph/commit/c43f71056322e1a149a444735bf65d80fec7a7ae> As for the perf counters, I don’t see anything interesting. I dumped the current state, but I don’t know how interesting this is: https://gist.github.com/href/a42c30e001789f005e9aa748f6f858fc <https://gist.github.com/href/a42c30e001789f005e9aa748f6f858fc> At the moment we don’t see any errors, but I do already count 135 incomplete requests in the current log (out of 3 Million). This number is typical for most days, where we’ll see something like 150 such requests. Our working theory is that out of the 1024 maximum outstanding requests of the throttler, ~150 get lost every day to those incomplete requests, until our need for up to 400 requests per instance can no longer be met (first a few will be over the watermark, then more, then all). For those incomplete requests we know that the following line is executed, producing “starting new request”: https://github.com/ceph/ceph/blob/8f393c0fc1886a369d213d5e5791c10cb1591828/s... <https://github.com/ceph/ceph/blob/8f393c0fc1886a369d213d5e5791c10cb1591828/src/rgw/rgw_process.cc#L187> However, it never reaches “req done” in the same function: https://github.com/ceph/ceph/blob/master/src/rgw/rgw_process.cc#L350 <https://github.com/ceph/ceph/blob/master/src/rgw/rgw_process.cc#L350> That entry, and the “beast” entry is missing for those few requests. Cheers, Denis
participants (2)
-
Abhishek Lekshmanan
-
Denis Krienbühl