RGW: rgw_qactive perf is constantly growing
Hi On two of our clusters (all v14.2.8) we observe a very strange behavior: Over a time rgw_qactive perf is constantly growing, within 12h to 6k entries. [image: image.png] We observe this situation only on two of our clusters where the common thing is an app uploading a lot of files as multipart uploads via ssl. How can we debug this situation? How can we check what operations are in a queue or why a perf counter has not been decreased? Jacek -- Jacek Suchenia jacek.suchenia@gmail.com
Jacek Suchenia writes:
On two of our clusters (all v14.2.8) we observe a very strange behavior:
Over a time rgw_qactive perf is constantly growing, within 12h to 6k entries.
As another data point, we're seeing the same here, on one of our two clusters, both also running 14.2.8. The growth is a bit slower here, about 300-700 connections per 24h across 6 RadosGW instances, but it's quite obvious. Our other cluster doesn't show this behavior, even though it is bigger and presumably has higher load.
image.png We observe this situation only on two of our clusters where the common thing is an app uploading a lot of files as multipart uploads via ssl.
Interesting. Our two clusters seem to have similar rates of multipart uploads, yet only one of them has the issue.
How can we debug this situation? How can we check what operations are in a queue or why a perf counter has not been decreased?
I'd be curious about that as well. Maybe it's just an accounting issue with some kinds of (failed/aborted?) requests. Looks a bit fishy... -- Simon.
I upgraded one cluster to 14.2.10 and this perf counter is still growing. Does any have an idea of how to debug this problem? Jacek sob., 4 lip 2020 o 18:49 Simon Leinen <simon.leinen@switch.ch> napisał(a):
Jacek Suchenia writes:
On two of our clusters (all v14.2.8) we observe a very strange behavior:
Over a time rgw_qactive perf is constantly growing, within 12h to 6k entries.
As another data point, we're seeing the same here, on one of our two clusters, both also running 14.2.8.
The growth is a bit slower here, about 300-700 connections per 24h across 6 RadosGW instances, but it's quite obvious.
Our other cluster doesn't show this behavior, even though it is bigger and presumably has higher load.
image.png We observe this situation only on two of our clusters where the common thing is an app uploading a lot of files as multipart uploads via ssl.
Interesting. Our two clusters seem to have similar rates of multipart uploads, yet only one of them has the issue.
How can we debug this situation? How can we check what operations are in a queue or why a perf counter has not been decreased?
I'd be curious about that as well. Maybe it's just an accounting issue with some kinds of (failed/aborted?) requests. Looks a bit fishy... -- Simon.
-- Jacek Suchenia jacek.suchenia@gmail.com
participants (2)
-
Jacek Suchenia
-
Simon Leinen