We set up a new Nautilus cluster and only have RGW on it. While we had a job doing 200k IOPs of really small objects, I noticed that HAProxy was kicking out RGW backends because they were taking more than 2 seconds to return. We GET a large ~4GB file each minute and use that as a health check to determine if the system is taking too long to service requests. It seems that other IO is being blocked by this large transfer. This seems to be the case with both civetweb and beast. But I'm double checking beast at the moment because I'm not 100% sure we were using it at the start. Any ideas how to mitigate this? It seems that IOs are being scheduled on a thread and if they get unlucky enough to be scheduled behind a big IO, they are just stuck, in this case HAProxy could kick out the backend before the IO is returned and it has to re-request it. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB Sometimes "ms async op threads" and "ms async max op threads" might help as well (we adjust them by default, but for other reasons) Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 9:54 PM Robert LeBlanc <robert@leblancnet.us> wrote:
We set up a new Nautilus cluster and only have RGW on it. While we had a job doing 200k IOPs of really small objects, I noticed that HAProxy was kicking out RGW backends because they were taking more than 2 seconds to return. We GET a large ~4GB file each minute and use that as a health check to determine if the system is taking too long to service requests. It seems that other IO is being blocked by this large transfer. This seems to be the case with both civetweb and beast. But I'm double checking beast at the moment because I'm not 100% sure we were using it at the start.
Any ideas how to mitigate this? It seems that IOs are being scheduled on a thread and if they get unlucky enough to be scheduled behind a big IO, they are just stuck, in this case HAProxy could kick out the backend before the IO is returned and it has to re-request it.
Thank you, Robert LeBlanc
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB
Sometimes "ms async op threads" and "ms async max op threads" might help as well (we adjust them by default, but for other reasons)
Thanks, I'll look into these options and see if they help. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB
On Nautilus the defaults already seem to be: objecter_inflight_op_bytes 104857600 default objecter_inflight_ops 24576 default
Sometimes "ms async op threads" and "ms async max op threads" might help as well (we adjust them by default, but for other reasons)
Thanks, I'll look into these options and see if they help.
Is it possible that I may be running into something like this? https://github.com/boostorg/beast/issues/1077 ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Wed, Oct 16, 2019 at 11:23 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB
On Nautilus the defaults already seem to be: objecter_inflight_op_bytes 104857600 default
= 100MiB
objecter_inflight_ops 24576 default
not sure where you got this from, but the default is still 1024 even in master: https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s... Paul
Sometimes "ms async op threads" and "ms async max op threads" might help as well (we adjust them by default, but for other reasons)
Thanks, I'll look into these options and see if they help.
Is it possible that I may be running into something like this?
https://github.com/boostorg/beast/issues/1077
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Wed, Oct 16, 2019 at 2:50 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Wed, Oct 16, 2019 at 11:23 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB
On Nautilus the defaults already seem to be: objecter_inflight_op_bytes 104857600 default
= 100MiB
objecter_inflight_ops 24576 default
not sure where you got this from, but the default is still 1024 even in master: https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
Looks like it is overridden in https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s... I got the value through `ceph config show-with-defaults rgw.<name>.rgw0` It's kind of reminiscent of buffer bloat where one big transfer just blocks all the other ones. It just feels like something blocking in the code like an awiat not passing control back properly. I'm just not understanding how your suggestions would help, the problem doesn't seem to be on the RADOS side (which it appears your tweaks target), but on the HTTP side as an HTTP health check takes a long time to come back when a big transfer is going on. Granted, we are really trying to get latency down in our environment because we do fast failover. If RGW doesn't have the object, then abort and check AWS so Time to First Byte is critical. It lets us know if the object is there or not quickly and any request being blocked by a long transfer will execute the fail path in our code. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Thu, Oct 17, 2019 at 12:17 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Wed, Oct 16, 2019 at 2:50 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Wed, Oct 16, 2019 at 11:23 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB
On Nautilus the defaults already seem to be: objecter_inflight_op_bytes 104857600 default
= 100MiB
objecter_inflight_ops 24576 default
not sure where you got this from, but the default is still 1024 even in master: https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
Looks like it is overridden in https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
you are right, this is new in Nautilus. Last time I had to play around with these settings was indeed on a Mimic deployment.
I'm just not understanding how your suggestions would help, the problem doesn't seem to be on the RADOS side (which it appears your tweaks target), but on the HTTP side as an HTTP health check takes a long time to come back when a big transfer is going on.
I was guessing a bottleneck on the RADOS side because you mentioned that you tried both civetweb and beast, somewhat unlikely to run into the exact same problem with both -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
On Thu, Oct 17, 2019 at 2:50 AM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Thu, Oct 17, 2019 at 12:17 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Wed, Oct 16, 2019 at 2:50 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Wed, Oct 16, 2019 at 11:23 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB
On Nautilus the defaults already seem to be: objecter_inflight_op_bytes 104857600 default
= 100MiB
objecter_inflight_ops 24576 default
not sure where you got this from, but the default is still 1024 even in master: https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
Looks like it is overridden in https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
you are right, this is new in Nautilus. Last time I had to play around with these settings was indeed on a Mimic deployment.
I'm just not understanding how your suggestions would help, the problem doesn't seem to be on the RADOS side (which it appears your tweaks target), but on the HTTP side as an HTTP health check takes a long time to come back when a big transfer is going on.
I was guessing a bottleneck on the RADOS side because you mentioned that you tried both civetweb and beast, somewhat unlikely to run into the exact same problem with both
Looping in ceph-dev in case they have some insights into the inner workings that may be helpful. From what I understand civitweb was not async and beast is, but if beast is not coded exactly right, then it could behave similarly as civitweb. It seems that with beast incoming requests are being assigned to BEAST threads and possibly it is doing as sync call to rados therefore blocking requests behind it until the RADOS call is completed. I tried looking through the code, but I'm not familiar with async in C++. I could see two options that may resolve this. First, have a seperate thread pool for accessing RADOS objects with a queue that BEAST dispatches to and callback the completion at the end. The second option is creating async RADOS calls so that it can yield the event loop to another RADOS task. I couldn't tell if either one of these are being done, but that should help small IO not get stuck behind large IO. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On 10/17/19 10:58 AM, Robert LeBlanc wrote:
On Thu, Oct 17, 2019 at 2:50 AM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Thu, Oct 17, 2019 at 12:17 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Wed, Oct 16, 2019 at 2:50 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Wed, Oct 16, 2019 at 11:23 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote: > Could the 4 GB GET limit saturate the connection from rgw to Ceph? > Simple to test: just rate-limit the health check GET I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
> Did you increase "objecter inflight ops" and "objecter inflight op bytes"? > You absolutely should adjust these settings for large RGW setups, > defaults of 1024 and 100 MB are way too low for many RGW setups, we > default to 8192 and 800MB On Nautilus the defaults already seem to be: objecter_inflight_op_bytes 104857600 default = 100MiB
objecter_inflight_ops 24576 default not sure where you got this from, but the default is still 1024 even in master: https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s... Looks like it is overridden in https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s... you are right, this is new in Nautilus. Last time I had to play around with these settings was indeed on a Mimic deployment.
I'm just not understanding how your suggestions would help, the problem doesn't seem to be on the RADOS side (which it appears your tweaks target), but on the HTTP side as an HTTP health check takes a long time to come back when a big transfer is going on. I was guessing a bottleneck on the RADOS side because you mentioned that you tried both civetweb and beast, somewhat unlikely to run into the exact same problem with both Looping in ceph-dev in case they have some insights into the inner workings that may be helpful.
From what I understand civitweb was not async and beast is, but if beast is not coded exactly right, then it could behave similarly as civitweb.
With respect to this issue, civetweb and beast should behave the same. Both frontends have a large thread pool, and their calls to process_request() run synchronously (including blocking on rados requests) on a frontend thread. So once there are more concurrent client connections than there are frontend threads, new connections will block until there's a thread available to service them.
It seems that with beast incoming requests are being assigned to BEAST threads and possibly it is doing as sync call to rados therefore blocking requests behind it until the RADOS call is completed. I tried looking through the code, but I'm not familiar with async in C++. I could see two options that may resolve this. First, have a seperate thread pool for accessing RADOS objects with a queue that BEAST dispatches to and callback the completion at the end. The second option is creating async RADOS calls so that it can yield the event loop to another RADOS task. I couldn't tell if either one of these are being done, but that should help small IO not get stuck behind large IO.
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Thu, Oct 17, 2019 at 9:22 AM Casey Bodley <cbodley@redhat.com> wrote:
With respect to this issue, civetweb and beast should behave the same. Both frontends have a large thread pool, and their calls to process_request() run synchronously (including blocking on rados requests) on a frontend thread. So once there are more concurrent client connections than there are frontend threads, new connections will block until there's a thread available to service them.
Okay, this really helps me understand what's going on here. Is there plans to remove the synchronous calls and make them async or improve this flow a bit? Currently I'm seeing 1024 max concurrent ops and 512 thread pool. Does this mean that on an equally distributed requests that one op could be processing on the backend RADOS with another queued behind it waiting? Is this done in round robin fashion so for 99% small io, a very long RADOS request can get many IO blocked behind it because it is being round-robin dispatched to the thread pool? (I assume the latter is what I'm seeing). rgw_max_concurrent_requests 1024 rgw_thread_pool_size 512 If I match the two, do you think it would help prevent small IO from being blocked by larger IO? I'm also happy to look into the code to suggest improvements if you can give me some quick points into the code to start will help. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On 10/17/19 12:59 PM, Robert LeBlanc wrote:
On Thu, Oct 17, 2019 at 9:22 AM Casey Bodley <cbodley@redhat.com> wrote:
With respect to this issue, civetweb and beast should behave the same. Both frontends have a large thread pool, and their calls to process_request() run synchronously (including blocking on rados requests) on a frontend thread. So once there are more concurrent client connections than there are frontend threads, new connections will block until there's a thread available to service them. Okay, this really helps me understand what's going on here. Is there plans to remove the synchronous calls and make them async or improve this flow a bit?
Absolutely yes, this work has been in progress for a long time now, and octopus does get a lot of concurrency here. Eventually, all of process_request() will be async-enabled, and we'll be able to run beast with a much smaller thread pool.
Currently I'm seeing 1024 max concurrent ops and 512 thread pool. Does this mean that on an equally distributed requests that one op could be processing on the backend RADOS with another queued behind it waiting? Is this done in round robin fashion so for 99% small io, a very long RADOS request can get many IO blocked behind it because it is being round-robin dispatched to the thread pool? (I assume the latter is what I'm seeing).
rgw_max_concurrent_requests 1024 rgw_thread_pool_size 512
If I match the two, do you think it would help prevent small IO from being blocked by larger IO?
rgw_max_concurrent_requests was added in support of the beast/async work, precisely because (post-Nautilus) the number of beast threads will no longer limit the number of concurrent requests. This variable is what throttles incoming requests to prevent radosgw's resource consumption from ballooning under heavy workload. And unlike the existing model where a request remains in the queue until a thread is ready to service it, any requests that exceed rgw_max_concurrent_requests will be rejected with '503 SlowDown' in s3 or '498 Rate Limited' in swift. With respect to prioritization, there isn't any by default but we do have a prototype request scheduler that uses dmclock to prioritize requests based on some hard-coded request classes. It's not especially useful in its current form, but we do have plans to further elaborate the classes and eventually pass the information down to osds for integrated QOS. As of nautilus, though, the thread pool size is the only effective knob you have.
I'm also happy to look into the code to suggest improvements if you can give me some quick points into the code to start will help.
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Thu, Oct 17, 2019 at 11:46 AM Casey Bodley <cbodley@redhat.com> wrote:
On 10/17/19 12:59 PM, Robert LeBlanc wrote:
On Thu, Oct 17, 2019 at 9:22 AM Casey Bodley <cbodley@redhat.com> wrote:
With respect to this issue, civetweb and beast should behave the same. Both frontends have a large thread pool, and their calls to process_request() run synchronously (including blocking on rados requests) on a frontend thread. So once there are more concurrent client connections than there are frontend threads, new connections will block until there's a thread available to service them. Okay, this really helps me understand what's going on here. Is there plans to remove the synchronous calls and make them async or improve this flow a bit?
Absolutely yes, this work has been in progress for a long time now, and octopus does get a lot of concurrency here. Eventually, all of process_request() will be async-enabled, and we'll be able to run beast with a much smaller thread pool.
This is great news. Anything we can do to help in this effort as it is very important for us?
Currently I'm seeing 1024 max concurrent ops and 512 thread pool. Does this mean that on an equally distributed requests that one op could be processing on the backend RADOS with another queued behind it waiting? Is this done in round robin fashion so for 99% small io, a very long RADOS request can get many IO blocked behind it because it is being round-robin dispatched to the thread pool? (I assume the latter is what I'm seeing).
rgw_max_concurrent_requests 1024 rgw_thread_pool_size 512
If I match the two, do you think it would help prevent small IO from being blocked by larger IO? rgw_max_concurrent_requests was added in support of the beast/async work, precisely because (post-Nautilus) the number of beast threads will no longer limit the number of concurrent requests. This variable is what throttles incoming requests to prevent radosgw's resource consumption from ballooning under heavy workload. And unlike the existing model where a request remains in the queue until a thread is ready to service it, any requests that exceed rgw_max_concurrent_requests will be rejected with '503 SlowDown' in s3 or '498 Rate Limited' in swift.
With respect to prioritization, there isn't any by default but we do have a prototype request scheduler that uses dmclock to prioritize requests based on some hard-coded request classes. It's not especially useful in its current form, but we do have plans to further elaborate the classes and eventually pass the information down to osds for integrated QOS.
As of nautilus, though, the thread pool size is the only effective knob you have.
Do you see any problems with running 2k-4k threads if we have the RAM to do so? ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
My impression is that running a second gateway (assuming 1 at present) on the same host would be preferable to running one with very high thread count, also that 1024 is a good maximum value for thread count. Matt On Thu, Oct 17, 2019 at 4:01 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Thu, Oct 17, 2019 at 11:46 AM Casey Bodley <cbodley@redhat.com> wrote:
On 10/17/19 12:59 PM, Robert LeBlanc wrote:
On Thu, Oct 17, 2019 at 9:22 AM Casey Bodley <cbodley@redhat.com> wrote:
With respect to this issue, civetweb and beast should behave the same. Both frontends have a large thread pool, and their calls to process_request() run synchronously (including blocking on rados requests) on a frontend thread. So once there are more concurrent client connections than there are frontend threads, new connections will block until there's a thread available to service them. Okay, this really helps me understand what's going on here. Is there plans to remove the synchronous calls and make them async or improve this flow a bit?
Absolutely yes, this work has been in progress for a long time now, and octopus does get a lot of concurrency here. Eventually, all of process_request() will be async-enabled, and we'll be able to run beast with a much smaller thread pool.
This is great news. Anything we can do to help in this effort as it is very important for us?
Currently I'm seeing 1024 max concurrent ops and 512 thread pool. Does this mean that on an equally distributed requests that one op could be processing on the backend RADOS with another queued behind it waiting? Is this done in round robin fashion so for 99% small io, a very long RADOS request can get many IO blocked behind it because it is being round-robin dispatched to the thread pool? (I assume the latter is what I'm seeing).
rgw_max_concurrent_requests 1024 rgw_thread_pool_size 512
If I match the two, do you think it would help prevent small IO from being blocked by larger IO? rgw_max_concurrent_requests was added in support of the beast/async work, precisely because (post-Nautilus) the number of beast threads will no longer limit the number of concurrent requests. This variable is what throttles incoming requests to prevent radosgw's resource consumption from ballooning under heavy workload. And unlike the existing model where a request remains in the queue until a thread is ready to service it, any requests that exceed rgw_max_concurrent_requests will be rejected with '503 SlowDown' in s3 or '498 Rate Limited' in swift.
With respect to prioritization, there isn't any by default but we do have a prototype request scheduler that uses dmclock to prioritize requests based on some hard-coded request classes. It's not especially useful in its current form, but we do have plans to further elaborate the classes and eventually pass the information down to osds for integrated QOS.
As of nautilus, though, the thread pool size is the only effective knob you have.
Do you see any problems with running 2k-4k threads if we have the RAM to do so?
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
On Thu, Oct 17, 2019 at 1:05 PM Matt Benjamin <mbenjami@redhat.com> wrote:
My impression is that running a second gateway (assuming 1 at present) on the same host would be preferable to running one with very high thread count, also that 1024 is a good maximum value for thread count.
We are running 4 RGW containers per host and have 4 hosts for 16 RGW instances. This cluster was doing 200,000+ IOs per second according to `ceph -s`. We can expect more large objects as the system load is increased, but the vast majority of objects will be really small. I'm almost tempted to have more threads in the thread pool than max concurrent requests so that there is a buffer of idle threads rather than IOs being scheduled behind one another. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On 10/17/19 4:00 PM, Robert LeBlanc wrote:
On Thu, Oct 17, 2019 at 11:46 AM Casey Bodley <cbodley@redhat.com> wrote:
On Thu, Oct 17, 2019 at 9:22 AM Casey Bodley <cbodley@redhat.com> wrote:
With respect to this issue, civetweb and beast should behave the same. Both frontends have a large thread pool, and their calls to process_request() run synchronously (including blocking on rados requests) on a frontend thread. So once there are more concurrent client connections than there are frontend threads, new connections will block until there's a thread available to service them. Okay, this really helps me understand what's going on here. Is there plans to remove the synchronous calls and make them async or improve this flow a bit? Absolutely yes, this work has been in progress for a long time now, and octopus does get a lot of concurrency here. Eventually, all of
On 10/17/19 12:59 PM, Robert LeBlanc wrote: process_request() will be async-enabled, and we'll be able to run beast with a much smaller thread pool.
This is great news. Anything we can do to help in this effort as it is very important for us?
We would love help here. While most of the groundwork is done, so the remaining work is mostly mechanical. To summarize the strategy, the beast frontend spawns a coroutine for each client connection, and that coroutine is represented by a boost::asio::yield_context. We wrap this in an 'optional_yield' struct that gets passed to process_request(). The civetweb frontend always passes an empty object (ie null_yield) so that everything runs synchronously. When making calls into librados, we have a rgw_rados_operate() function that supports this optional_yield argument. If it gets a null_yield, it calls the blocking version of librados::IoCtx::operate(). Otherwise it calls a special librados::async_operate() function which suspends the coroutine until completion instead of blocking the thread. So most of the remaining work is in plumbing this optional_yield variable through all of the code paths under process_request() that call into librados. The rgw_rados_operate() helpers will log a "WARNING: blocking librados call" whenever they block inside of a beast frontend thread, so we can go through the rgw log to identify all of the places that still need a yield context. By iterating on this process, we can eventually remove all of the blocking calls, then set up regression testing to verify that no rgw logs contain that warning. Here's an example pr from Ali that adds the optional_yield to requests for bucket instance info: https://github.com/ceph/ceph/pull/27898. It extends the get_bucket_info() call to take optional_yield, and passes one in where available, using null_yield to mark the synchronous cases where one isn't available.
Currently I'm seeing 1024 max concurrent ops and 512 thread pool. Does this mean that on an equally distributed requests that one op could be processing on the backend RADOS with another queued behind it waiting? Is this done in round robin fashion so for 99% small io, a very long RADOS request can get many IO blocked behind it because it is being round-robin dispatched to the thread pool? (I assume the latter is what I'm seeing).
rgw_max_concurrent_requests 1024 rgw_thread_pool_size 512
If I match the two, do you think it would help prevent small IO from being blocked by larger IO? rgw_max_concurrent_requests was added in support of the beast/async work, precisely because (post-Nautilus) the number of beast threads will no longer limit the number of concurrent requests. This variable is what throttles incoming requests to prevent radosgw's resource consumption from ballooning under heavy workload. And unlike the existing model where a request remains in the queue until a thread is ready to service it, any requests that exceed rgw_max_concurrent_requests will be rejected with '503 SlowDown' in s3 or '498 Rate Limited' in swift.
With respect to prioritization, there isn't any by default but we do have a prototype request scheduler that uses dmclock to prioritize requests based on some hard-coded request classes. It's not especially useful in its current form, but we do have plans to further elaborate the classes and eventually pass the information down to osds for integrated QOS.
As of nautilus, though, the thread pool size is the only effective knob you have. Do you see any problems with running 2k-4k threads if we have the RAM to do so?
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Thu, Oct 17, 2019 at 2:03 PM Casey Bodley <cbodley@redhat.com> wrote:
This is great news. Anything we can do to help in this effort as it is very important for us?
We would love help here. While most of the groundwork is done, so the remaining work is mostly mechanical.
To summarize the strategy, the beast frontend spawns a coroutine for each client connection, and that coroutine is represented by a boost::asio::yield_context. We wrap this in an 'optional_yield' struct that gets passed to process_request(). The civetweb frontend always passes an empty object (ie null_yield) so that everything runs synchronously. When making calls into librados, we have a rgw_rados_operate() function that supports this optional_yield argument. If it gets a null_yield, it calls the blocking version of librados::IoCtx::operate(). Otherwise it calls a special librados::async_operate() function which suspends the coroutine until completion instead of blocking the thread.
So most of the remaining work is in plumbing this optional_yield variable through all of the code paths under process_request() that call into librados. The rgw_rados_operate() helpers will log a "WARNING: blocking librados call" whenever they block inside of a beast frontend thread, so we can go through the rgw log to identify all of the places that still need a yield context. By iterating on this process, we can eventually remove all of the blocking calls, then set up regression testing to verify that no rgw logs contain that warning.
Here's an example pr from Ali that adds the optional_yield to requests for bucket instance info: https://github.com/ceph/ceph/pull/27898. It extends the get_bucket_info() call to take optional_yield, and passes one in where available, using null_yield to mark the synchronous cases where one isn't available.
I'll work to get familiar with the code base and see if I can submit some PRs to help out. Things are a bit crazy, but this is very important to us too. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Thanks very much, Robert. Matt On Thu, Oct 17, 2019 at 5:24 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Thu, Oct 17, 2019 at 2:03 PM Casey Bodley <cbodley@redhat.com> wrote:
This is great news. Anything we can do to help in this effort as it is very important for us?
We would love help here. While most of the groundwork is done, so the remaining work is mostly mechanical.
To summarize the strategy, the beast frontend spawns a coroutine for each client connection, and that coroutine is represented by a boost::asio::yield_context. We wrap this in an 'optional_yield' struct that gets passed to process_request(). The civetweb frontend always passes an empty object (ie null_yield) so that everything runs synchronously. When making calls into librados, we have a rgw_rados_operate() function that supports this optional_yield argument. If it gets a null_yield, it calls the blocking version of librados::IoCtx::operate(). Otherwise it calls a special librados::async_operate() function which suspends the coroutine until completion instead of blocking the thread.
So most of the remaining work is in plumbing this optional_yield variable through all of the code paths under process_request() that call into librados. The rgw_rados_operate() helpers will log a "WARNING: blocking librados call" whenever they block inside of a beast frontend thread, so we can go through the rgw log to identify all of the places that still need a yield context. By iterating on this process, we can eventually remove all of the blocking calls, then set up regression testing to verify that no rgw logs contain that warning.
Here's an example pr from Ali that adds the optional_yield to requests for bucket instance info: https://github.com/ceph/ceph/pull/27898. It extends the get_bucket_info() call to take optional_yield, and passes one in where available, using null_yield to mark the synchronous cases where one isn't available.
I'll work to get familiar with the code base and see if I can submit some PRs to help out. Things are a bit crazy, but this is very important to us too.
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
participants (4)
-
Casey Bodley
-
Matt Benjamin
-
Paul Emmerich
-
Robert LeBlanc