On Thu, Oct 17, 2019 at 2:03 PM Casey Bodley <cbodley@redhat.com> wrote:
This is great news. Anything we can do to help in this effort as it is very important for us?
We would love help here. While most of the groundwork is done, so the remaining work is mostly mechanical.
To summarize the strategy, the beast frontend spawns a coroutine for each client connection, and that coroutine is represented by a boost::asio::yield_context. We wrap this in an 'optional_yield' struct that gets passed to process_request(). The civetweb frontend always passes an empty object (ie null_yield) so that everything runs synchronously. When making calls into librados, we have a rgw_rados_operate() function that supports this optional_yield argument. If it gets a null_yield, it calls the blocking version of librados::IoCtx::operate(). Otherwise it calls a special librados::async_operate() function which suspends the coroutine until completion instead of blocking the thread.
So most of the remaining work is in plumbing this optional_yield variable through all of the code paths under process_request() that call into librados. The rgw_rados_operate() helpers will log a "WARNING: blocking librados call" whenever they block inside of a beast frontend thread, so we can go through the rgw log to identify all of the places that still need a yield context. By iterating on this process, we can eventually remove all of the blocking calls, then set up regression testing to verify that no rgw logs contain that warning.
Here's an example pr from Ali that adds the optional_yield to requests for bucket instance info: https://github.com/ceph/ceph/pull/27898. It extends the get_bucket_info() call to take optional_yield, and passes one in where available, using null_yield to mark the synchronous cases where one isn't available.
I'll work to get familiar with the code base and see if I can submit some PRs to help out. Things are a bit crazy, but this is very important to us too. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1