On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB
On Nautilus the defaults already seem to be: objecter_inflight_op_bytes 104857600 default objecter_inflight_ops 24576 default
Sometimes "ms async op threads" and "ms async max op threads" might help as well (we adjust them by default, but for other reasons)
Thanks, I'll look into these options and see if they help.
Is it possible that I may be running into something like this? https://github.com/boostorg/beast/issues/1077 ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1