On Thu, Oct 17, 2019 at 12:17 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Wed, Oct 16, 2019 at 2:50 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Wed, Oct 16, 2019 at 11:23 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us> wrote:
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Could the 4 GB GET limit saturate the connection from rgw to Ceph? Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has multiple paths, but we aren't balancing on port yet, so RGW to HAProxy is probably limited to one link.
Did you increase "objecter inflight ops" and "objecter inflight op bytes"? You absolutely should adjust these settings for large RGW setups, defaults of 1024 and 100 MB are way too low for many RGW setups, we default to 8192 and 800MB
On Nautilus the defaults already seem to be: objecter_inflight_op_bytes 104857600 default
= 100MiB
objecter_inflight_ops 24576 default
not sure where you got this from, but the default is still 1024 even in master: https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
Looks like it is overridden in https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
you are right, this is new in Nautilus. Last time I had to play around with these settings was indeed on a Mimic deployment.
I'm just not understanding how your suggestions would help, the problem doesn't seem to be on the RADOS side (which it appears your tweaks target), but on the HTTP side as an HTTP health check takes a long time to come back when a big transfer is going on.
I was guessing a bottleneck on the RADOS side because you mentioned that you tried both civetweb and beast, somewhat unlikely to run into the exact same problem with both -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90