On Tue, Feb 9, 2021 at 11:00 AM Matthew Vernon <mv3@sanger.ac.uk> wrote:
On 07/02/2021 22:19, Marc wrote:
I was wondering if someone could post a config for haproxy. Is there
something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.
Ours is templated out by ceph-ansible; to try and condense out just the interesting bits:
(snipped the config...)
The aim is to use all available CPU on the RGWs at peak load, but to also try and prevent one user overwhelming the service for everyone else - hence the dropping of idle connections and soft (and then hard) limits on per-IP connections.
Can I ask a followup question to this: how many haproxy instances do you then run - one on each of your gateways, with keepalived to manage which is active? I ask because, since before I was involved with our ceph object store, it has been load-balanced between multiple rgw servers directly using bgp-ecmp. It doesn't sound like this is common practise in the ceph community, and I'm wondering what the pros and cons are. The bgp-ecmp load balancing has the flaw that it's not truly fault tolerant, at least without additional checks to shut down the local quagga instance if rgw isn't responding - it's only fault tolerant in the case of an entire server going down, which meets our original goals of rolling maintenance/updates, but not a radosgw process going unresponsive. In addition I think we have always seen some background level of clients being sent "connection reset by peer" errors, which I have never tracked down within radosgw; I wonder if these might be masked by an haproxy frontend? The converse is that all client gateway traffic must generally pass through a single haproxy instance, while bgp-ecmp distributes the connections across all nodes. Perhaps haproxy is lightweight and efficient enough that this makes little difference to performance? Graham