On 07/02/2021 22:19, Marc wrote:
I was wondering if someone could post a config for haproxy. Is there something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.
Ours is templated out by ceph-ansible; to try and condense out just the interesting bits: global nbthread 24 #this plus 8 rados handles and 600 civetweb threads lets us use all the #CPU on our RGW systems defaults timeout connect 60s timeout client 2m timeout server 2m # give clients chance to benefit from keepalive; but don't # let idle connections linger timeout http-keep-alive 1s frontend listen_https mode http option forwardfor bind :443 ssl crt /etc/ceph/rgwtls.pem stick-table type ip size 1m expire 1h store conn_cur tcp-request content track-sc0 src # tcp-request is processed before http-request # these soft and hard limits templated tcp-request content reject if { sc_conn_cur(0) gt 170 } http-request set-nice 1000 if { sc_conn_cur(0) gt 113 } default_backend rgw_servers backend rgw_servers balance roundrobin #Use our server if it's got connections spare use-server sto-rgw-1 if { srv_conn(sto-rgw-1) le 341 } server sto-rgw-1 172.27.50.8:8443 check ssl verifyhost cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2 maxconn 341 weight 0 #Otherwise, prefer the two network-local servers server sto-rgw-2 172.27.50.9:8443 check ssl verifyhost cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2 maxconn 341 weight 100 server sto-rgw-3 172.27.50.10:8443 check ssl verifyhost cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2 maxconn 341 weight 100 #Finally, the more remote options server sto-rgw-4 172.27.50.136:8443 check ssl verifyhost cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2 maxconn 341 weight 5 server sto-rgw-5 172.27.50.137:8443 check ssl verifyhost cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2 maxconn 341 weight 5 server sto-rgw-6 172.27.50.138:8443 check ssl verifyhost cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2 maxconn 341 weight 5 The aim is to use all available CPU on the RGWs at peak load, but to also try and prevent one user overwhelming the service for everyone else - hence the dropping of idle connections and soft (and then hard) limits on per-IP connections. Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.