Something like this works… # HAProxy configuration #------------------ # Global settings #------------------ global log /dev/log local0 log /dev/log local1 notice user haproxy group haproxy chroot /var/lib/haproxy daemon stats socket /var/lib/haproxy/stats mode 660 level admin maxconn 65536 spread-checks 4 tune.ssl.default-dh-param 2048 ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets #------------------ # common defaults that all the 'listen' and 'backend' sections will # use- if not designated in their block #------------------ defaults log global mode http retries 3 balance roundrobin option abortonclose option redispatch option dontlognull option log-health-checks maxconn 20480 timeout connect 5s timeout client 50s timeout server 50s timeout http-request 20s timeout http-keep-alive 30s timeout check 10s errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http #------------------ # frontend instances #------------------ frontend ext-http-in bind 10.1.2.10:80 name s3 bind 10.1.2.10:443 ssl crt certificate.pem name secure-s3 maxconn 25000 option forwardfor if-none option http-server-close option httplog default_backend be_rgw-zone1 use_backend be_rgw-zone1 if host_s3 #------------------ # backend instances #------------------ backend be_rgw-zone1 mode http option http-server-close http-request set-header X-Forwarded-Port %[dst_port] http-request add-header X-Forwarded-Proto https if { ssl_fc } server radosgw-vip1 10.1.2.1:80 check server radosgw-vip2 10.1.2.2:80 check server radosgw-vip3 10.1.2.3:80 check From: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Date: Sunday, February 7, 2021 at 8:25 PM To: Marc <Marc@f1-outsourcing.eu>, ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: share haproxy config for radosgw Let me join to this thread, I'd be interested also with HTTPS and beast configuration on HA+Proxy level. Haven't managed to make it ever work. -----Original Message----- From: Marc <Marc@f1-outsourcing.eu> Sent: Monday, February 8, 2021 5:19 AM To: ceph-users@ceph.io Subject: [ceph-users] share haproxy config for radosgw Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________ I was wondering if someone could post a config for haproxy. Is there something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Mine is this with Freddy's option added. frontend radosgw mode http bind abns@radosgw accept-proxy ssl crt xxx.xxxxxxxx.xxx.pem http-request track-sc0 src table per_ip_rates http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 } option forwardfor default_backend radosgw backend radosgw mode http balance roundrobin option http-server-close http-request set-header X-Forwarded-Port %[dst_port] http-request add-header X-Forwarded-Proto https if { ssl_fc } #option log-health-checks option httpchk GET /swift/healthcheck default-server check resolvers dnssvrs1 server-template radosgw1 1 _https._rgw1.prod._tcp.marathon.mesos ssl weight 100 resolve-opts server-template radosgw2 1 _https._rgw2.prod._tcp.marathon.mesos ssl weight 100 resolve-opts

-----Original Message----- From: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Sent: 08 February 2021 04:25 To: Marc <Marc@f1-outsourcing.eu>; ceph-users@ceph.io Subject: RE: share haproxy config for radosgw

Let me join to this thread, I'd be interested also with HTTPS and beast configuration on HA+Proxy level. Haven't managed to make it ever work.

On Tue, Feb 9, 2021 at 11:00 AM Matthew Vernon <mv3@sanger.ac.uk> wrote:

On 07/02/2021 22:19, Marc wrote:

...

I was wondering if someone could post a config for haproxy. Is there

something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.

Ours is templated out by ceph-ansible; to try and condense out just the interesting bits:

(snipped the config...)

The aim is to use all available CPU on the RGWs at peak load, but to also try and prevent one user overwhelming the service for everyone else - hence the dropping of idle connections and soft (and then hard) limits on per-IP connections.

Can I ask a followup question to this: how many haproxy instances do you then run - one on each of your gateways, with keepalived to manage which is active? I ask because, since before I was involved with our ceph object store, it has been load-balanced between multiple rgw servers directly using bgp-ecmp. It doesn't sound like this is common practise in the ceph community, and I'm wondering what the pros and cons are. The bgp-ecmp load balancing has the flaw that it's not truly fault tolerant, at least without additional checks to shut down the local quagga instance if rgw isn't responding - it's only fault tolerant in the case of an entire server going down, which meets our original goals of rolling maintenance/updates, but not a radosgw process going unresponsive. In addition I think we have always seen some background level of clients being sent "connection reset by peer" errors, which I have never tracked down within radosgw; I wonder if these might be masked by an haproxy frontend? The converse is that all client gateway traffic must generally pass through a single haproxy instance, while bgp-ecmp distributes the connections across all nodes. Perhaps haproxy is lightweight and efficient enough that this makes little difference to performance? Graham

On 14/02/2021 21:31, Graham Allan wrote:

On Tue, Feb 9, 2021 at 11:00 AM Matthew Vernon <mv3@sanger.ac.uk <mailto:mv3@sanger.ac.uk>> wrote:

On 07/02/2021 22:19, Marc wrote: > > I was wondering if someone could post a config for haproxy. Is there something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.

Ours is templated out by ceph-ansible; to try and condense out just the interesting bits:

(snipped the config...)

The aim is to use all available CPU on the RGWs at peak load, but to also try and prevent one user overwhelming the service for everyone else - hence the dropping of idle connections and soft (and then hard) limits on per-IP connections.

Can I ask a followup question to this: how many haproxy instances do you then run - one on each of your gateways, with keepalived to manage which is active?

One on each gateway, yes. We use RIP - each RGW listens on each of the 6 service ips (and knows about all 6 RGWs so haproxy can hand off traffic if over-loaded). The switches do some work to make sure traffic from our OpenStack goes to its "nearest" RGW where possible. Like the setup you describe, RIP has no way of knowing if the radosgw has gone down but the host is otherwise up; but haproxy can tell that, which I think is an advantage. We needed to tune the haproxy and radosgw setup to get as much out of the gateway hardware as possible (we used cosbench); redoing the benchmarking bypassing haproxy showed that haproxy had very little impact on performance. Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

You are unlikely to manage to bottleneck HAProxy on anything except the NIC, at least using normal configurations. On Tue, Feb 16, 2021 at 9:12 AM Chip Cox <chip@softiron.com> wrote:

Is this your Graham?

...

On Feb 14, 2021, at 4:31 PM, Graham Allan <gta@umn.edu> wrote:

On Tue, Feb 9, 2021 at 11:00 AM Matthew Vernon <mv3@sanger.ac.uk> wrote:

...

On 07/02/2021 22:19, Marc wrote:

...

I was wondering if someone could post a config for haproxy. Is there

something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.

Ours is templated out by ceph-ansible; to try and condense out just the interesting bits:

(snipped the config...)

The aim is to use all available CPU on the RGWs at peak load, but to also try and prevent one user overwhelming the service for everyone else - hence the dropping of idle connections and soft (and then hard) limits on per-IP connections.

Can I ask a followup question to this: how many haproxy instances do you then run - one on each of your gateways, with keepalived to manage which is active?

I ask because, since before I was involved with our ceph object store, it has been load-balanced between multiple rgw servers directly using bgp-ecmp. It doesn't sound like this is common practise in the ceph community, and I'm wondering what the pros and cons are.

The bgp-ecmp load balancing has the flaw that it's not truly fault tolerant, at least without additional checks to shut down the local quagga instance if rgw isn't responding - it's only fault tolerant in the case of an entire server going down, which meets our original goals of rolling maintenance/updates, but not a radosgw process going unresponsive. In addition I think we have always seen some background level of clients being sent "connection reset by peer" errors, which I have never tracked down within radosgw; I wonder if these might be masked by an haproxy frontend?

The converse is that all client gateway traffic must generally pass through a single haproxy instance, while bgp-ecmp distributes the connections across all nodes. Perhaps haproxy is lightweight and efficient enough that this makes little difference to performance?

Graham _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Chip Cox Director, Sales | SoftIron 770.314.8300 <tel:770.314.8300> chip@softiron.com <mailto:chip@softiron.com>

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io