RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures
Hey Ceph-Users, RGW does have options [1] to rate limit ops or bandwidth per bucket or user. But those only come into play when the request is authenticated. I'd like to also protect the authentication subsystem from malicious or invalid requests. So in case e.g. some EC2 credentials are not valid (anymore) and clients start hammering the RGW with those requests, I'd like to make it cheap to deal with those requests. Especially in case some external authentication like OpenStack Keystone [2] is used, valid access tokens are cached within the RGW. But requests with invalid credentials end up being sent at full rate to the external API [3] as there is no negative caching. And even if there was, that would only limit the external auth requests for the same set of invalid credentials, but it would surely reduce the load in that case: Since the HTTP request is blocking ....
[...] 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 sending request to https://keystone.example.com/v3/s3tokens 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 register_request mgr=0x561a407ae0c0 req_data->id=778, curl_handle=0x7fedaccb36e0 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 WARNING: blocking http request 2023-12-18T15:25:55.861+0000 7fede37fe640 20 link_request req_data=0x561a40a418b0 req_data->id=778, curl_handle=0x7fedaccb36e0 [...]
this does not only stress the external authentication API (keystone in this case), but also blocks RGW threads for the duration of the external call. I am currently looking into using the capabilities of HAProxy to rate limit requests based on their resulting http-response [4]. So in essence to rate-limit or tarpit clients that "produce" a high number of 403 "InvalidAccessKeyId" responses. To have less collateral it might make sense to limit based on the presented credentials themselves. But this would require to extract and track HTTP headers or URL parameters (presigned URLs) [5] and to put them into tables. * What are your thoughts on the matter? * What kind of measures did you put in place? * Does it make sense to extend RGWs capabilities to deal with those cases itself? ** adding negative caching ** rate limits on concurrent external authentication requests (or is there a pool of connections for those requests?) Regards Christian [1] https://docs.ceph.com/en/latest/radosgw/admin/#rate-limit-management [2] https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstack... [3] https://github.com/ceph/ceph/blob/86bb77eb9633bfd002e73b5e58b863bc2d0df594/s... [4] https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#4... [5] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-reques...
Happy New Year Ceph-Users! With the holidays and people likely being away, I take the liberty to bluntly BUMP this question about protecting RGW from DoS below: On 22.12.23 10:24, Christian Rohmann wrote:
Hey Ceph-Users,
RGW does have options [1] to rate limit ops or bandwidth per bucket or user. But those only come into play when the request is authenticated.
I'd like to also protect the authentication subsystem from malicious or invalid requests. So in case e.g. some EC2 credentials are not valid (anymore) and clients start hammering the RGW with those requests, I'd like to make it cheap to deal with those requests. Especially in case some external authentication like OpenStack Keystone [2] is used, valid access tokens are cached within the RGW. But requests with invalid credentials end up being sent at full rate to the external API [3] as there is no negative caching. And even if there was, that would only limit the external auth requests for the same set of invalid credentials, but it would surely reduce the load in that case:
Since the HTTP request is blocking ....
[...] 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 sending request to https://keystone.example.com/v3/s3tokens 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 register_request mgr=0x561a407ae0c0 req_data->id=778, curl_handle=0x7fedaccb36e0 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 WARNING: blocking http request 2023-12-18T15:25:55.861+0000 7fede37fe640 20 link_request req_data=0x561a40a418b0 req_data->id=778, curl_handle=0x7fedaccb36e0 [...]
this does not only stress the external authentication API (keystone in this case), but also blocks RGW threads for the duration of the external call.
I am currently looking into using the capabilities of HAProxy to rate limit requests based on their resulting http-response [4]. So in essence to rate-limit or tarpit clients that "produce" a high number of 403 "InvalidAccessKeyId" responses. To have less collateral it might make sense to limit based on the presented credentials themselves. But this would require to extract and track HTTP headers or URL parameters (presigned URLs) [5] and to put them into tables.
* What are your thoughts on the matter? * What kind of measures did you put in place? * Does it make sense to extend RGWs capabilities to deal with those cases itself? ** adding negative caching ** rate limits on concurrent external authentication requests (or is there a pool of connections for those requests?)
Regards
Christian
[1] https://docs.ceph.com/en/latest/radosgw/admin/#rate-limit-management [2] https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstack... [3] https://github.com/ceph/ceph/blob/86bb77eb9633bfd002e73b5e58b863bc2d0df594/s... [4] https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#4... [5] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-reques...
Hi, I'm using in the frontend https config on haproxy like this, it works so far good: stick-table type ip size 1m expire 10s store http_req_rate(10s) tcp-request inspect-delay 10s tcp-request content track-sc0 src http-request deny deny_status 429 if { sc_http_req_rate(0) gt 10000 } Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ From: Christian Rohmann <christian.rohmann@inovex.de> Sent: Tuesday, January 9, 2024 3:33 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Happy New Year Ceph-Users! With the holidays and people likely being away, I take the liberty to bluntly BUMP this question about protecting RGW from DoS below: On 22.12.23 10:24, Christian Rohmann wrote:
Hey Ceph-Users,
RGW does have options [1] to rate limit ops or bandwidth per bucket or user. But those only come into play when the request is authenticated.
I'd like to also protect the authentication subsystem from malicious or invalid requests. So in case e.g. some EC2 credentials are not valid (anymore) and clients start hammering the RGW with those requests, I'd like to make it cheap to deal with those requests. Especially in case some external authentication like OpenStack Keystone [2] is used, valid access tokens are cached within the RGW. But requests with invalid credentials end up being sent at full rate to the external API [3] as there is no negative caching. And even if there was, that would only limit the external auth requests for the same set of invalid credentials, but it would surely reduce the load in that case:
Since the HTTP request is blocking ....
[...] 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 sending request to https://keystone.example.com/v3/s3tokens 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 register_request mgr=0x561a407ae0c0 req_data->id=778, curl_handle=0x7fedaccb36e0 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 WARNING: blocking http request 2023-12-18T15:25:55.861+0000 7fede37fe640 20 link_request req_data=0x561a40a418b0 req_data->id=778, curl_handle=0x7fedaccb36e0 [...]
this does not only stress the external authentication API (keystone in this case), but also blocks RGW threads for the duration of the external call.
I am currently looking into using the capabilities of HAProxy to rate limit requests based on their resulting http-response [4]. So in essence to rate-limit or tarpit clients that "produce" a high number of 403 "InvalidAccessKeyId" responses. To have less collateral it might make sense to limit based on the presented credentials themselves. But this would require to extract and track HTTP headers or URL parameters (presigned URLs) [5] and to put them into tables.
* What are your thoughts on the matter? * What kind of measures did you put in place? * Does it make sense to extend RGWs capabilities to deal with those cases itself? ** adding negative caching ** rate limits on concurrent external authentication requests (or is there a pool of connections for those requests?)
Regards
Christian
[1] https://docs.ceph.com/en/latest/radosgw/admin/#rate-limit-management [2] https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstack... [3] https://github.com/ceph/ceph/blob/86bb77eb9633bfd002e73b5e58b863bc2d0df594/s... [4] https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#4... [5] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-reques...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hey Istvan, On 10.01.24 03:27, Szabo, Istvan (Agoda) wrote:
I'm using in the frontend https config on haproxy like this, it works so far good:
stick-table type ip size 1m expire 10s store http_req_rate(10s)
tcp-request inspect-delay 10s tcp-request content track-sc0 src http-request deny deny_status 429 if { sc_http_req_rate(0) gt 10000 }
But this serves as a basic rate limit for all request coming from a single IP address, right? My question was rather about limiting clients in regards to authentication requests / unauthorized requests, which end up hammering the auth system (Keystone in my case) at full rate. Regards Christian
participants (2)
-
Christian Rohmann
-
Szabo, Istvan (Agoda)