RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
Hello Ceph-Users, context or motivation of my question is S3 bucket policies and other cases using the source IP address as condition. I was wondering if and how RadosGW is able to access the source IP address of clients if receiving their connections via a loadbalancer / reverse proxy like HAProxy. So naturally that is where the connection originates from in that case, rendering a policy based on IP addresses useless. Depending on whether the connection balanced as HTTP or TCP there are two ways to carry information about the actual source: * In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log" ([1])? Or is this info used also when evaluating the source IP condition within a bucket policy? * In case of TCP loadbalancing, there is the proxy protocol v2. This unfortunately seems not even supposed by the BEAST library which RGW uses. I opened feature requests ... ** https://tracker.ceph.com/issues/59422 ** https://github.com/chriskohlhoff/asio/issues/1091 ** https://github.com/boostorg/beast/issues/2484 but there is no outcome yet. Regards Christian [1] https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_...
On Thu, Jun 15, 2023 at 7:23 AM Christian Rohmann <christian.rohmann@inovex.de> wrote:
Hello Ceph-Users,
context or motivation of my question is S3 bucket policies and other cases using the source IP address as condition.
I was wondering if and how RadosGW is able to access the source IP address of clients if receiving their connections via a loadbalancer / reverse proxy like HAProxy. So naturally that is where the connection originates from in that case, rendering a policy based on IP addresses useless.
Depending on whether the connection balanced as HTTP or TCP there are two ways to carry information about the actual source:
* In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log" ([1])? Or is this info used also when evaluating the source IP condition within a bucket policy?
yes, the aws:SourceIp condition key does use the value from X-Forwarded-For when present
* In case of TCP loadbalancing, there is the proxy protocol v2. This unfortunately seems not even supposed by the BEAST library which RGW uses. I opened feature requests ...
** https://tracker.ceph.com/issues/59422 ** https://github.com/chriskohlhoff/asio/issues/1091 ** https://github.com/boostorg/beast/issues/2484
but there is no outcome yet.
Regards
Christian
[1] https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, Jun 15, 2023 at 4:48 PM Casey Bodley <cbodley@redhat.com> wrote:
On Thu, Jun 15, 2023 at 7:23 AM Christian Rohmann <christian.rohmann@inovex.de> wrote:
Hello Ceph-Users,
context or motivation of my question is S3 bucket policies and other cases using the source IP address as condition.
I was wondering if and how RadosGW is able to access the source IP address of clients if receiving their connections via a loadbalancer / reverse proxy like HAProxy. So naturally that is where the connection originates from in that case, rendering a policy based on IP addresses useless.
Depending on whether the connection balanced as HTTP or TCP there are two ways to carry information about the actual source:
* In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log"
([1])?
Or is this info used also when evaluating the source IP condition within a bucket policy?
yes, the aws:SourceIp condition key does use the value from X-Forwarded-For when present
maybe use lua scripting to implement your policy based on this HTTP header? (won't help in the TCP balancer case, as lua only see HTTP/S3 level stuff)
* In case of TCP loadbalancing, there is the proxy protocol v2. This unfortunately seems not even supposed by the BEAST library which RGW
uses.
I opened feature requests ...
** https://tracker.ceph.com/issues/59422 ** https://github.com/chriskohlhoff/asio/issues/1091 ** https://github.com/boostorg/beast/issues/2484
but there is no outcome yet.
Regards
Christian
[1]
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_...
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 15/06/2023 15:46, Casey Bodley wrote:
* In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log" ([1])? Or is this info used also when evaluating the source IP condition within a bucket policy? yes, the aws:SourceIp condition key does use the value from X-Forwarded-For when present
I have an HAProxy in front of the RGWs which has "option forwardfor" set to add the "X-Forwarded-For" header. Then the RGWs have "rgw remote addr param = http_x_forwarded_for" set, according to https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_... and I also see remote_addr properly logged within the rgw ops log. But when applying a bucket policy with aws:SourceIp it seems to only work if I set the internal IP of the HAProxy instance, not the public IP of the client. So the actual remote address is NOT used in my case. Did I miss any config setting anywhere? Regards and thanks for your help Christian
On Fri, Jun 16, 2023 at 2:55 AM Christian Rohmann <christian.rohmann@inovex.de> wrote:
On 15/06/2023 15:46, Casey Bodley wrote:
* In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log" ([1])? Or is this info used also when evaluating the source IP condition within a bucket policy?
yes, the aws:SourceIp condition key does use the value from X-Forwarded-For when present
I have an HAProxy in front of the RGWs which has
"option forwardfor" set to add the "X-Forwarded-For" header.
Then the RGWs have "rgw remote addr param = http_x_forwarded_for" set, according to https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_...
and I also see remote_addr properly logged within the rgw ops log.
But when applying a bucket policy with aws:SourceIp it seems to only work if I set the internal IP of the HAProxy instance, not the public IP of the client. So the actual remote address is NOT used in my case.
Did I miss any config setting anywhere?
Regards and thanks for your help
Christian
your 'rgw remote addr param' config looks right. with that same config, i was able to set a bucket policy that denied access based on that X-Forwarded-For header: $ cat bucketpolicy.json { "Version": "2012-10-17", "Id": "S3PolicyId1", "Statement": [ { "Sid": "IPAllow", "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": [ "arn:aws:s3:::testbucket", "arn:aws:s3:::testbucket/*" ], "Condition": { "IpAddress": { "aws:SourceIp": "127.0.0.1" } } } ] } $ s3cmd mb s3://testbucket $ s3cmd setpolicy bucketpolicy.json s3://testbucket $ s3cmd --add-header=X-Forwarded-For:127.0.0.2 put bucketpolicy.json s3://testbucket upload: 'bucketpolicy.json' -> 's3://testbucket/bucketpolicy.json' [1 of 1] 489 of 489 100% in 0s 42.95 KB/s done $ s3cmd --add-header=X-Forwarded-For:127.0.0.1 put bucketpolicy.json s3://testbucket upload: 'bucketpolicy.json' -> 's3://testbucket/bucketpolicy.json' [1 of 1] 489 of 489 100% in 0s 11.08 KB/s done ERROR: S3 error: 403 (AccessDenied)
Hey Casey, all, On 16/06/2023 17:00, Casey Bodley wrote:
But when applying a bucket policy with aws:SourceIp it seems to only work if I set the internal IP of the HAProxy instance, not the public IP of the client. So the actual remote address is NOT used in my case.
Did I miss any config setting anywhere?
your 'rgw remote addr param' config looks right. with that same config, i was able to set a bucket policy that denied access based on
I found the issue. Embarrassingly it was simply a NAT-Hairpin which was applied to the traffic from the server I was testing with. In short: Even though I targeted the public IP from the HAProxy instance the internal IP address of my test server was maintained as source since both machines are on the same network segment. That is why I first thought the LB IP was applied to the policy, but not the actual public source IP of the client. In reality it was simply the private, RFC1918, IP of the test machine that came in as source. Sorry for the noise and thanks for your help. Christian P.S. With IPv6, this would not have happened.
participants (3)
-
Casey Bodley
-
Christian Rohmann
-
Yuval Lifshitz