15 Jun
2023
15 Jun
'23
2:22 p.m.
Hello Ceph-Users,
context or motivation of my question is S3 bucket policies and other
cases using the source IP address as condition.
I was wondering if and how RadosGW is able to access the source IP
address of clients if receiving their connections via a loadbalancer /
reverse proxy like HAProxy.
So naturally that is where the connection originates from in that case,
rendering a policy based on IP addresses useless.
Depending on whether the connection balanced as HTTP or TCP there are
two ways to carry information about the actual source:
* In case of HTTP via headers like "X-Forwarded-For". This is
apparently supported only for logging the source in the "rgw ops log" ([1])?
Or is this info used also when evaluating the source IP condition within
a bucket policy?
* In case of TCP loadbalancing, there is the proxy protocol v2. This
unfortunately seems not even supposed by the BEAST library which RGW uses.
I opened feature requests ...
** https://tracker.ceph.com/issues/59422
** https://github.com/chriskohlhoff/asio/issues/1091
** https://github.com/boostorg/beast/issues/2484
but there is no outcome yet.
Regards
Christian
[1]
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr…
15 Jun
15 Jun
4:46 p.m.
On Thu, Jun 15, 2023 at 7:23 AM Christian Rohmann
<christian.rohmann(a)inovex.de> wrote:
Hello Ceph-Users,
context or motivation of my question is S3 bucket policies and other
cases using the source IP address as condition.
I was wondering if and how RadosGW is able to access the source IP
address of clients if receiving their connections via a loadbalancer /
reverse proxy like HAProxy.
So naturally that is where the connection originates from in that case,
rendering a policy based on IP addresses useless.
Depending on whether the connection balanced as HTTP or TCP there are
two ways to carry information about the actual source:
* In case of HTTP via headers like "X-Forwarded-For". This is
apparently supported only for logging the source in the "rgw ops log" ([1])?
Or is this info used also when evaluating the source IP condition within
a bucket policy?
yes, the aws:SourceIp condition key does use the value from
X-Forwarded-For when present
* In case of TCP loadbalancing, there is the proxy protocol v2. This
unfortunately seems not even supposed by the BEAST library which RGW uses.
I opened feature requests ...
** https://tracker.ceph.com/issues/59422
** https://github.com/chriskohlhoff/asio/issues/1091
** https://github.com/boostorg/beast/issues/2484
but there is no outcome yet.
Regards
Christian
[1]
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr…
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:21 p.m.
On Thu, Jun 15, 2023 at 4:48 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Thu, Jun 15, 2023 at 7:23 AM Christian Rohmann
<christian.rohmann(a)inovex.de> wrote:
maybe use lua scripting to implement your policy based on this HTTP header?
(won't help in the TCP balancer case, as lua only see HTTP/S3 level stuff)
Hello Ceph-Users,
context or motivation of my question is S3 bucket policies and other
cases using the source IP address as condition.
I was wondering if and how RadosGW is able to access the source IP
address of clients if receiving their connections via a loadbalancer /
reverse proxy like HAProxy.
So naturally that is where the connection originates from in that case,
rendering a policy based on IP addresses useless.
Depending on whether the connection balanced as HTTP or TCP there are
two ways to carry information about the actual source:
* In case of HTTP via headers like "X-Forwarded-For". This is
apparently supported only for logging the source in the "rgw ops log"
([1])?
Or is this info used also when evaluating the
source IP condition within
a bucket policy?
yes, the aws:SourceIp condition key does use the value from
X-Forwarded-For when present
* In case of TCP loadbalancing, there is the proxy protocol v2. This
unfortunately seems not even supposed by the BEAST library which RGW
uses.
I opened feature requests ...
** https://tracker.ceph.com/issues/59422
** https://github.com/chriskohlhoff/asio/issues/1091
** https://github.com/boostorg/beast/issues/2484
but there is no outcome yet.
Regards
Christian
[1]
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr…
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
16 Jun
16 Jun
9:54 a.m.
On 15/06/2023 15:46, Casey Bodley wrote:
I have an HAProxy in front of the RGWs which has
"option forwardfor" set to add the "X-Forwarded-For" header.
Then the RGWs have "rgw remote addr param = http_x_forwarded_for" set,
according to
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr…
and I also see remote_addr properly logged within the rgw ops log.
But when applying a bucket policy with aws:SourceIp it seems to only
work if I set the internal IP of the HAProxy instance, not the public IP
of the client.
So the actual remote address is NOT used in my case.
Did I miss any config setting anywhere?
Regards and thanks for your help
Christian
* In case
of HTTP via headers like "X-Forwarded-For". This is
apparently supported only for logging the source in the "rgw ops log" ([1])?
Or is this info used also when evaluating the source IP condition within
a bucket policy?
yes, the aws:SourceIp condition key does use the value from
X-Forwarded-For when present
6 p.m.
On Fri, Jun 16, 2023 at 2:55 AM Christian Rohmann
<christian.rohmann(a)inovex.de> wrote:
On 15/06/2023 15:46, Casey Bodley wrote:
* In case of HTTP via headers like "X-Forwarded-For". This is
apparently supported only for logging the source in the "rgw ops log" ([1])?
Or is this info used also when evaluating the source IP condition within
a bucket policy?
yes, the aws:SourceIp condition key does use the value from
X-Forwarded-For when present
I have an HAProxy in front of the RGWs which has
"option forwardfor" set to add the "X-Forwarded-For" header.
Then the RGWs have "rgw remote addr param = http_x_forwarded_for" set,
according to
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr…
and I also see remote_addr properly logged within the rgw ops log.
But when applying a bucket policy with aws:SourceIp it seems to only work if I set the
internal IP of the HAProxy instance, not the public IP of the client.
So the actual remote address is NOT used in my case.
Did I miss any config setting anywhere?
Regards and thanks for your help
Christian
your 'rgw remote addr param' config looks right. with that same
config, i was able to set a bucket policy that denied access based on
that X-Forwarded-For header:
$ cat bucketpolicy.json
{
"Version": "2012-10-17",
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::testbucket",
"arn:aws:s3:::testbucket/*"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": "127.0.0.1"
}
}
}
]
}
$ s3cmd mb s3://testbucket
$ s3cmd setpolicy bucketpolicy.json s3://testbucket
$ s3cmd --add-header=X-Forwarded-For:127.0.0.2 put bucketpolicy.json
s3://testbucket
upload: 'bucketpolicy.json' -> 's3://testbucket/bucketpolicy.json' [1
of 1]
489 of 489 100% in 0s 42.95 KB/s done
$ s3cmd --add-header=X-Forwarded-For:127.0.0.1 put bucketpolicy.json
s3://testbucket
upload: 'bucketpolicy.json' -> 's3://testbucket/bucketpolicy.json' [1
of 1]
489 of 489 100% in 0s 11.08 KB/s done
ERROR: S3 error: 403 (AccessDenied)
6 Jul
6 Jul
11:28 a.m.
Hey Casey, all,
On 16/06/2023 17:00, Casey Bodley wrote:
I found the issue. Embarrassingly it was simply a NAT-Hairpin which was
applied to the traffic from the server I was testing with.
In short: Even though I targeted the public IP from the HAProxy instance
the internal IP address of my test server was maintained as source since
both machines are on the same network segment.
That is why I first thought the LB IP was applied to the policy, but not
the actual public source IP of the client. In reality it was simply the
private, RFC1918, IP of the test machine that came in as source.
Sorry for the noise and thanks for your help.
Christian
P.S. With IPv6, this would not have happened.
But when applying a bucket policy with
aws:SourceIp it seems to only work if I set the internal IP of the HAProxy instance, not
the public IP of the client.
So the actual remote address is NOT used in my case.
Did I miss any config setting anywhere?
your 'rgw remote addr param' config looks right. with that same
config, i was able to set a bucket policy that denied access based on
319
days inactive
340
days old
5 comments
3 participants
participants (3)
-
Casey Bodley -
Christian Rohmann -
Yuval Lifshitz