Radogw ignoring HTTP_X_FORWARDED_FOR header
Hello, We are working on setting up an nginx sidecar container running along a RadosGW container inside the same kubernetes pod. We are facing an issue with getting the right value for the header HTTP_X_FORWARDED_FOR when getting client requests. We need this value to do the source ip check validation. Currently, RGW sees that all requests come from 127.0.0.1. So it is still considering the nginx ip address and not the client who made the request. Even though, we configured our RGW the following rgw_remote_addr_param = HTTP_X_FORWARDED_FOR And from the nginx side, we have the following configuration to set the headers. location / { proxy_pass http://localhost:7480; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Port $server_port; Any idea what is the issue here ? Thank you in advance, Yoke
Hello Yosr, On 26/06/2023 11:41, Yosr Kchaou wrote:
We are facing an issue with getting the right value for the header HTTP_X_FORWARDED_FOR when getting client requests. We need this value to do the source ip check validation.
[...]
Currently, RGW sees that all requests come from 127.0.0.1. So it is still considering the nginx ip address and not the client who made the request. May I point you to my recent post to this ML about this very question: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/IKGLAROSVWH...
I am still planning to reproduce this issue with simple examples and headers set manually via e.g. curl to rule out anything stupid I might have misconfigured in my case. I just did not find the time yet. But did you sniff any traffic to the backend or verified how the headers look like in your case? Any debug logging "debug rgw = 20" where you can see what RGW things of the incoming request? Did you test with S3 bucket policies or how did you come to the conclusion that RGW is not using the X_FORWARDED_FOR header? Or what is your indication that things are not working as expected? From what I can see, the rgw client log does NOT print the external IP from the header, but the source IP of the incoming TCP connection: 2023-06-26T11:14:37.070+0000 7f0389e0b700 1 beast: 0x7f051c776660: 192.168.1.1 - someid [26/Jun/2023:11:14:36.990 +0000] "PUT /bucket/object HTTP/1.1" 200 43248 - "aws-sdk-go/1.27.0 (go1.16.15; linux; amd64) S3Manager" - latency=0.079999469s while the rgw ops log does indeed print the remote_address in remote_addr: {"bucket":"bucket","time":"2023-06-26T11:16:08.721465Z","time_local":"2023-06-26T11:16:08.721465+0000","remote_addr":"xxx.xxx.xxx.xxx","user":"someuser","operation":"put_obj","uri":"PUT /bucket/object HTTP/1.1","http_status":"200","error_code":"","bytes_sent":0,"bytes_received":64413,"object_size":64413,"total_time":155,"user_agent":"aws-sdk-go/1.27.0 (go1.16.15; linux; amd64) S3Manager","referrer":"","trans_id":"REDACTED","authentication_type":"Keystone","access_key_id":"REDACTED","temp_url":false} So in my case it's not that RGW does not receive and logs this info, but more about it not applying this in a bucket policy (as far as my analysis of the issue goes). Regards Christian
Hello Christian, Thanks for your reply. So running rgw in debug mode shows the correct value of the header HTTP_X_FORWARDED_FOR. And my test scenario is trying to execute a GET request on a bucket on which policies have been applied. So it should accept requests only from a specific ip. And now I am getting 403 Access denied which makes sense since, the ip the rgw is evaluating, is the proxy ip and not the originator ip. Yosr
Hello, Issue has been found. RGW was not supposed to show the originator's ip in the logs. And my test scenario was not correct. My client was missing some permissions, that's why I get access denied. Yosr
participants (3)
-
Christian Rohmann
-
Yosr Kchaou
-
yosr.kchaou96＠gmail.com