Ceph OIDC Integration
Hello, we have integrated Ceph's RGW with LDAP and have authenticated users using the mail attribute successfully. We would like to shift to SSO and are evaluating the new OIDC feature in Ceph together with dexIdP with an LDAP connector as an upstream IdP. We are trying to understand the flow of the user authentication and how it will effect my current LDAP users buckets which are already created in Ceph as LDAP users. Will the Ceph RGW be able to pass the token to be verified to the IdP and what type of user will then be created in Ceph? Is this the intended way of OIDC integration? Thanks for any assistance
Hello, If it is possible for the uid that has been used for LDAP users to be the same for OIDC users (which is based off the 'sub' field of the OpenID connect token), then there are no extra migration steps needed. Which version of Ceph are you using? In octopus, offline token validation has been introduced, where an incoming web token is validated using the certificate of the IDP. Uptil Octopus, there were no shadow users for OIDC users, but we have introduced shadow user creation in the 'master' branch, and that is done automatically when an AssumeRoleWithWebIdentity call is made. So the metadata to look at right now would be <tenant>$<uid>$buckets which stores the user stats and make sure that the same uid is being used across both LDAP and OIDC (if that is possible), else there is a radosgw-admin user rename command that will rename the user and update all other metadata. Also, please note that currently AssumeRoleWithWebIdentity has been tested only with Keycloak. The documentation for STS in Octopus is here: https://docs.ceph.com/en/octopus/radosgw/STS/ Thanks, Pritha On Mon, Oct 5, 2020 at 9:56 PM <technical@zylacomputing.com> wrote:
Hello, we have integrated Ceph's RGW with LDAP and have authenticated users using the mail attribute successfully. We would like to shift to SSO and are evaluating the new OIDC feature in Ceph together with dexIdP with an LDAP connector as an upstream IdP.
We are trying to understand the flow of the user authentication and how it will effect my current LDAP users buckets which are already created in Ceph as LDAP users.
Will the Ceph RGW be able to pass the token to be verified to the IdP and what type of user will then be created in Ceph? Is this the intended way of OIDC integration?
Thanks for any assistance _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Pritha and thanks for your reply. We are using Ceph Octopus and we have switched to Keycloak from dexIdP. Having said that we have followed the guide from https://docs.ceph.com/en/octopus/radosgw/STS/ but we are constantly having an issue with the AssumeRoleWithWebIdentity example. We are using 2 different accounts for role creation and policy creation and those 2 parts of the example script are working fine but when we move over to the assume_role_with_web_identity part we have forbidden error from Ceph. We have used cephadm to install Ceph which is at: # ceph --version ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable) We used the following command to add the role capabilities for both users: radosgw-admin caps add --uid="TESTER" --caps="roles=*" radosgw-admin caps add --uid="TESTER1" --caps="roles=*" We have set the capabilities for the 2 users mentioned above as shown here: buckets (*) metadata (*) roles (*) usage (*) user-policy (*) users (*) zone (*) --- Can you please confirm that the key values have actually spaces in them or are they missing an underscore? [client.radosgw.gateway] rgw sts key = {sts key for encrypting the session token} rgw s3 auth use sts = true --- We are also getting "NameError: name 'client' is not defined" error from AssumeRoleWithWebIdentity example in this part shown below. Shouldn't it be "sts_client.assume_role_with_web_identity" from "client.assume_role_with_web_identity" as it is being defined as sts_client in the code above it? sts_client = boto3.client('sts', aws_access_key_id=<access_key of TESTER1>, aws_secret_access_key=<secret_key of TESTER1>, endpoint_url=<STS URL>, region_name='', ) response = client.assume_role_with_web_identity( RoleArn=role_response['Role']['Arn'], RoleSessionName='Bob', DurationSeconds=3600, WebIdentityToken=<Web Token> ) Can you or anyone give us some pointers to this issue please?
Hello, rgw sts key should be a key of length 16 since we use AES 128 for encryption (e.g. rgw sts key = abcdefghijklmnop) Yes it should be 'sts_client' and not 'client'. The errors in documentation have been noted and will be corrected. Also please note that the backport to octopus of the new changes is underway (https://github.com/ceph/ceph/pull/37640), and this should be available in the next Octopus release. Thanks, Pritha On Tue, Oct 13, 2020 at 9:22 PM <technical@zylacomputing.com> wrote:
Hi Pritha and thanks for your reply. We are using Ceph Octopus and we have switched to Keycloak from dexIdP.
Having said that we have followed the guide from https://docs.ceph.com/en/octopus/radosgw/STS/ but we are constantly having an issue with the AssumeRoleWithWebIdentity example.
We are using 2 different accounts for role creation and policy creation and those 2 parts of the example script are working fine but when we move over to the assume_role_with_web_identity part we have forbidden error from Ceph.
We have used cephadm to install Ceph which is at: # ceph --version ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)
We used the following command to add the role capabilities for both users: radosgw-admin caps add --uid="TESTER" --caps="roles=*" radosgw-admin caps add --uid="TESTER1" --caps="roles=*"
We have set the capabilities for the 2 users mentioned above as shown here: buckets (*) metadata (*) roles (*) usage (*) user-policy (*) users (*) zone (*)
---
Can you please confirm that the key values have actually spaces in them or are they missing an underscore? [client.radosgw.gateway] rgw sts key = {sts key for encrypting the session token} rgw s3 auth use sts = true
---
We are also getting "NameError: name 'client' is not defined" error from AssumeRoleWithWebIdentity example in this part shown below. Shouldn't it be "sts_client.assume_role_with_web_identity" from "client.assume_role_with_web_identity" as it is being defined as sts_client in the code above it?
sts_client = boto3.client('sts', aws_access_key_id=<access_key of TESTER1>, aws_secret_access_key=<secret_key of TESTER1>, endpoint_url=<STS URL>, region_name='', )
response = client.assume_role_with_web_identity( RoleArn=role_response['Role']['Arn'], RoleSessionName='Bob', DurationSeconds=3600, WebIdentityToken=<Web Token> )
Can you or anyone give us some pointers to this issue please? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Pritha and thanks again for your reply. Unfortunately we are still stuck at the AssumeRoleWithWebIdentity API call as shown below: 2020-10-14T08:24:26.314+0000 7ff6600ff700 1 ====== starting new request req=0x7ff6b69496b0 ===== 2020-10-14T08:24:26.314+0000 7ff6600ff700 2 req 7 0s initializing for trans_id = tx000000000000000000007-005f86b5ba-1e4f8-ZylaS3 2020-10-14T08:24:26.314+0000 7ff6600ff700 10 rgw api priority: s3=8 s3website=7 2020-10-14T08:24:26.314+0000 7ff6600ff700 10 host=192.168.35.28 2020-10-14T08:24:26.314+0000 7ff6600ff700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2020-10-14T08:24:26.314+0000 7ff6600ff700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/ 2020-10-14T08:24:26.314+0000 7ff6600ff700 20 req 7 0s get_handler handler=26RGWHandler_REST_Service_S3 2020-10-14T08:24:26.314+0000 7ff6600ff700 10 handler=26RGWHandler_REST_Service_S3 2020-10-14T08:24:26.314+0000 7ff6600ff700 2 req 7 0s getting op 4 2020-10-14T08:24:26.314+0000 7ff6600ff700 10 Content of POST: Action=AssumeRoleWithWebIdentity&Version=2011-06-15&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2FS3Access&RoleSessionName=KC-Client&DurationSeconds=3600&WebIdentityToken=<token_here> 2020-10-14T08:24:26.315+0000 7ff6600ff700 10 req 7 0.001000005s sts:assume_role_web_identity scheduling with dmclock client=0 cost=1 2020-10-14T08:24:26.315+0000 7ff6600ff700 10 op=31RGWSTSAssumeRoleWithWebIdentity 2020-10-14T08:24:26.315+0000 7ff6600ff700 2 req 7 0.001000005s sts:assume_role_web_identity verifying requester 2020-10-14T08:24:26.315+0000 7ff6600ff700 20 req 7 0.001000005s sts:assume_role_web_identity rgw::auth::sts::DefaultStrategy: trying rgw::auth::sts::WebTokenEngine 2020-10-14T08:24:26.315+0000 7ff6600ff700 20 req 7 0.001000005s sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with reason=-13 2020-10-14T08:24:26.315+0000 7ff6600ff700 5 req 7 0.001000005s sts:assume_role_web_identity Failed the auth strategy, reason=-13 2020-10-14T08:24:26.315+0000 7ff6600ff700 10 failed to authorize request 2020-10-14T08:24:26.315+0000 7ff6600ff700 1 op->ERRORHANDLER: err_no=-13 new_err_no=-13 2020-10-14T08:24:26.315+0000 7ff6600ff700 2 req 7 0.001000005s sts:assume_role_web_identity op status=0 2020-10-14T08:24:26.315+0000 7ff6600ff700 2 req 7 0.001000005s sts:assume_role_web_identity http status=403 2020-10-14T08:24:26.315+0000 7ff6600ff700 1 ====== req done req=0x7ff6b69496b0 op status=0 http_status=403 latency=0.001000005s ====== 2020-10-14T08:24:26.315+0000 7ff6600ff700 1 beast: 0x7ff6b69496b0: 192.168.35.116 - - [2020-10-14T08:24:26.315479+0000] "POST / HTTP/1.1" 403 185 - "Boto3/1.15.16 Python/3.8.5 Linux/5.4.0-48-generic Botocore/1.18.16" - 2020-10-14T08:24:26.383+0000 7ff66690c700 20 failed to read header: end of stream to get the web token we are using the below script and pasting the output in the AssumeRoleWithWebIdentity API call: #!/bin/bash KC_REALM=myrealm KC_CLIENT=rgw-app KC_CLIENT_SECRET=<string from credentials tab in KC of the client> KC_SERVER=<IP>:<port> KC_CONTEXT=auth # Request Tokens for credentials KC_RESPONSE=$( \ curl -k --no-progress-meter -X POST \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "scope=openid" \ -d "grant_type=client_credentials" \ -d "client_id=$KC_CLIENT" \ -d "client_secret=$KC_CLIENT_SECRET" \ "https://$KC_SERVER/$KC_CONTEXT/realms/$KC_REALM/protocol/openid-connect/toke..." \ | jq . ) KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) echo $KC_ACCESS_TOKEN For the above token request to work we had to turn "ON" the option "Service Accounts Enabled" in the settings tab of the client I suspect that the issue is more in relation to the capabilities of the user when calling the STS engine for the AssumeRoleWithWebIdentity API call. If anyone can help... You are more than welcome :-)
Hello again, The issues that you are seeing are because as I mentioned in my previous email, I missed backporting some commits to Octopus (apologies for the same), and I have opened a backport PR (( https://github.com/ceph/ceph/pull/37640) and this should be available in the next Octopus release. Thanks, Pritha On Wed, Oct 14, 2020 at 11:48 PM <technical@zylacomputing.com> wrote:
Hi Pritha and thanks again for your reply. Unfortunately we are still stuck at the AssumeRoleWithWebIdentity API call as shown below:
2020-10-14T08:24:26.314+0000 7ff6600ff700 1 ====== starting new request req=0x7ff6b69496b0 ===== 2020-10-14T08:24:26.314+0000 7ff6600ff700 2 req 7 0s initializing for trans_id = tx000000000000000000007-005f86b5ba-1e4f8-ZylaS3 2020-10-14T08:24:26.314+0000 7ff6600ff700 10 rgw api priority: s3=8 s3website=7 2020-10-14T08:24:26.314+0000 7ff6600ff700 10 host=192.168.35.28 2020-10-14T08:24:26.314+0000 7ff6600ff700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2020-10-14T08:24:26.314+0000 7ff6600ff700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/ 2020-10-14T08:24:26.314+0000 7ff6600ff700 20 req 7 0s get_handler handler=26RGWHandler_REST_Service_S3 2020-10-14T08:24:26.314+0000 7ff6600ff700 10 handler=26RGWHandler_REST_Service_S3 2020-10-14T08:24:26.314+0000 7ff6600ff700 2 req 7 0s getting op 4 2020-10-14T08:24:26.314+0000 7ff6600ff700 10 Content of POST: Action=AssumeRoleWithWebIdentity&Version=2011-06-15&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2FS3Access&RoleSessionName=KC-Client&DurationSeconds=3600&WebIdentityToken=<token_here> 2020-10-14T08:24:26.315+0000 7ff6600ff700 10 req 7 0.001000005s sts:assume_role_web_identity scheduling with dmclock client=0 cost=1 2020-10-14T08:24:26.315+0000 7ff6600ff700 10 op=31RGWSTSAssumeRoleWithWebIdentity 2020-10-14T08:24:26.315+0000 7ff6600ff700 2 req 7 0.001000005s sts:assume_role_web_identity verifying requester 2020-10-14T08:24:26.315+0000 7ff6600ff700 20 req 7 0.001000005s sts:assume_role_web_identity rgw::auth::sts::DefaultStrategy: trying rgw::auth::sts::WebTokenEngine 2020-10-14T08:24:26.315+0000 7ff6600ff700 20 req 7 0.001000005s sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with reason=-13 2020-10-14T08:24:26.315+0000 7ff6600ff700 5 req 7 0.001000005s sts:assume_role_web_identity Failed the auth strategy, reason=-13 2020-10-14T08:24:26.315+0000 7ff6600ff700 10 failed to authorize request 2020-10-14T08:24:26.315+0000 7ff6600ff700 1 op->ERRORHANDLER: err_no=-13 new_err_no=-13 2020-10-14T08:24:26.315+0000 7ff6600ff700 2 req 7 0.001000005s sts:assume_role_web_identity op status=0 2020-10-14T08:24:26.315+0000 7ff6600ff700 2 req 7 0.001000005s sts:assume_role_web_identity http status=403 2020-10-14T08:24:26.315+0000 7ff6600ff700 1 ====== req done req=0x7ff6b69496b0 op status=0 http_status=403 latency=0.001000005s ====== 2020-10-14T08:24:26.315+0000 7ff6600ff700 1 beast: 0x7ff6b69496b0: 192.168.35.116 - - [2020-10-14T08:24:26.315479+0000] "POST / HTTP/1.1" 403 185 - "Boto3/1.15.16 Python/3.8.5 Linux/5.4.0-48-generic Botocore/1.18.16" - 2020-10-14T08:24:26.383+0000 7ff66690c700 20 failed to read header: end of stream
to get the web token we are using the below script and pasting the output in the AssumeRoleWithWebIdentity API call:
#!/bin/bash
KC_REALM=myrealm KC_CLIENT=rgw-app KC_CLIENT_SECRET=<string from credentials tab in KC of the client> KC_SERVER=<IP>:<port> KC_CONTEXT=auth
# Request Tokens for credentials KC_RESPONSE=$( \ curl -k --no-progress-meter -X POST \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "scope=openid" \ -d "grant_type=client_credentials" \ -d "client_id=$KC_CLIENT" \ -d "client_secret=$KC_CLIENT_SECRET" \ "https://$KC_SERVER/$KC_CONTEXT/realms/$KC_REALM/protocol/openid-connect/toke..." \ | jq . )
KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) echo $KC_ACCESS_TOKEN
For the above token request to work we had to turn "ON" the option "Service Accounts Enabled" in the settings tab of the client
I suspect that the issue is more in relation to the capabilities of the user when calling the STS engine for the AssumeRoleWithWebIdentity API call.
If anyone can help... You are more than welcome :-) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Pritha, thanks a lot for your feedback and apologies for missing your comment about the backporting. Would you have a rough estimate on the next Octopus release by any chance? On another note on the same subject, would you be able to give us some feedback on how the users will be created in Ceph? (for example when we used ldap, an ldap user used to be created in Ceph for "mapping", will it be the same in this case) If we have multiple tenants (unique usernames "emails" in KeyCloak) how will the introspect url's be defined for different tenants? Thanks in advance
Hello, The next Octopus release should be there in 3-4 weeks. In Octopus, shadow users aren't created ((for federated oidc users). But we later realised that shadow users are needed to maintain user stats, hence the code for the same is under the process of being added as of now and should be available in Pacific release. We have also done away with the token introspection url in the latest code and have switched over to offline token validation using the IDP's certs since using token introspection url wouldn't scale well for multiple clients. There is a related Ceph Tech Talk here, that you can watch: https://www.youtube.com/watch?v=Lc32meILfNI&t=410s Thanks, Pritha On Mon, Oct 19, 2020 at 8:30 PM <technical@zylacomputing.com> wrote:
Dear Pritha, thanks a lot for your feedback and apologies for missing your comment about the backporting. Would you have a rough estimate on the next Octopus release by any chance?
On another note on the same subject, would you be able to give us some feedback on how the users will be created in Ceph? (for example when we used ldap, an ldap user used to be created in Ceph for "mapping", will it be the same in this case)
If we have multiple tenants (unique usernames "emails" in KeyCloak) how will the introspect url's be defined for different tenants?
Thanks in advance _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Pritha Srivastava
-
technical＠zylacomputing.com