How do I troubleshoot radosgw errors STS?
I've setup RadosGW with STS ontop of my ceph cluster. It works great and fine but I'm also trying to setup authentication with an OpenIDConnect provider. I'm have a hard time troubleshooting issues because the radosgw log file doesn't have much information in it. For example when I try to use the `sts:AssumeRoleWithWebIdentity` API it fails with `{'Code': 'AccessDenied', ...}` and all I see is the beat log showing an HTTP 403. Is there a way to enable more verbose logging so I can see what is failing and why I'm getting certain errors with STS, S3, or IAM apis? My ceph.conf looks like this for each node (mildly redacted): ``` [client.radosgw.pve4] host = pve4 keyring = /etc/pve/priv/ceph.client.radosgw.keyring log file = /var/log/ceph/client.radosgw.$host.log rgw_dns_name = s3.lab rgw_frontends = beast endpoint=0.0.0.0:7480 ssl_endpoint=0.0.0.0:443 ssl_certificate=/etc/pve/priv/ceph/s3.lab.crt ssl_private_key=/etc/pve/priv/ceph/s3.lab.key rgw_sts_key = 1111111111111111 rgw_s3_auth_use_sts = true rgw_enable_apis = s3, s3website, admin, sts, iam ```
Hi, What version of ceph are you using? Can you share the trust policy that is attached to the role being assumed? Thanks, Pritha On Wed, Mar 1, 2023 at 9:07 PM <mat@hazmat.dev> wrote:
I've setup RadosGW with STS ontop of my ceph cluster. It works great and fine but I'm also trying to setup authentication with an OpenIDConnect provider. I'm have a hard time troubleshooting issues because the radosgw log file doesn't have much information in it. For example when I try to use the `sts:AssumeRoleWithWebIdentity` API it fails with `{'Code': 'AccessDenied', ...}` and all I see is the beat log showing an HTTP 403.
Is there a way to enable more verbose logging so I can see what is failing and why I'm getting certain errors with STS, S3, or IAM apis?
My ceph.conf looks like this for each node (mildly redacted):
``` [client.radosgw.pve4] host = pve4 keyring = /etc/pve/priv/ceph.client.radosgw.keyring log file = /var/log/ceph/client.radosgw.$host.log rgw_dns_name = s3.lab rgw_frontends = beast endpoint=0.0.0.0:7480 ssl_endpoint=0.0.0.0:443 ssl_certificate=/etc/pve/priv/ceph/s3.lab.crt ssl_private_key=/etc/pve/priv/ceph/s3.lab.key rgw_sts_key = 1111111111111111 rgw_s3_auth_use_sts = true rgw_enable_apis = s3, s3website, admin, sts, iam ``` _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, I just submitted: https://tracker.ceph.com/issues/58890 Here are more details about the configuration. Note that I've tried a URL with and without a trailing `/` slash like what appears in the ISS. STS OpenIDConnectProvider <pre> { "ClientIDList": [ "radosgw" ], "CreateDate": "2023-03-01T04:05:45.930000+00:00", "ThumbprintList": [ "16A1FBBEE0DC3F78C2013326B2EBA2B9F6D59575" ], "Url": "https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c" } </pre> Role document with the ARN used in the AssumeRoleWithIdentity call. The token returns a "sub" claim with the value of "mathew.utter", e.g. me. <pre> { "RoleId": "53186307-cc98-4904-b867-aa6c2fb10291", "RoleName": "AssumeRoleWithWebIdentityForOIDC", "Path": "/", "Arn": "arn:aws:iam:::role/AssumeRoleWithWebIdentityForOIDC", "CreateDate": "2023-03-01T04:05:46.417Z", "MaxSessionDuration": 3600, "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c:sub\":\"mathew.utter\"}}}]}" } </pre> Policy attached to the role: <pre> { "Permission policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"s3:*\"],\"Resource\":[\"arn:aws:s3:::*\"]}]}" } </pre> There would be a role and policy created for each OIDC user, which is why I'm user the "sub" in the Role.
I will look into the bug that you submitted. Thanks, Pritha On Thu, Mar 2, 2023 at 3:46 AM <mat@hazmat.dev> wrote:
Hello,
I just submitted: https://tracker.ceph.com/issues/58890
Here are more details about the configuration. Note that I've tried a URL with and without a trailing `/` slash like what appears in the ISS.
STS OpenIDConnectProvider
<pre> { "ClientIDList": [ "radosgw" ], "CreateDate": "2023-03-01T04:05:45.930000+00:00", "ThumbprintList": [ "16A1FBBEE0DC3F78C2013326B2EBA2B9F6D59575" ], "Url": "https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c " } </pre>
Role document with the ARN used in the AssumeRoleWithIdentity call. The token returns a "sub" claim with the value of "mathew.utter", e.g. me.
<pre> { "RoleId": "53186307-cc98-4904-b867-aa6c2fb10291", "RoleName": "AssumeRoleWithWebIdentityForOIDC", "Path": "/", "Arn": "arn:aws:iam:::role/AssumeRoleWithWebIdentityForOIDC", "CreateDate": "2023-03-01T04:05:46.417Z", "MaxSessionDuration": 3600, "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c:sub\":\"mathew.utter\"}}}]}" } </pre>
Policy attached to the role:
<pre> { "Permission policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"s3:*\"],\"Resource\":[\"arn:aws:s3:::*\"]}]}" } </pre>
There would be a role and policy created for each OIDC user, which is why I'm user the "sub" in the Role. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
mat＠hazmat.dev
-
Pritha Srivastava