RGW STS Token Forbidden error since upgrading to Quincy 17.2.6
Hi, We are using STS tokens to grant temporary access for users. I was running ceph version 15.2.16 and this method worked as expected until the cluster was upgraded to 17.2.6. Using boto3 I am able to assume the correct role, but when trying to use the temporary credentials a forbidden error is reported. This was working before the upgrade. I have also tested on a 15.2.17 cluster and experience the same issue. I have pasted the created role, and boto3 code I am using as well as the error I am running into. Any insight on this issue would be greatly appreciated. Error: botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden Role: { "RoleId": "499eba48-8431-48f7-9aca-76000b9c01cc", "RoleName": "DefaultNoS3Access", "Path": "/", "Arn": "arn:aws:iam:::role/DefaultNoS3Access", "CreateDate": "2023-05-11T12:04:44.367Z", "MaxSessionDuration": 3600, "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principa l\":{\"AWS\":[\"arn:aws:iam:::user/user\"]},\"Action\":[\"sts:AssumeRole\"]} ]}" }, Boto3 Code: response = sts_client.assume_role( RoleArn=role_arn, RoleSessionName="test", Policy=json.dumps( { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObjectAcl", "s3:PutObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::bucket1/*", "arn:aws:s3:::bucket2/*" ] } ] }, separators=(',', ':') ) ) credentials = response['Credentials'] s3_client = boto3.client( 's3', aws_access_key_id=credentials['AccessKeyId'], aws_secret_access_key=credentials['SecretAccessKey'], aws_session_token=credentials['SessionToken'], endpoint_url="http://IP:8080", region_name="" ) bucket_name = 'bucket1' file_key = 'test.txt' local_file_path = '/' s3_client.download_file(bucket_name, file_key, local_file_path)
Hi Austin, Can you try by adding the bucket arn to the Resource section of the policy, like the following: "Resource": [ "arn:aws:s3:::bucket1", "arn:aws:s3:::bucket1/*", "arn:aws:s3:::bucket2/*" ] Thanks, Pritha On Tue, Jun 13, 2023 at 6:02 PM Austin Axworthy <aaxworthy@45drives.com> wrote:
Hi,
We are using STS tokens to grant temporary access for users. I was running ceph version 15.2.16 and this method worked as expected until the cluster was upgraded to 17.2.6.
Using boto3 I am able to assume the correct role, but when trying to use the temporary credentials a forbidden error is reported. This was working before the upgrade. I have also tested on a 15.2.17 cluster and experience the same issue.
I have pasted the created role, and boto3 code I am using as well as the error I am running into. Any insight on this issue would be greatly appreciated.
Error: botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Role:
{
"RoleId": "499eba48-8431-48f7-9aca-76000b9c01cc",
"RoleName": "DefaultNoS3Access",
"Path": "/",
"Arn": "arn:aws:iam:::role/DefaultNoS3Access",
"CreateDate": "2023-05-11T12:04:44.367Z",
"MaxSessionDuration": 3600,
"AssumeRolePolicyDocument":
"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principa
l\":{\"AWS\":[\"arn:aws:iam:::user/user\"]},\"Action\":[\"sts:AssumeRole\"]} ]}"
},
Boto3 Code:
response = sts_client.assume_role(
RoleArn=role_arn,
RoleSessionName="test",
Policy=json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObjectAcl",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket1/*",
"arn:aws:s3:::bucket2/*"
]
}
]
}, separators=(',', ':')
)
)
credentials = response['Credentials']
s3_client = boto3.client(
's3',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
endpoint_url="http://IP:8080",
region_name=""
)
bucket_name = 'bucket1'
file_key = 'test.txt'
local_file_path = '/'
s3_client.download_file(bucket_name, file_key, local_file_path)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Pritha, I have added the bucket to the resource, but I am still running into the same Forbidden response. Thanks, Austin -----Original Message----- From: Pritha Srivastava <prsrivas@redhat.com> Sent: June 14, 2023 4:59 AM To: Austin Axworthy <aaxworthy@45drives.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: RGW STS Token Forbidden error since upgrading to Quincy 17.2.6 Hi Austin, Can you try by adding the bucket arn to the Resource section of the policy, like the following: "Resource": [ "arn:aws:s3:::bucket1", "arn:aws:s3:::bucket1/*", "arn:aws:s3:::bucket2/*" ] Thanks, Pritha On Tue, Jun 13, 2023 at 6:02 PM Austin Axworthy <aaxworthy@45drives.com> wrote:
Hi,
We are using STS tokens to grant temporary access for users. I was running ceph version 15.2.16 and this method worked as expected until the cluster was upgraded to 17.2.6.
Using boto3 I am able to assume the correct role, but when trying to use the temporary credentials a forbidden error is reported. This was working before the upgrade. I have also tested on a 15.2.17 cluster and experience the same issue.
I have pasted the created role, and boto3 code I am using as well as the error I am running into. Any insight on this issue would be greatly appreciated.
Error: botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Role:
{
"RoleId": "499eba48-8431-48f7-9aca-76000b9c01cc",
"RoleName": "DefaultNoS3Access",
"Path": "/",
"Arn": "arn:aws:iam:::role/DefaultNoS3Access",
"CreateDate": "2023-05-11T12:04:44.367Z",
"MaxSessionDuration": 3600,
"AssumeRolePolicyDocument":
"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Pr incipa
l\":{\"AWS\":[\"arn:aws:iam:::user/user\"]},\"Action\":[\"sts:AssumeRo le\"]} ]}"
},
Boto3 Code:
response = sts_client.assume_role(
RoleArn=role_arn,
RoleSessionName="test",
Policy=json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObjectAcl",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket1/*",
"arn:aws:s3:::bucket2/*"
]
}
]
}, separators=(',', ':')
)
)
credentials = response['Credentials']
s3_client = boto3.client(
's3',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
endpoint_url="http://IP:8080",
region_name=""
)
bucket_name = 'bucket1'
file_key = 'test.txt'
local_file_path = '/'
s3_client.download_file(bucket_name, file_key, local_file_path)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Austin, Do you have rgw debug logs that can help debug this? Can you provide more information, as to which user is trying to assume the role - which tenants the user and role belong to? Can you please open a tracker issue with all this information? Thanks, Pritha On Wed, Jun 14, 2023 at 6:14 PM Austin Axworthy <aaxworthy@45drives.com> wrote:
Hi Pritha,
I have added the bucket to the resource, but I am still running into the same Forbidden response.
Thanks, Austin
-----Original Message----- From: Pritha Srivastava <prsrivas@redhat.com> Sent: June 14, 2023 4:59 AM To: Austin Axworthy <aaxworthy@45drives.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: RGW STS Token Forbidden error since upgrading to Quincy 17.2.6
Hi Austin,
Can you try by adding the bucket arn to the Resource section of the policy, like the following:
"Resource": [ "arn:aws:s3:::bucket1", "arn:aws:s3:::bucket1/*", "arn:aws:s3:::bucket2/*"
]
Thanks, Pritha
On Tue, Jun 13, 2023 at 6:02 PM Austin Axworthy <aaxworthy@45drives.com> wrote:
Hi,
We are using STS tokens to grant temporary access for users. I was running ceph version 15.2.16 and this method worked as expected until the cluster was upgraded to 17.2.6.
Using boto3 I am able to assume the correct role, but when trying to use the temporary credentials a forbidden error is reported. This was working before the upgrade. I have also tested on a 15.2.17 cluster and experience the same issue.
I have pasted the created role, and boto3 code I am using as well as the error I am running into. Any insight on this issue would be greatly appreciated.
Error: botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Role:
{
"RoleId": "499eba48-8431-48f7-9aca-76000b9c01cc",
"RoleName": "DefaultNoS3Access",
"Path": "/",
"Arn": "arn:aws:iam:::role/DefaultNoS3Access",
"CreateDate": "2023-05-11T12:04:44.367Z",
"MaxSessionDuration": 3600,
"AssumeRolePolicyDocument":
"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Pr incipa
l\":{\"AWS\":[\"arn:aws:iam:::user/user\"]},\"Action\":[\"sts:AssumeRo le\"]} ]}"
},
Boto3 Code:
response = sts_client.assume_role(
RoleArn=role_arn,
RoleSessionName="test",
Policy=json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObjectAcl",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket1/*",
"arn:aws:s3:::bucket2/*"
]
}
]
}, separators=(',', ':')
)
)
credentials = response['Credentials']
s3_client = boto3.client(
's3',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
endpoint_url="http://IP:8080",
region_name=""
)
bucket_name = 'bucket1'
file_key = 'test.txt'
local_file_path = '/'
s3_client.download_file(bucket_name, file_key, local_file_path)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Pritha, I have increased the debug logs and pasted the output below. I have 2 users, austin and test. Austin is the owner user on the buckets, and I am trying to assume the role with the test user. I have also tried to assume the role of austin with the same user, but still get the same forbidden response. I am able to get the temporary credentials in both cases. I have put in a request on the tracker for an account on Friday. Just waiting to be approved to post there. 2023-06-20T17:07:53.459+0000 7f25335ce700 20 req 14769804432280246561 0.004000006s s3:get_obj get_obj_state: setting s->obj_tag to 5c7714c3-1d41-484a-9ecb-8c4e534b3b02.14130.12619634195847866073 2023-06-20T17:07:53.459+0000 7f25335ce700 15 req 14769804432280246561 0.004000006s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>austin</ID><DisplayName>austin</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>austin</ID><DisplayName>austin</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2023-06-20T17:07:53.459+0000 7f25335ce700 2 req 14769804432280246561 0.004000006s s3:get_obj init op 2023-06-20T17:07:53.459+0000 7f25335ce700 20 req 14769804432280246561 0.004000006s s3:get_obj get_system_obj_state: rctx=0x7f26486cf3f8 obj=default.rgw.meta:users.uid:austin state=0x7f262c05b810 s->prefetch_data=0 2023-06-20T17:07:53.459+0000 7f25335ce700 10 req 14769804432280246561 0.004000006s s3:get_obj cache get: name=default.rgw.meta+users.uid+austin : expiry miss 2023-06-20T17:07:53.459+0000 7f2532dcd700 10 req 14769804432280246561 0.004000006s s3:get_obj cache put: name=default.rgw.meta+users.uid+austin info.flags=0x16 2023-06-20T17:07:53.459+0000 7f2532dcd700 10 req 14769804432280246561 0.004000006s s3:get_obj adding default.rgw.meta+users.uid+austin to cache LRU end 2023-06-20T17:07:53.459+0000 7f2532dcd700 10 req 14769804432280246561 0.004000006s s3:get_obj updating xattr: name=ceph.objclass.version bl.length()=42 2023-06-20T17:07:53.459+0000 7f2532dcd700 20 req 14769804432280246561 0.004000006s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2023-06-20T17:07:53.459+0000 7f2532dcd700 20 req 14769804432280246561 0.004000006s s3:get_obj Read xattr: user.rgw.idtag 2023-06-20T17:07:53.459+0000 7f2532dcd700 10 req 14769804432280246561 0.004000006s s3:get_obj cache get: name=default.rgw.meta+users.uid+austin : type miss (requested=0x13, cached=0x16) 2023-06-20T17:07:53.459+0000 7f2532dcd700 20 req 14769804432280246561 0.004000006s s3:get_obj rados->read ofs=0 len=0 2023-06-20T17:07:53.467+0000 7f25325cc700 20 req 14769804432280246561 0.012000017s s3:get_obj rados_obj.operate() r=0 bl.length=417 2023-06-20T17:07:53.467+0000 7f25325cc700 10 req 14769804432280246561 0.012000017s s3:get_obj cache put: name=default.rgw.meta+users.uid+austin info.flags=0x13 2023-06-20T17:07:53.467+0000 7f25325cc700 10 req 14769804432280246561 0.012000017s s3:get_obj moving default.rgw.meta+users.uid+austin to cache LRU end 2023-06-20T17:07:53.467+0000 7f25325cc700 10 req 14769804432280246561 0.012000017s s3:get_obj updating xattr: name=ceph.objclass.version bl.length()=42 2023-06-20T17:07:53.467+0000 7f25325cc700 2 req 14769804432280246561 0.012000017s s3:get_obj verifying op mask 2023-06-20T17:07:53.467+0000 7f25325cc700 20 req 14769804432280246561 0.012000017s s3:get_obj required_mask= 1 user.op_mask=7 2023-06-20T17:07:53.467+0000 7f25325cc700 2 req 14769804432280246561 0.012000017s s3:get_obj verifying op permissions 2023-06-20T17:07:53.467+0000 7f25325cc700 1 req 14769804432280246561 0.012000017s op->ERRORHANDLER: err_no=-13 new_err_no=-13 2023-06-20T17:07:53.467+0000 7f25325cc700 20 req 14769804432280246561 0.012000017s get_system_obj_state: rctx=0x7f26486cf730 obj=default.rgw.log:script.postrequest. state=0x7f262c05b810 s->prefetch_data=0 2023-06-20T17:07:53.467+0000 7f25325cc700 10 req 14769804432280246561 0.012000017s cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) 2023-06-20T17:07:53.467+0000 7f25325cc700 2 req 14769804432280246561 0.012000017s s3:get_obj op status=0 2023-06-20T17:07:53.467+0000 7f25325cc700 2 req 14769804432280246561 0.012000017s s3:get_obj http status=403 2023-06-20T17:07:53.467+0000 7f25325cc700 1 ====== req done req=0x7f26486d06f0 op status=0 http_status=403 latency=0.012000017s ====== 2023-06-20T17:07:53.467+0000 7f25325cc700 1 beast: 0x7f26486d06f0: 192.168.175.200 - test [20/Jun/2023:17:07:53.455 +0000] "HEAD /active-data/test.txt HTTP/1.1" 403 0 - "Boto3/1.9.253 Python/3.8.10 Linux/5.4. Thanks, Austin -----Original Message----- From: Pritha Srivastava <prsrivas@redhat.com> Sent: June 15, 2023 1:02 AM To: Austin Axworthy <aaxworthy@45drives.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: RGW STS Token Forbidden error since upgrading to Quincy 17.2.6 Hi Austin, Do you have rgw debug logs that can help debug this? Can you provide more information, as to which user is trying to assume the role - which tenants the user and role belong to? Can you please open a tracker issue with all this information? Thanks, Pritha On Wed, Jun 14, 2023 at 6:14 PM Austin Axworthy <aaxworthy@45drives.com> wrote:
Hi Pritha,
I have added the bucket to the resource, but I am still running into the same Forbidden response.
Thanks, Austin
-----Original Message----- From: Pritha Srivastava <prsrivas@redhat.com> Sent: June 14, 2023 4:59 AM To: Austin Axworthy <aaxworthy@45drives.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: RGW STS Token Forbidden error since upgrading to Quincy 17.2.6
Hi Austin,
Can you try by adding the bucket arn to the Resource section of the policy, like the following:
"Resource": [ "arn:aws:s3:::bucket1", "arn:aws:s3:::bucket1/*", "arn:aws:s3:::bucket2/*"
]
Thanks, Pritha
On Tue, Jun 13, 2023 at 6:02 PM Austin Axworthy <aaxworthy@45drives.com> wrote:
Hi,
We are using STS tokens to grant temporary access for users. I was running ceph version 15.2.16 and this method worked as expected until the cluster was upgraded to 17.2.6.
Using boto3 I am able to assume the correct role, but when trying to use the temporary credentials a forbidden error is reported. This was working before the upgrade. I have also tested on a 15.2.17 cluster and experience the same issue.
I have pasted the created role, and boto3 code I am using as well as the error I am running into. Any insight on this issue would be greatly appreciated.
Error: botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Role:
{
"RoleId": "499eba48-8431-48f7-9aca-76000b9c01cc",
"RoleName": "DefaultNoS3Access",
"Path": "/",
"Arn": "arn:aws:iam:::role/DefaultNoS3Access",
"CreateDate": "2023-05-11T12:04:44.367Z",
"MaxSessionDuration": 3600,
"AssumeRolePolicyDocument":
"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\" Pr incipa
l\":{\"AWS\":[\"arn:aws:iam:::user/user\"]},\"Action\":[\"sts:Assume Ro le\"]} ]}"
},
Boto3 Code:
response = sts_client.assume_role(
RoleArn=role_arn,
RoleSessionName="test",
Policy=json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObjectAcl",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket1/*",
"arn:aws:s3:::bucket2/*"
]
}
]
}, separators=(',', ':')
)
)
credentials = response['Credentials']
s3_client = boto3.client(
's3',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
endpoint_url="http://IP:8080",
region_name=""
)
bucket_name = 'bucket1'
file_key = 'test.txt'
local_file_path = '/'
s3_client.download_file(bucket_name, file_key, local_file_path)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Austin Axworthy
-
Pritha Srivastava