Hi An RGW access denied problem that I can't get anywhere with... * Bucket mybucket owned by user "c" * Bucket policy grants s3:listBucket on mybucket, and s3:putObject & s3:deleteObject on mybucket/* to user "j", and s3:getObject to * (I even granted s3:* on mybucket/* to "j" with no effect) * User "j" can create objects in mybucket, and can delete individual objects (using DELETE) * User "j" get 403 when trying to do a multi-object-delete (POST /mybucket/?delete with a list of 4 object keys) Code is a Java servlet running in Wildfly, loading its credentials from the default ~/.aws/credentials file. It enables path-style access. If I change the credentials in there to those of the bucket owner "c" it works... What's different about permissioning for multi-object-delete? Log file shows access has been granted, but further down there is a suspicious "Permissions for user not found" (don't know if that is expected or not). Thanks, Chris ------- Extract from RGW log with debugging at level 20: 2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete rgw::auth::s3::LocalEngine granted access 2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete rgw::auth::s3::AWSAuthStrategy granted access 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete normalizing buckets and tenants 2020-07-11T17:55:54.038+0100 7f45adad7700 10 s->object=<NULL> s->bucket=mybucket 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete init permissions 2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: rctx=0x7f45adacc288 obj=default.rgw.meta:root:mybucket state=0x5628b912e9a0 s->prefetch_data=0 2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+root+mybucket : hit (requested=0x16, cached=0x17) 2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: s->obj_tag was set empty 2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+root+mybucket : hit (requested=0x11, cached=0x17) 2020-07-11T17:55:54.038+0100 7f45adad7700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>c</ID><DisplayName>C</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>c</ID><DisplayName>C</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: rctx=0x7f45adacc668 obj=default.rgw.meta:users.uid:j state=0x5628b912e9a0 s->prefetch_data=0 2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+users.uid+j : hit (requested=0x6, cached=0x17) 2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: s->obj_tag was set empty 2020-07-11T17:55:54.038+0100 7f45adad7700 20 Read xattr: user.rgw.idtag 2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+users.uid+j : hit (requested=0x3, cached=0x17) 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete recalculating target 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete reading permissions 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete init op 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete verifying op mask 2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete required_mask= 4 user.op_mask=7 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete verifying op permissions 2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete -- Getting permissions begin with perm_mask=50 2020-07-11T17:55:54.038+0100 7f45adad7700 5 req 15 0.004000002s s3:multi_object_delete Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0) mask=50 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for uid=j 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for user not found 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for group=1 mask=50 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for group not found 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for group=2 mask=50 2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for group not found 2020-07-11T17:55:54.038+0100 7f45adad7700 5 req 15 0.004000002s s3:multi_object_delete -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0), owner=c, perm=0 2020-07-11T17:55:54.038+0100 7f45adad7700 10 req 15 0.004000002s s3:multi_object_delete identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=0, user_perm_mask=2, acl perm=0 2020-07-11T17:55:54.038+0100 7f45adad7700 1 op->ERRORHANDLER: err_no=-13 new_err_no=-13 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete op status=0 2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete http status=403 2020-07-11T17:55:54.038+0100 7f45adad7700 1 ====== req done req=0x7f45adaced50 op status=0 http_status=403 latency=0.004000002s ====== 2020-07-11T17:55:54.038+0100 7f45adad7700 20 process_request() returned -13 2020-07-11T17:55:54.038+0100 7f45adad7700 1 civetweb: 0x5628b9424000: 192.168.80.135 - - [11/Jul/2020:17:55:54 +0100] "POST /mybucket/?delete HTTP/1.1" 403 464 - aws-sdk-java/1.11.820 Linux/5.7.7-200.fc32.x86_64 OpenJDK_64-Bit_Server_VM/14.0.1+7 java/14.0.1 vendor/Red_Hat,_Inc.