RGW: Cannot write to bucket anymore
Hello, there is one bucket for a user in our Ceph cluster who is suddenly not able to write to one of his buckets. Reading works fine. All other buckets work fine. If we copy the bucket to another bucket on the same cluster, the error stays. Writing is not possible in the new bucket, too. Interesting: If we copy the contents of the bucket to a bucket in another Ceph cluster the error is gone. So now we know how to solve this but we do not finde the root cause. I checked the policies, lifecycle and versioning. Nothing. The user has FULL_CONTROL. Same settings for the user's other buckets he can still write to. Wenn setting debugging to higher numbers all I can see is something like this while trying to write to the bucket: s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions op->ERRORHANDLER: err_no=-13 new_err_no=-13 cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) s3:put_obj op status=0 s3:put_obj http status=403 1 ====== req done req=0x7fe8bb60a710 op status=0 http_status=403 latency=0.000000000s ====== I still think there is something with a policy or so. When we copy the bucket to another bucket in the same cluster, at first, while copying you can write to the new bucket but when the copy job progresses at one point writing is not possible anymore. But what is it? Best, Malte
On Tue, Mar 19, 2024 at 01:19:34PM +0100, Malte Stroem wrote:
I checked the policies, lifecycle and versioning.
Nothing. The user has FULL_CONTROL. Same settings for the user's other buckets he can still write to.
Wenn setting debugging to higher numbers all I can see is something like this while trying to write to the bucket: Did you get to debug_rgw=20 & debug_ms=1?
s3:put_obj reading permissions
s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions op->ERRORHANDLER: err_no=-13 new_err_no=-13 cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) s3:put_obj op status=0 s3:put_obj http status=403 1 ====== req done req=0x7fe8bb60a710 op status=0 http_status=403 latency=0.000000000s ====== Does an object of the same name exist, possibly versioned, somehow owned by a different user?
`radosgw-admin object stat --bucket=... --object=...` IIRC there would be specific messages saying it was denied by policy, but I haven't checked that part of the codebase in some time. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Hello Robin, thanks a lot. Yes, I set debug to debug_rgw=20 & debug_ms=1. It's that 403 I always get. There is no versioning enabled. There is a lifecycle policy for removing the files after one day. That's all I can find. Do you have any more ideas? Best, Malte On 19.03.24 17:23, Robin H. Johnson wrote:
On Tue, Mar 19, 2024 at 01:19:34PM +0100, Malte Stroem wrote:
I checked the policies, lifecycle and versioning.
Nothing. The user has FULL_CONTROL. Same settings for the user's other buckets he can still write to.
Wenn setting debugging to higher numbers all I can see is something like this while trying to write to the bucket: Did you get to debug_rgw=20 & debug_ms=1?
s3:put_obj reading permissions
s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions op->ERRORHANDLER: err_no=-13 new_err_no=-13 cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) s3:put_obj op status=0 s3:put_obj http status=403 1 ====== req done req=0x7fe8bb60a710 op status=0 http_status=403 latency=0.000000000s ====== Does an object of the same name exist, possibly versioned, somehow owned by a different user?
`radosgw-admin object stat --bucket=... --object=...`
IIRC there would be specific messages saying it was denied by policy, but I haven't checked that part of the codebase in some time.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, Mar 21, 2024 at 11:20:44AM +0100, Malte Stroem wrote:
Hello Robin,
thanks a lot.
Yes, I set debug to debug_rgw=20 & debug_ms=1.
It's that 403 I always get.
There is no versioning enabled.
There is a lifecycle policy for removing the files after one day. Did the object stat call return anything?
Can you show more of the debug output (redact the keys/hostname/filename)? -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Hello Robin, thank you. The object-stat did not show anything suspicious. And the logs do show s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>XXXXX</ID><DisplayName>YYYYY </DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>XXXXX</ID><DisplayName>YYYYY </DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy and than it fails with s3:put_obj http status=403 So we do not see any errors or something. Everything looks the same to the other working buckets. No versioning. But there has to be something. Where can I have a look? I tried almost anything with the aws cli to find something. But there is nothing. Are there any rados or other commands to debug this? Best, Malte On 22.03.24 02:35, Robin H. Johnson wrote:
On Thu, Mar 21, 2024 at 11:20:44AM +0100, Malte Stroem wrote:
Hello Robin,
thanks a lot.
Yes, I set debug to debug_rgw=20 & debug_ms=1.
It's that 403 I always get.
There is no versioning enabled.
There is a lifecycle policy for removing the files after one day. Did the object stat call return anything?
Can you show more of the debug output (redact the keys/hostname/filename)?
participants (2)
-
Malte Stroem
-
Robin H. Johnson