Re: RGW/Swift 404 error when listing/deleting a newly created empty bucket
I am bumping this email to hopefully get some more eyes on it. We are continuing to have this problem. Unfortunately the cluster is very lightly used currently until we go full production so we do not have the level of traffic that would generate a lot of statistics. We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to correlate with when the errors started popping up. Our current plan is to roll back the version to 14.2.10 again and rerun the test that causes the issue. I noted there was another email thread regarding latencies for a user who also updated to 14.2.16 recently and I'm not sure if this could be related or not to my issue. Any suggestions you may have are very welcomed. Cheers, -- Mike Cave On 2021-02-11, 8:37 AM, "Mike Cave" <mcave@uvic.ca> wrote: So, as the subject states I have an issue with buckets returning a 404 error when they are listed immediately after being created; as well the bucket fails to be deleted if you try to delete it immediately after creation. The behaviour is intermittent. If I leave the bucket in place for a few minutes, the bucket behaves normally. I’m thinking this is a metadata issue or something along those lines but I’m out of my depth now. To the best of our knowledge the cluster has not changed in any way since the same tests were run in December with no errors. We are running Ceph 14.2.16 on all parts of the cluster. I am using the python-swift client for the connection on a CentOS7 machine. Can replicate the results from the mons or an external client as well. I’m willing to share my test script as well if you would like to see how I’m generating the error. Here is a piece of the logs in case I missed something in the interpretation (log level at 20): 14:23:17.069 7faba00df700 1 ====== starting new request req=0x55fb7a138700 ===== 14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id = tx000000000000000000094-0060245cd5-2b8949-default 14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7 14:23:17.069 7faba00df700 10 host=<NameRemoved> 14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 14:23:17.069 7faba00df700 -1 res_query() failed 14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/404test 14:23:17.069 7faba00df700 10 ver=v1 first=404test req= 14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket scheduling with dmclock client=3 cost=1 14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying requester 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::TempURLEngine denied with reason=-13 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket swift_user=xmcc:swift 14:23:17.069 7faba00df700 20 build_token token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::SignedTokenEngine granted access 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket normalizing buckets and tenants 14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init permissions 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+root+404test : hit (negative entry) 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket recalculating target 14:23:17.069 7faba00df700 10 Starting retarget 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket reading permissions 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op mask 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket required_mask= 4 user.op_mask=7 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op permissions 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- Getting permissions begin with perm_mask=50 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50 14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc 14:23:17.069 7faba00df700 5 Found permission: 15 14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op params 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket pre-executing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket executing 14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: bucket 404test not found 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket completing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op status=-2002 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http status=404 14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op status=-2002 http_status=404 latency=0s ====== -- Mike Cave I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Mike, do you create ticket for this issue, especially with logs and reproducer? k Sent from my iPhone
On 17 Feb 2021, at 20:54, Mike Cave <mcave@uvic.ca> wrote:
I am bumping this email to hopefully get some more eyes on it.
We are continuing to have this problem. Unfortunately the cluster is very lightly used currently until we go full production so we do not have the level of traffic that would generate a lot of statistics.
We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to correlate with when the errors started popping up.
Our current plan is to roll back the version to 14.2.10 again and rerun the test that causes the issue.
I noted there was another email thread regarding latencies for a user who also updated to 14.2.16 recently and I'm not sure if this could be related or not to my issue.
Any suggestions you may have are very welcomed.
Hi Konstantin, thank you for your response. I have not created a bug report yet, as I was not sure if it was a bug or if I had a configuration issue. A new piece of information is that if I create a bucket, then try to delete it, it fails with a 404. Now if I create a bucket, restart the rgw service, and then try to delete the bucket, it deletes just fine. I'll include this in the bug report. Thank you, -- Mike Cave I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day. On 2021-02-17, 11:51 AM, "Konstantin Shalygin" <k0ste@k0ste.ru> wrote: Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information. Mike, do you create ticket for this issue, especially with logs and reproducer? k Sent from my iPhone > On 17 Feb 2021, at 20:54, Mike Cave <mcave@uvic.ca> wrote: > > I am bumping this email to hopefully get some more eyes on it. > > We are continuing to have this problem. Unfortunately the cluster is very lightly used currently until we go full production so we do not have the level of traffic that would generate a lot of statistics. > > We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to correlate with when the errors started popping up. > > Our current plan is to roll back the version to 14.2.10 again and rerun the test that causes the issue. > > I noted there was another email thread regarding latencies for a user who also updated to 14.2.16 recently and I'm not sure if this could be related or not to my issue. > > Any suggestions you may have are very welcomed.
Hey Mike, Maybe it's this? https://tracker.ceph.com/issues/48632 Cheers, Dan On Wed, Feb 17, 2021, 6:53 PM Mike Cave <mcave@uvic.ca> wrote:
I am bumping this email to hopefully get some more eyes on it.
We are continuing to have this problem. Unfortunately the cluster is very lightly used currently until we go full production so we do not have the level of traffic that would generate a lot of statistics.
We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to correlate with when the errors started popping up.
Our current plan is to roll back the version to 14.2.10 again and rerun the test that causes the issue.
I noted there was another email thread regarding latencies for a user who also updated to 14.2.16 recently and I'm not sure if this could be related or not to my issue.
Any suggestions you may have are very welcomed.
Cheers, -- Mike Cave
On 2021-02-11, 8:37 AM, "Mike Cave" <mcave@uvic.ca> wrote:
So, as the subject states I have an issue with buckets returning a 404 error when they are listed immediately after being created; as well the bucket fails to be deleted if you try to delete it immediately after creation.
The behaviour is intermittent.
If I leave the bucket in place for a few minutes, the bucket behaves normally. I’m thinking this is a metadata issue or something along those lines but I’m out of my depth now.
To the best of our knowledge the cluster has not changed in any way since the same tests were run in December with no errors.
We are running Ceph 14.2.16 on all parts of the cluster.
I am using the python-swift client for the connection on a CentOS7 machine.
Can replicate the results from the mons or an external client as well.
I’m willing to share my test script as well if you would like to see how I’m generating the error.
Here is a piece of the logs in case I missed something in the interpretation (log level at 20):
14:23:17.069 7faba00df700 1 ====== starting new request req=0x55fb7a138700 ===== 14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id = tx000000000000000000094-0060245cd5-2b8949-default 14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7 14:23:17.069 7faba00df700 10 host=<NameRemoved> 14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 14:23:17.069 7faba00df700 -1 res_query() failed 14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/404test 14:23:17.069 7faba00df700 10 ver=v1 first=404test req= 14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket scheduling with dmclock client=3 cost=1 14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying requester 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::TempURLEngine denied with reason=-13 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket swift_user=xmcc:swift 14:23:17.069 7faba00df700 20 build_token token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::SignedTokenEngine granted access 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket normalizing buckets and tenants 14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init permissions 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+root+404test : hit (negative entry) 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket recalculating target 14:23:17.069 7faba00df700 10 Starting retarget 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket reading permissions 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op mask 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket required_mask= 4 user.op_mask=7 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op permissions 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- Getting permissions begin with perm_mask=50 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50 14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc 14:23:17.069 7faba00df700 5 Found permission: 15 14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op params 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket pre-executing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket executing 14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: bucket 404test not found 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket completing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op status=-2002 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http status=404 14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op status=-2002 http_status=404 latency=0s ======
-- Mike Cave
I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for that link Dan. I had searched for a bug but did not find that one. Rolling back to 14.2.10 (our previous version) has resolved our issue for now. I’ll keep an eye out for an update with this bug fix. Cheers, -- Mike Cave From: Dan van der Ster <dan@vanderster.com> Date: Wednesday, February 17, 2021 at 12:53 PM To: Cave Mike <mcave@uvic.ca> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: RGW/Swift 404 error when listing/deleting a newly created empty bucket Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information. Hey Mike, Maybe it's this? https://tracker.ceph.com/issues/48632 Cheers, Dan On Wed, Feb 17, 2021, 6:53 PM Mike Cave <mcave@uvic.ca<mailto:mcave@uvic.ca>> wrote: I am bumping this email to hopefully get some more eyes on it. We are continuing to have this problem. Unfortunately the cluster is very lightly used currently until we go full production so we do not have the level of traffic that would generate a lot of statistics. We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to correlate with when the errors started popping up. Our current plan is to roll back the version to 14.2.10 again and rerun the test that causes the issue. I noted there was another email thread regarding latencies for a user who also updated to 14.2.16 recently and I'm not sure if this could be related or not to my issue. Any suggestions you may have are very welcomed. Cheers, -- Mike Cave On 2021-02-11, 8:37 AM, "Mike Cave" <mcave@uvic.ca<mailto:mcave@uvic.ca>> wrote: So, as the subject states I have an issue with buckets returning a 404 error when they are listed immediately after being created; as well the bucket fails to be deleted if you try to delete it immediately after creation. The behaviour is intermittent. If I leave the bucket in place for a few minutes, the bucket behaves normally. I’m thinking this is a metadata issue or something along those lines but I’m out of my depth now. To the best of our knowledge the cluster has not changed in any way since the same tests were run in December with no errors. We are running Ceph 14.2.16 on all parts of the cluster. I am using the python-swift client for the connection on a CentOS7 machine. Can replicate the results from the mons or an external client as well. I’m willing to share my test script as well if you would like to see how I’m generating the error. Here is a piece of the logs in case I missed something in the interpretation (log level at 20): 14:23:17.069 7faba00df700 1 ====== starting new request req=0x55fb7a138700 ===== 14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id = tx000000000000000000094-0060245cd5-2b8949-default 14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7 14:23:17.069 7faba00df700 10 host=<NameRemoved> 14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 14:23:17.069 7faba00df700 -1 res_query() failed 14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/404test 14:23:17.069 7faba00df700 10 ver=v1 first=404test req= 14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket scheduling with dmclock client=3 cost=1 14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying requester 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::TempURLEngine denied with reason=-13 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket swift_user=xmcc:swift 14:23:17.069 7faba00df700 20 build_token token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::SignedTokenEngine granted access 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket normalizing buckets and tenants 14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init permissions 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+root+404test : hit (negative entry) 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket recalculating target 14:23:17.069 7faba00df700 10 Starting retarget 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket reading permissions 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op mask 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket required_mask= 4 user.op_mask=7 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op permissions 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- Getting permissions begin with perm_mask=50 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50 14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc 14:23:17.069 7faba00df700 5 Found permission: 15 14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op params 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket pre-executing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket executing 14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: bucket 404test not found 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket completing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op status=-2002 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http status=404 14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op status=-2002 http_status=404 latency=0s ====== -- Mike Cave I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
participants (3)
-
Dan van der Ster
-
Konstantin Shalygin
-
Mike Cave