Hey Mike,
Maybe it's this?
https://tracker.ceph.com/issues/48632
Cheers, Dan
On Wed, Feb 17, 2021, 6:53 PM Mike Cave <mcave(a)uvic.ca> wrote:
I am bumping this email to hopefully get some more
eyes on it.
We are continuing to have this problem. Unfortunately the cluster is very
lightly used currently until we go full production so we do not have the
level of traffic that would generate a lot of statistics.
We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to
correlate with when the errors started popping up.
Our current plan is to roll back the version to 14.2.10 again and rerun
the test that causes the issue.
I noted there was another email thread regarding latencies for a user who
also updated to 14.2.16 recently and I'm not sure if this could be related
or not to my issue.
Any suggestions you may have are very welcomed.
Cheers,
--
Mike Cave
On 2021-02-11, 8:37 AM, "Mike Cave" <mcave(a)uvic.ca> wrote:
So, as the subject states I have an issue with buckets returning a 404
error when they are listed immediately after being created; as well the
bucket fails to be deleted if you try to delete it immediately after
creation.
The behaviour is intermittent.
If I leave the bucket in place for a few minutes, the bucket behaves
normally. I’m thinking this is a metadata issue or something along those
lines but I’m out of my depth now.
To the best of our knowledge the cluster has not changed in any way
since the same tests were run in December with no errors.
We are running Ceph 14.2.16 on all parts of the cluster.
I am using the python-swift client for the connection on a CentOS7
machine.
Can replicate the results from the mons or an external client as well.
I’m willing to share my test script as well if you would like to see
how I’m generating the error.
Here is a piece of the logs in case I missed something in the
interpretation (log level at 20):
14:23:17.069 7faba00df700 1 ====== starting new request
req=0x55fb7a138700 =====
14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id
= tx000000000000000000094-0060245cd5-2b8949-default
14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7
14:23:17.069 7faba00df700 10 host=<NameRemoved>
14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0
in_hosted_domain_s3website=0
14:23:17.069 7faba00df700 -1 res_query() failed
14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain=
in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain=
s->info.request_uri=/swift/v1/404test
14:23:17.069 7faba00df700 10 ver=v1 first=404test req=
14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT
14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2
14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket
scheduling with dmclock client=3 cost=1
14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
verifying requester
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket
rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket
rgw::auth::swift::TempURLEngine denied with reason=-13
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket
rgw::auth::swift::DefaultStrategy: trying
rgw::auth::swift::SignedTokenEngine
14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket
swift_user=xmcc:swift
14:23:17.069 7faba00df700 20 build_token
token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket
rgw::auth::swift::SignedTokenEngine granted access
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
normalizing buckets and tenants
14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init
permissions
14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770
obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0
14:23:17.069 7faba00df700 10 cache get:
name=default.rgw.meta+root+404test : hit (negative entry)
14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130
obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0
14:23:17.069 7faba00df700 10 cache get:
name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17)
14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set
empty
14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag
14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130
obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0
14:23:17.069 7faba00df700 10 cache get:
name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17)
14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set
empty
14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
recalculating target
14:23:17.069 7faba00df700 10 Starting retarget
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
reading permissions
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
verifying op mask
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket
required_mask= 4 user.op_mask=7
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
verifying op permissions
14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket --
Getting permissions begin with perm_mask=50
14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket
Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() ->
rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc,
acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50
14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc
14:23:17.069 7faba00df700 5 Found permission: 15
14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50
14:23:17.069 7faba00df700 5 Permissions for group not found
14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50
14:23:17.069 7faba00df700 5 Permissions for group not found
14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket --
Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier()
-> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc,
acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2
14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket
identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier
-> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift,
perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2,
user_perm_mask=2, acl perm=2
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
verifying op params
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
pre-executing
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
executing
14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR:
bucket 404test not found
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket
completing
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op
status=-2002
14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http
status=404
14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op
status=-2002 http_status=404 latency=0s ======
--
Mike Cave
I acknowledge and respect the Lekwungen-speaking Peoples on whose
traditional territories the university stands and the Songhees, Esquimalt
and WSANEC peoples whose historical relationships with the land continue to
this day.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io