Modify user op status=-125
Hi all, I couldn't understand what does the status -125 mean from the docs. I'm getting 500 response status code when I call rgw admin APIs and the only log in the rgw log files is as follows. s3:get_obj recalculating target initializing for trans_id = tx00000aa90f570fb8281cf-006537bf9e-84395fa-default s3:get_obj reading permissions getting op 1 s3:put_obj verifying requester s3:put_obj normalizing buckets and tenants s3:put_obj init permissions s3:put_obj recalculating target s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions s3:put_obj verifying op params s3:put_obj pre-executing s3:put_obj executing :modify_user completing WARNING: set_req_state_err err_no=125 resorting to 500 :modify_user op status=-125 :modify_user http status=500 ====== req done req=0x7f3f85a78620 op status=-125 http_status=500 latency=0.076000459s ====== Can anyone explain what this error means and why it's happening? Best Regards, Mahnoosh
errno 125 is ECANCELED, which is the code we use when we detect a racing write. so it sounds like something else is modifying that user at the same time. does it eventually succeed if you retry? On Tue, Oct 24, 2023 at 9:21 AM mahnoosh shahidi <mahnooosh.shd@gmail.com> wrote:
Hi all,
I couldn't understand what does the status -125 mean from the docs. I'm getting 500 response status code when I call rgw admin APIs and the only log in the rgw log files is as follows.
s3:get_obj recalculating target initializing for trans_id = tx00000aa90f570fb8281cf-006537bf9e-84395fa-default s3:get_obj reading permissions getting op 1 s3:put_obj verifying requester s3:put_obj normalizing buckets and tenants s3:put_obj init permissions s3:put_obj recalculating target s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions s3:put_obj verifying op params s3:put_obj pre-executing s3:put_obj executing :modify_user completing WARNING: set_req_state_err err_no=125 resorting to 500 :modify_user op status=-125 :modify_user http status=500 ====== req done req=0x7f3f85a78620 op status=-125 http_status=500 latency=0.076000459s ======
Can anyone explain what this error means and why it's happening?
Best Regards, Mahnoosh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Casey for your explanation, Yes it succeeded eventually. Sometimes after about 100 retries. It's odd that it stays in racing condition for that much time. Best Regards, Mahnoosh On Tue, Oct 24, 2023 at 5:17 PM Casey Bodley <cbodley@redhat.com> wrote:
errno 125 is ECANCELED, which is the code we use when we detect a racing write. so it sounds like something else is modifying that user at the same time. does it eventually succeed if you retry?
On Tue, Oct 24, 2023 at 9:21 AM mahnoosh shahidi <mahnooosh.shd@gmail.com> wrote:
Hi all,
I couldn't understand what does the status -125 mean from the docs. I'm getting 500 response status code when I call rgw admin APIs and the only log in the rgw log files is as follows.
s3:get_obj recalculating target initializing for trans_id = tx00000aa90f570fb8281cf-006537bf9e-84395fa-default s3:get_obj reading permissions getting op 1 s3:put_obj verifying requester s3:put_obj normalizing buckets and tenants s3:put_obj init permissions s3:put_obj recalculating target s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions s3:put_obj verifying op params s3:put_obj pre-executing s3:put_obj executing :modify_user completing WARNING: set_req_state_err err_no=125 resorting to 500 :modify_user op status=-125 :modify_user http status=500 ====== req done req=0x7f3f85a78620 op status=-125 http_status=500 latency=0.076000459s ======
Can anyone explain what this error means and why it's happening?
Best Regards, Mahnoosh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
i don't suppose you're using sts roles with AssumeRole? https://tracker.ceph.com/issues/59495 tracks a bug where each AssumeRole request was writing to the user metadata unnecessarily, which would race with your admin api requests On Tue, Oct 24, 2023 at 9:56 AM mahnoosh shahidi <mahnooosh.shd@gmail.com> wrote:
Thanks Casey for your explanation,
Yes it succeeded eventually. Sometimes after about 100 retries. It's odd that it stays in racing condition for that much time.
Best Regards, Mahnoosh
On Tue, Oct 24, 2023 at 5:17 PM Casey Bodley <cbodley@redhat.com> wrote:
errno 125 is ECANCELED, which is the code we use when we detect a racing write. so it sounds like something else is modifying that user at the same time. does it eventually succeed if you retry?
On Tue, Oct 24, 2023 at 9:21 AM mahnoosh shahidi <mahnooosh.shd@gmail.com> wrote:
Hi all,
I couldn't understand what does the status -125 mean from the docs. I'm getting 500 response status code when I call rgw admin APIs and the only log in the rgw log files is as follows.
s3:get_obj recalculating target initializing for trans_id = tx00000aa90f570fb8281cf-006537bf9e-84395fa-default s3:get_obj reading permissions getting op 1 s3:put_obj verifying requester s3:put_obj normalizing buckets and tenants s3:put_obj init permissions s3:put_obj recalculating target s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions s3:put_obj verifying op params s3:put_obj pre-executing s3:put_obj executing :modify_user completing WARNING: set_req_state_err err_no=125 resorting to 500 :modify_user op status=-125 :modify_user http status=500 ====== req done req=0x7f3f85a78620 op status=-125 http_status=500 latency=0.076000459s ======
Can anyone explain what this error means and why it's happening?
Best Regards, Mahnoosh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes this can be the reason. Thanks for your help. Best Regards, Mahnoosh On Tue, Oct 24, 2023 at 5:45 PM Casey Bodley <cbodley@redhat.com> wrote:
i don't suppose you're using sts roles with AssumeRole? https://tracker.ceph.com/issues/59495 tracks a bug where each AssumeRole request was writing to the user metadata unnecessarily, which would race with your admin api requests
On Tue, Oct 24, 2023 at 9:56 AM mahnoosh shahidi <mahnooosh.shd@gmail.com> wrote:
Thanks Casey for your explanation,
Yes it succeeded eventually. Sometimes after about 100 retries. It's odd
that it stays in racing condition for that much time.
Best Regards, Mahnoosh
On Tue, Oct 24, 2023 at 5:17 PM Casey Bodley <cbodley@redhat.com> wrote:
errno 125 is ECANCELED, which is the code we use when we detect a racing write. so it sounds like something else is modifying that user at the same time. does it eventually succeed if you retry?
On Tue, Oct 24, 2023 at 9:21 AM mahnoosh shahidi <mahnooosh.shd@gmail.com> wrote:
Hi all,
I couldn't understand what does the status -125 mean from the docs.
I'm
getting 500 response status code when I call rgw admin APIs and the only log in the rgw log files is as follows.
s3:get_obj recalculating target initializing for trans_id = tx00000aa90f570fb8281cf-006537bf9e-84395fa-default s3:get_obj reading permissions getting op 1 s3:put_obj verifying requester s3:put_obj normalizing buckets and tenants s3:put_obj init permissions s3:put_obj recalculating target s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions s3:put_obj verifying op params s3:put_obj pre-executing s3:put_obj executing :modify_user completing WARNING: set_req_state_err err_no=125 resorting to 500 :modify_user op status=-125 :modify_user http status=500 ====== req done req=0x7f3f85a78620 op status=-125 http_status=500 latency=0.076000459s ======
Can anyone explain what this error means and why it's happening?
Best Regards, Mahnoosh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Casey Bodley
-
mahnoosh shahidi