25 Oct
2023
25 Oct
'23
11:10 p.m.
I have a bucket which got injected with bucket policy which locks the
bucket even to the bucket owner. The bucket now cannot be accessed (even
get its info or delete bucket policy does not work) I have looked in the
radosgw-admin command for a way to delete a bucket policy but do not see
anything. I presume I will need to somehow remove the bucket policy from
however it is stored in the bucket metadata / omap etc. If anyone can point
me in the right direction on that I would appreciate it. Thanks
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
25 Oct
25 Oct
11:40 p.m.
if you have an administrative user (created with --admin), you should
be able to use its credentials with awscli to delete or overwrite this
bucket policy
On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <wes(a)wesdillingham.com> wrote:
I have a bucket which got injected with bucket policy which locks the
bucket even to the bucket owner. The bucket now cannot be accessed (even
get its info or delete bucket policy does not work) I have looked in the
radosgw-admin command for a way to delete a bucket policy but do not see
anything. I presume I will need to somehow remove the bucket policy from
however it is stored in the bucket metadata / omap etc. If anyone can point
me in the right direction on that I would appreciate it. Thanks
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:58 p.m.
Thank you, I am not sure (inherited cluster). I presume such an admin user
created after-the-fact would work? Is there a good way to discover an admin
user other than iterate over all users and retrieve user information? (I
presume radosgw-admin user info --uid=<user>" would illustrate such
administrative access?
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley(a)redhat.com> wrote:
if you have an administrative user (created with
--admin), you should
be able to use its credentials with awscli to delete or overwrite this
bucket policy
On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
I have a bucket which got injected with bucket policy which locks the
bucket even to the bucket owner. The bucket now cannot be accessed (even
get its info or delete bucket policy does not work) I have looked in the
radosgw-admin command for a way to delete a bucket policy but do not see
anything. I presume I will need to somehow remove the bucket policy from
however it is stored in the bucket metadata / omap etc. If anyone can
point
me in the right direction on that I would
appreciate it. Thanks
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
26 Oct
26 Oct
12:10 a.m.
On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham <wes(a)wesdillingham.com> wrote:
Thank you, I am not sure (inherited cluster). I presume such an admin user created
after-the-fact would work?
yes
Is there a good way to discover an admin user other
than iterate over all users and retrieve user information? (I presume radosgw-admin user
info --uid=<user>" would illustrate such administrative access?
not sure there's an easy way to search existing users, but you could
create a temporary admin user for this repair
Respectfully,
Wes Dillingham
wes(a)wesdillingham.com
LinkedIn
On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> if you have an administrative user (created with --admin), you should
> be able to use its credentials with awscli to delete or overwrite this
> bucket policy
>
> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
> >
> > I have a bucket which got injected with bucket policy which locks the
> > bucket even to the bucket owner. The bucket now cannot be accessed (even
> > get its info or delete bucket policy does not work) I have looked in the
> > radosgw-admin command for a way to delete a bucket policy but do not see
> > anything. I presume I will need to somehow remove the bucket policy from
> > however it is stored in the bucket metadata / omap etc. If anyone can point
> > me in the right direction on that I would appreciate it. Thanks
> >
> > Respectfully,
> >
> > *Wes Dillingham*
> > wes(a)wesdillingham.com
> > LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
>
4:43 p.m.
Thank you, this has worked to remove the policy.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham
<wes(a)wesdillingham.com>
wrote:
Thank you, I am not sure (inherited cluster). I presume such an admin
user created
after-the-fact would work?
yes
Is there a good way to discover an admin user
other than iterate over
all users and retrieve user information? (I presume
radosgw-admin user info
--uid=<user>" would illustrate such administrative access?
not sure there's an easy way to search existing users, but you could
create a temporary admin user for this repair
Respectfully,
Wes Dillingham
wes(a)wesdillingham.com
LinkedIn
On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> if you have an administrative user (created with --admin), you should
> be able to use its credentials with awscli to delete or overwrite this
> bucket policy
>
> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
wes(a)wesdillingham.com> wrote:
> >
> > I have a bucket which got injected with bucket policy which locks the
> > bucket even to the bucket owner. The bucket now cannot be accessed
(even
> > get its info or delete bucket policy
does not work) I have looked in
the
> > radosgw-admin command for a way to
delete a bucket policy but do not
see
> > anything. I presume I will need to
somehow remove the bucket policy
from
> > however it is stored in the bucket
metadata / omap etc. If anyone can
point
> > me in the right direction on that I
would appreciate it. Thanks
> >
> > Respectfully,
> >
> > *Wes Dillingham*
> > wes(a)wesdillingham.com
> > LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
>
7 Nov
7 Nov
8:41 p.m.
Hello Wesley and Casey,
We've ended up with the same issue and here it appears that even the user
with "--admin" isn't able to do anything. We're now unable to figure out
if
it is due to bucket policies, ACLs or IAM of some sort. I'm seeing
these IAM errors in the logs
```
Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851
0.003999968s s3:get_obj Error reading *IAM* Policy: Terminate parsing due
to Handler error.
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due
to Handler error.
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
0.000000000s s3:list_bucket init_permissions on
:window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req 13293029267332025583
0.000000000s op->ERRORHANDLER: err_no=-13 new_err_no=-13
```
Please help what's wrong here. We're in Ceph v17.2.7.
Regards,
Jayanth
On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
Thank you, this has worked to remove the policy.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley(a)redhat.com> wrote:
the
can
On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham
<wes(a)wesdillingham.com
wrote:
info
Thank you, I am not sure (inherited cluster). I presume such an admin
user created
after-the-fact would work?
yes
Is there a good way to discover an admin user
other than iterate over
all users and retrieve user information? (I presume
radosgw-admin user --uid=<user>" would illustrate such
administrative access?
not sure there's an easy way to search existing users, but you could
create a temporary admin user for this repair
>
> Respectfully,
>
> Wes Dillingham
> wes(a)wesdillingham.com
> LinkedIn
>
>
> On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
>
> if you have an administrative user (created with --admin), you should
> be able to use its credentials with awscli to delete or overwrite this
> bucket policy
>
> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
wes(a)wesdillingham.com> wrote:
>> >
>> > I have a bucket which got injected with bucket policy which locks > >
bucket even to the bucket owner. The bucket now cannot be accessed
(even
> > get its info or delete bucket policy
does not work) I have looked in
the
> > radosgw-admin command for a way to
delete a bucket policy but do not
see
> > anything. I presume I will need to
somehow remove the bucket policy
from
>> > however it is stored in the bucket metadata / omap etc. If anyone point
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > me in the right direction on that I
would appreciate it. Thanks
> >
> > Respectfully,
> >
> > *Wes Dillingham*
> > wes(a)wesdillingham.com
> > LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
>
9:20 p.m.
On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
<jayanthreddy5666(a)gmail.com> wrote:
Hello Wesley and Casey,
We've ended up with the same issue and here it appears that even the user with
"--admin" isn't able to do anything. We're now unable to figure out if
it is due to bucket policies, ACLs or IAM of some sort. I'm seeing these IAM errors in
the logs
```
Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851 0.003999968s s3:get_obj
Error reading IAM Policy: Terminate parsing due to Handler error.
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 0.000000000s
s3:list_bucket Error reading IAM Policy: Terminate parsing due to Handler error.
it's failing to parse the bucket policy document, but the error
message doesn't say what's wrong with it
disabling rgw_policy_reject_invalid_principals might help if it's
failing on the Principal
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req
13293029267332025583 0.000000000s s3:list_bucket init_permissions on
:window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req 13293029267332025583 0.000000000s
op->ERRORHANDLER: err_no=-13 new_err_no=-13
```
Please help what's wrong here. We're in Ceph v17.2.7.
Regards,
Jayanth
On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <wes(a)wesdillingham.com> wrote:
>
> Thank you, this has worked to remove the policy.
>
> Respectfully,
>
> *Wes Dillingham*
> wes(a)wesdillingham.com
> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>
>
> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham <wes(a)wesdillingham.com>
> > wrote:
> > >
> > > Thank you, I am not sure (inherited cluster). I presume such an admin
> > user created after-the-fact would work?
> >
> > yes
> >
> > > Is there a good way to discover an admin user other than iterate over
> > all users and retrieve user information? (I presume radosgw-admin user info
> > --uid=<user>" would illustrate such administrative access?
> >
> > not sure there's an easy way to search existing users, but you could
> > create a temporary admin user for this repair
> >
> > >
> > > Respectfully,
> > >
> > > Wes Dillingham
> > > wes(a)wesdillingham.com
> > > LinkedIn
> > >
> > >
> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
> > >>
> > >> if you have an administrative user (created with --admin), you should
> > >> be able to use its credentials with awscli to delete or overwrite this
> > >> bucket policy
> > >>
> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
> > wes(a)wesdillingham.com> wrote:
> > >> >
> > >> > I have a bucket which got injected with bucket policy which locks
the
> > >> > bucket even to the bucket owner. The bucket now cannot be
accessed
> > (even
> > >> > get its info or delete bucket policy does not work) I have looked
in
> > the
> > >> > radosgw-admin command for a way to delete a bucket policy but do
not
> > see
> > >> > anything. I presume I will need to somehow remove the bucket
policy
> > from
> > >> > however it is stored in the bucket metadata / omap etc. If anyone
can
> > point
> > >> > me in the right direction on that I would appreciate it. Thanks
> > >> >
> > >> > Respectfully,
> > >> >
> > >> > *Wes Dillingham*
> > >> > wes(a)wesdillingham.com
> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> > >> > _______________________________________________
> > >> > ceph-users mailing list -- ceph-users(a)ceph.io
> > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > >> >
> > >>
> >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:29 p.m.
Hello Casey,
Thank you for the quick response. I see
`rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please
let me know.
Regards
Jayanth
On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
<jayanthreddy5666(a)gmail.com> wrote:
Hello Wesley and Casey,
We've ended up with the same issue and here it appears that even the
user with
"--admin" isn't able to do anything. We're now unable to figure
out if it is due to bucket policies, ACLs or IAM of some sort. I'm seeing
these IAM errors in the logs
```
Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851
0.003999968s
s3:get_obj Error reading IAM Policy: Terminate parsing due to
Handler error.
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
0.000000000s
s3:list_bucket Error reading IAM Policy: Terminate parsing due
to Handler error.
it's failing to parse the bucket policy document, but the error
message doesn't say what's wrong with it
disabling rgw_policy_reject_invalid_principals might help if it's
failing on the Principal
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req
13293029267332025583
0.000000000s s3:list_bucket init_permissions on
:window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]:
req 13293029267332025583
0.000000000s op->ERRORHANDLER: err_no=-13
new_err_no=-13
```
Please help what's wrong here. We're in Ceph v17.2.7.
Regards,
Jayanth
On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
>
> Thank you, this has worked to remove the policy.
>
> Respectfully,
>
> *Wes Dillingham*
> wes(a)wesdillingham.com
> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>
>
> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
>
> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham <
wes(a)wesdillingham.com>
> > wrote:
> > >
> > > Thank you, I am not sure (inherited cluster). I presume such an
admin
> > user created after-the-fact would work?
> >
> > yes
> >
> > > Is there a good way to discover an admin user other than iterate
over
> > all users and retrieve user information?
(I presume radosgw-admin
user info
> > --uid=<user>" would
illustrate such administrative access?
> >
> > not sure there's an easy way to search existing users, but you could
> > create a temporary admin user for this repair
> >
> > >
> > > Respectfully,
> > >
> > > Wes Dillingham
> > > wes(a)wesdillingham.com
> > > LinkedIn
> > >
> > >
> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
> > >>
> > >> if you have an administrative user (created with --admin), you
should
> > >> be able to use its credentials
with awscli to delete or overwrite
this
> > >> bucket policy
> > >>
> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
> > wes(a)wesdillingham.com> wrote:
> > >> >
> > >> > I have a bucket which got injected with bucket policy which
locks the
> > >> > bucket even to the bucket
owner. The bucket now cannot be
accessed
> > (even
> > >> > get its info or delete bucket policy does not work) I have
looked in
> > the
> > >> > radosgw-admin command for a way to delete a bucket policy but do
not
> > see
> > >> > anything. I presume I will need to somehow remove the bucket
policy
> > from
> > >> > however it is stored in the bucket metadata / omap etc. If
anyone can
> > point
> > >> > me in the right direction on that I would appreciate it. Thanks
> > >> >
> > >> > Respectfully,
> > >> >
> > >> > *Wes Dillingham*
> > >> > wes(a)wesdillingham.com
> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> > >> > _______________________________________________
> > >> > ceph-users mailing list -- ceph-users(a)ceph.io
> > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > >> >
> > >>
> >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8 Nov
8 Nov
4:43 a.m.
Hello Casey,
And on further inspection, we identified that there were bucket policies set from the
initial days; we were in v16.2.12.
We upgraded the cluster to v17.2.7 two days ago and it seems obvious that the IAM error
logs are generated the next minute rgw daemon upgraded from v16.2.12 to v17.2.7. Looks
like there is some issue with parsing.
I'm thinking to downgrade back to v17.2.6 and earlier, please let me know if this is a
good option for now.
Thanks,
Jayanth
________________________________
From: Jayanth Reddy <jayanthreddy5666(a)gmail.com>
Sent: Tuesday, November 7, 2023 11:59:38 PM
To: Casey Bodley <cbodley(a)redhat.com>
Cc: Wesley Dillingham <wes(a)wesdillingham.com>om>; ceph-users
<ceph-users(a)ceph.io>io>; Adam Emerson <aemerson(a)redhat.com>
Subject: Re: [ceph-users] Re: owner locked out of bucket via bucket policy
Hello Casey,
Thank you for the quick response. I see `rgw_policy_reject_invalid_principals` is not
present in v17.2.7. Please let me know.
Regards
Jayanth
On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley
<cbodley@redhat.com<mailto:cbodley@redhat.com>> wrote:
On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
<jayanthreddy5666@gmail.com<mailto:jayanthreddy5666@gmail.com>> wrote:
Hello Wesley and Casey,
We've ended up with the same issue and here it appears that even the user with
"--admin" isn't able to do anything. We're now unable to figure out if
it is due to bucket policies, ACLs or IAM of some sort. I'm seeing these IAM errors in
the logs
```
Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851 0.003999968s s3:get_obj
Error reading IAM Policy: Terminate parsing due to Handler error.
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 0.000000000s
s3:list_bucket Error reading IAM Policy: Terminate parsing due to Handler error.
it's failing to parse the bucket policy document, but the error
message doesn't say what's wrong with it
disabling rgw_policy_reject_invalid_principals might help if it's
failing on the Principal
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req
13293029267332025583 0.000000000s s3:list_bucket init_permissions on
:window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req 13293029267332025583 0.000000000s
op->ERRORHANDLER: err_no=-13 new_err_no=-13
```
Please help what's wrong here. We're in Ceph v17.2.7.
Regards,
Jayanth
On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham
<wes@wesdillingham.com<mailto:wes@wesdillingham.com>> wrote:
>
> Thank you, this has worked to remove the policy.
>
> Respectfully,
>
> *Wes Dillingham*
> wes@wesdillingham.com<mailto:wes@wesdillingham.com>
> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>
>
> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley
<cbodley@redhat.com<mailto:cbodley@redhat.com>> wrote:
>
> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham
<wes@wesdillingham.com<mailto:wes@wesdillingham.com>>
> > wrote:
> > >
> > > Thank you, I am not sure (inherited cluster). I presume such an admin
> > user created after-the-fact would work?
> >
> > yes
> >
> > > Is there a good way to discover an admin user other than iterate over
> > all users and retrieve user information? (I presume radosgw-admin user info
> > --uid=<user>" would illustrate such administrative access?
> >
> > not sure there's an easy way to search existing users, but you could
> > create a temporary admin user for this repair
> >
> > >
> > > Respectfully,
> > >
> > > Wes Dillingham
> > > wes@wesdillingham.com<mailto:wes@wesdillingham.com>
> > > LinkedIn
> > >
> > >
> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley
<cbodley@redhat.com<mailto:cbodley@redhat.com>> wrote:
> > >>
> > >> if you have an administrative user (created with --admin), you should
> > >> be able to use its credentials with awscli to delete or overwrite
this
> > >> bucket policy
> > >>
> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
> > wes@wesdillingham.com<mailto:wes@wesdillingham.com>> wrote:
> > >> >
> > >> > I have a bucket which got injected with bucket policy which locks
the
> > >> > bucket even to the bucket owner. The bucket now cannot be
accessed
> > (even
> > >> > get its info or delete bucket policy does not work) I have looked
in
> > the
> > >> > radosgw-admin command for a way to delete a bucket policy but do
not
> > see
> > >> > anything. I presume I will need to somehow remove the bucket
policy
> > from
> > >> > however it is stored in the bucket metadata / omap etc. If anyone
can
> > point
> > >> > me in the right direction on that I would appreciate it. Thanks
> > >> >
> > >> > Respectfully,
> > >> >
> > >> > *Wes Dillingham*
> > >> > wes@wesdillingham.com<mailto:wes@wesdillingham.com>
> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> > >> > _______________________________________________
> > >> > ceph-users mailing list --
ceph-users@ceph.io<mailto:ceph-users@ceph.io>
> > >> > To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
> > >> >
> > >>
> >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
> To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
2:15 p.m.
Hello Casey,
Our Production buckets are impacted due to this issue. We have downgraded Ceph version
from 17.2.7 to 17.2.6 but still we are getting "bucket policy parsing" error
while accessing the buckets. rgw_policy_reject_invalid_principals is not present in 17.2.6
as configurable parameter. Would appreciate response from your end.
Nov 8 16:39:03 [1485064]: req 4696096351995892977 0.000000000s s3:get_obj Error reading
IAM Policy: Terminate parsing due to Handler error.
Nov 8 16:39:03 [1485064]: req 8949648957608194335 0.000000000s s3:get_obj Error reading
IAM Policy: Terminate parsing due to Handler error.
Nov 8 16:39:03 1485064]: req 3856551010860810445 0.003999948s s3:get_obj Error reading
IAM Policy: Terminate parsing due to Handler error.
Nov 8 16:39:03 [1485064]: req 18116384331500039920 0.000000000s s3:get_obj Error reading
IAM Policy: Terminate parsing due to Handler error.
4:29 p.m.
Hello Casey,
We're totally stuck at this point and none of the options seem to work.
Please let us know if there is something in metadata or index to remove
those applied bucket policies. We downgraded to v17.2.6 and encountering
the same.
Regards,
Jayanth
On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
wrote:
Hello Casey,
And on further inspection, we identified that there were bucket policies
set from the initial days; we were in v16.2.12.
We upgraded the cluster to v17.2.7 two days ago and it seems obvious that
the IAM error logs are generated the next minute rgw daemon upgraded from
v16.2.12 to v17.2.7. Looks like there is some issue with parsing.
I'm thinking to downgrade back to v17.2.6 and earlier, please let me know
if this is a good option for now.
Thanks,
Jayanth
------------------------------
*From:* Jayanth Reddy <jayanthreddy5666(a)gmail.com>
*Sent:* Tuesday, November 7, 2023 11:59:38 PM
*To:* Casey Bodley <cbodley(a)redhat.com>
*Cc:* Wesley Dillingham <wes(a)wesdillingham.com>om>; ceph-users <
ceph-users(a)ceph.io>gt;; Adam Emerson <aemerson(a)redhat.com>
*Subject:* Re: [ceph-users] Re: owner locked out of bucket via bucket
policy
Hello Casey,
Thank you for the quick response. I see
`rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please
let me know.
Regards
Jayanth
On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
<jayanthreddy5666(a)gmail.com> wrote:
Hello Wesley and Casey,
We've ended up with the same issue and here it appears that even the
user with
"--admin" isn't able to do anything. We're now unable to figure
out if it is due to bucket policies, ACLs or IAM of some sort. I'm seeing
these IAM errors in the logs
```
Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851
0.003999968s
s3:get_obj Error reading IAM Policy: Terminate parsing due to
Handler error.
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
0.000000000s
s3:list_bucket Error reading IAM Policy: Terminate parsing due
to Handler error.
it's failing to parse the bucket policy document, but the error
message doesn't say what's wrong with it
disabling rgw_policy_reject_invalid_principals might help if it's
failing on the Principal
Nov 7 22:51:40 ceph-05 radosgw[4054570]: req
13293029267332025583
0.000000000s s3:list_bucket init_permissions on
:window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]:
req 13293029267332025583
0.000000000s op->ERRORHANDLER: err_no=-13
new_err_no=-13
```
Please help what's wrong here. We're in Ceph v17.2.7.
Regards,
Jayanth
On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
>
> Thank you, this has worked to remove the policy.
>
> Respectfully,
>
> *Wes Dillingham*
> wes(a)wesdillingham.com
> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>
>
> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
>
> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham <
wes(a)wesdillingham.com>
> > wrote:
> > >
> > > Thank you, I am not sure (inherited cluster). I presume such an
admin
> > user created after-the-fact would work?
> >
> > yes
> >
> > > Is there a good way to discover an admin user other than iterate
over
> > all users and retrieve user information?
(I presume radosgw-admin
user info
> > --uid=<user>" would
illustrate such administrative access?
> >
> > not sure there's an easy way to search existing users, but you could
> > create a temporary admin user for this repair
> >
> > >
> > > Respectfully,
> > >
> > > Wes Dillingham
> > > wes(a)wesdillingham.com
> > > LinkedIn
> > >
> > >
> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
> > >>
> > >> if you have an administrative user (created with --admin), you
should
> > >> be able to use its credentials
with awscli to delete or overwrite
this
> > >> bucket policy
> > >>
> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
> > wes(a)wesdillingham.com> wrote:
> > >> >
> > >> > I have a bucket which got injected with bucket policy which
locks the
> > >> > bucket even to the bucket
owner. The bucket now cannot be
accessed
> > (even
> > >> > get its info or delete bucket policy does not work) I have
looked in
> > the
> > >> > radosgw-admin command for a way to delete a bucket policy but do
not
> > see
> > >> > anything. I presume I will need to somehow remove the bucket
policy
> > from
> > >> > however it is stored in the bucket metadata / omap etc. If
anyone can
> > point
> > >> > me in the right direction on that I would appreciate it. Thanks
> > >> >
> > >> > Respectfully,
> > >> >
> > >> > *Wes Dillingham*
> > >> > wes(a)wesdillingham.com
> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> > >> > _______________________________________________
> > >> > ceph-users mailing list -- ceph-users(a)ceph.io
> > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > >> >
> > >>
> >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:03 p.m.
Jaynath:
Just to be clear with the "--admin" user's key's you have attempted to
delete the bucket policy using the following method:
https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy…
This is what worked for me (on a 16.2.14 cluster). I didn't attempt to
interact with the affected bucket in any way other than "aws s3api
delete-bucket-policy"
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
wrote:
Hello Casey,
We're totally stuck at this point and none of the options seem to work.
Please let us know if there is something in metadata or index to remove
those applied bucket policies. We downgraded to v17.2.6 and encountering
the same.
Regards,
Jayanth
On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
wrote:
> Hello Casey,
>
> And on further inspection, we identified that there were bucket policies
> set from the initial days; we were in v16.2.12.
> We upgraded the cluster to v17.2.7 two days ago and it seems obvious that
> the IAM error logs are generated the next minute rgw daemon upgraded from
> v16.2.12 to v17.2.7. Looks like there is some issue with parsing.
>
> I'm thinking to downgrade back to v17.2.6 and earlier, please let me know
> if this is a good option for now.
>
> Thanks,
> Jayanth
> ------------------------------
> *From:* Jayanth Reddy <jayanthreddy5666(a)gmail.com>
> *Sent:* Tuesday, November 7, 2023 11:59:38 PM
> *To:* Casey Bodley <cbodley(a)redhat.com>
> *Cc:* Wesley Dillingham <wes(a)wesdillingham.com>om>; ceph-users <
> ceph-users(a)ceph.io>gt;; Adam Emerson <aemerson(a)redhat.com>
> *Subject:* Re: [ceph-users] Re: owner locked out of bucket via bucket
> policy
>
> Hello Casey,
>
> Thank you for the quick response. I see
> `rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please
> let me know.
>
> Regards
> Jayanth
>
> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
> <jayanthreddy5666(a)gmail.com> wrote:
> >
> > Hello Wesley and Casey,
> >
> > We've ended up with the same issue and here it appears that even the
> user with "--admin" isn't able to do anything. We're now unable to
figure
> out if it is due to bucket policies, ACLs or IAM of some sort. I'm seeing
> these IAM errors in the logs
> >
> > ```
> >
> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851
> 0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due to
> Handler error.
> >
> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
> 0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due
> to Handler error.
>
> it's failing to parse the bucket policy document, but the error
> message doesn't say what's wrong with it
>
> disabling rgw_policy_reject_invalid_principals might help if it's
> failing on the Principal
>
> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
> 0.000000000s s3:list_bucket init_permissions on
> :window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
> > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req 13293029267332025583
> 0.000000000s op->ERRORHANDLER: err_no=-13 new_err_no=-13
> >
> > ```
> >
> > Please help what's wrong here. We're in Ceph v17.2.7.
> >
> > Regards,
> > Jayanth
> >
> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <
> wes(a)wesdillingham.com> wrote:
> >>
> >> Thank you, this has worked to remove the policy.
> >>
> >> Respectfully,
> >>
> >> *Wes Dillingham*
> >> wes(a)wesdillingham.com
> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> >>
> >>
> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley(a)redhat.com>
> wrote:
> >>
> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham <
> wes(a)wesdillingham.com>
> >> > wrote:
> >> > >
> >> > > Thank you, I am not sure (inherited cluster). I presume such an
> admin
> >> > user created after-the-fact would work?
> >> >
> >> > yes
> >> >
> >> > > Is there a good way to discover an admin user other than iterate
> over
> >> > all users and retrieve user information? (I presume radosgw-admin
> user info
> >> > --uid=<user>" would illustrate such administrative access?
> >> >
> >> > not sure there's an easy way to search existing users, but you
could
> >> > create a temporary admin user for this repair
> >> >
> >> > >
> >> > > Respectfully,
> >> > >
> >> > > Wes Dillingham
> >> > > wes(a)wesdillingham.com
> >> > > LinkedIn
> >> > >
> >> > >
> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley
<cbodley(a)redhat.com>
> wrote:
> >> > >>
> >> > >> if you have an administrative user (created with --admin),
you
> should
> >> > >> be able to use its credentials with awscli to delete or
overwrite
> this
> >> > >> bucket policy
> >> > >>
> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
> >> > wes(a)wesdillingham.com> wrote:
> >> > >> >
> >> > >> > I have a bucket which got injected with bucket policy
which
> locks the
> >> > >> > bucket even to the bucket owner. The bucket now cannot
be
> accessed
> >> > (even
> >> > >> > get its info or delete bucket policy does not work) I
have
> looked in
> >> > the
> >> > >> > radosgw-admin command for a way to delete a bucket policy
but
> do not
> >> > see
> >> > >> > anything. I presume I will need to somehow remove the
bucket
> policy
> >> > from
> >> > >> > however it is stored in the bucket metadata / omap etc.
If
> anyone can
> >> > point
> >> > >> > me in the right direction on that I would appreciate it.
Thanks
> >> > >> >
> >> > >> > Respectfully,
> >> > >> >
> >> > >> > *Wes Dillingham*
> >> > >> > wes(a)wesdillingham.com
> >> > >> > LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
> >> > >> > _______________________________________________
> >> > >> > ceph-users mailing list -- ceph-users(a)ceph.io
> >> > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >> > >> >
> >> > >>
> >> >
> >> >
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
5:06 p.m.
Hello Wesley,
Thank you for the response. I tried the same but ended up with 403.
Regards,
Jayanth
On Wed, Nov 8, 2023 at 7:34 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
Jaynath:
Just to be clear with the "--admin" user's key's you have attempted to
delete the bucket policy using the following method:
https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy…
This is what worked for me (on a 16.2.14 cluster). I didn't attempt to
interact with the affected bucket in any way other than "aws s3api
delete-bucket-policy"
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
wrote:
> Hello Casey,
>
> We're totally stuck at this point and none of the options seem to work.
> Please let us know if there is something in metadata or index to remove
> those applied bucket policies. We downgraded to v17.2.6 and encountering
> the same.
>
> Regards,
> Jayanth
>
> On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
> wrote:
>
>> Hello Casey,
>>
>> And on further inspection, we identified that there were bucket policies
>> set from the initial days; we were in v16.2.12.
>> We upgraded the cluster to v17.2.7 two days ago and it seems obvious
>> that the IAM error logs are generated the next minute rgw daemon upgraded
>> from v16.2.12 to v17.2.7. Looks like there is some issue with parsing.
>>
>> I'm thinking to downgrade back to v17.2.6 and earlier, please let me
>> know if this is a good option for now.
>>
>> Thanks,
>> Jayanth
>> ------------------------------
>> *From:* Jayanth Reddy <jayanthreddy5666(a)gmail.com>
>> *Sent:* Tuesday, November 7, 2023 11:59:38 PM
>> *To:* Casey Bodley <cbodley(a)redhat.com>
>> *Cc:* Wesley Dillingham <wes(a)wesdillingham.com>om>; ceph-users <
>> ceph-users(a)ceph.io>gt;; Adam Emerson <aemerson(a)redhat.com>
>> *Subject:* Re: [ceph-users] Re: owner locked out of bucket via bucket
>> policy
>>
>> Hello Casey,
>>
>> Thank you for the quick response. I see
>> `rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please
>> let me know.
>>
>> Regards
>> Jayanth
>>
>> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>>
>> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
>> <jayanthreddy5666(a)gmail.com> wrote:
>> >
>> > Hello Wesley and Casey,
>> >
>> > We've ended up with the same issue and here it appears that even the
>> user with "--admin" isn't able to do anything. We're now unable
to figure
>> out if it is due to bucket policies, ACLs or IAM of some sort. I'm seeing
>> these IAM errors in the logs
>> >
>> > ```
>> >
>> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851
>> 0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due to
>> Handler error.
>> >
>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
>> 0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due
>> to Handler error.
>>
>> it's failing to parse the bucket policy document, but the error
>> message doesn't say what's wrong with it
>>
>> disabling rgw_policy_reject_invalid_principals might help if it's
>> failing on the Principal
>>
>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
>> 0.000000000s s3:list_bucket init_permissions on
>> :window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
>> > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req
>> 13293029267332025583 0.000000000s op->ERRORHANDLER: err_no=-13
>> new_err_no=-13
>> >
>> > ```
>> >
>> > Please help what's wrong here. We're in Ceph v17.2.7.
>> >
>> > Regards,
>> > Jayanth
>> >
>> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <
>> wes(a)wesdillingham.com> wrote:
>> >>
>> >> Thank you, this has worked to remove the policy.
>> >>
>> >> Respectfully,
>> >>
>> >> *Wes Dillingham*
>> >> wes(a)wesdillingham.com
>> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>> >>
>> >>
>> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley(a)redhat.com>
>> wrote:
>> >>
>> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham <
>> wes(a)wesdillingham.com>
>> >> > wrote:
>> >> > >
>> >> > > Thank you, I am not sure (inherited cluster). I presume such
an
>> admin
>> >> > user created after-the-fact would work?
>> >> >
>> >> > yes
>> >> >
>> >> > > Is there a good way to discover an admin user other than
iterate
>> over
>> >> > all users and retrieve user information? (I presume radosgw-admin
>> user info
>> >> > --uid=<user>" would illustrate such administrative
access?
>> >> >
>> >> > not sure there's an easy way to search existing users, but you
could
>> >> > create a temporary admin user for this repair
>> >> >
>> >> > >
>> >> > > Respectfully,
>> >> > >
>> >> > > Wes Dillingham
>> >> > > wes(a)wesdillingham.com
>> >> > > LinkedIn
>> >> > >
>> >> > >
>> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley
<cbodley(a)redhat.com>
>> wrote:
>> >> > >>
>> >> > >> if you have an administrative user (created with --admin),
you
>> should
>> >> > >> be able to use its credentials with awscli to delete or
>> overwrite this
>> >> > >> bucket policy
>> >> > >>
>> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
>> >> > wes(a)wesdillingham.com> wrote:
>> >> > >> >
>> >> > >> > I have a bucket which got injected with bucket policy
which
>> locks the
>> >> > >> > bucket even to the bucket owner. The bucket now
cannot be
>> accessed
>> >> > (even
>> >> > >> > get its info or delete bucket policy does not work) I
have
>> looked in
>> >> > the
>> >> > >> > radosgw-admin command for a way to delete a bucket
policy but
>> do not
>> >> > see
>> >> > >> > anything. I presume I will need to somehow remove the
bucket
>> policy
>> >> > from
>> >> > >> > however it is stored in the bucket metadata / omap
etc. If
>> anyone can
>> >> > point
>> >> > >> > me in the right direction on that I would appreciate
it. Thanks
>> >> > >> >
>> >> > >> > Respectfully,
>> >> > >> >
>> >> > >> > *Wes Dillingham*
>> >> > >> > wes(a)wesdillingham.com
>> >> > >> > LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
>> >> > >> > _______________________________________________
>> >> > >> > ceph-users mailing list -- ceph-users(a)ceph.io
>> >> > >> > To unsubscribe send an email to
ceph-users-leave(a)ceph.io
>> >> > >> >
>> >> > >>
>> >> >
>> >> >
>> >> _______________________________________________
>> >> ceph-users mailing list -- ceph-users(a)ceph.io
>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>>
5:45 p.m.
i've opened https://tracker.ceph.com/issues/63485 to allow
admin/system users to override policy parsing errors like this. i'm
not sure yet where this parsing regression was introduced. in reef,
https://github.com/ceph/ceph/pull/49395 added better error messages
here, along with a rgw_policy_reject_invalid_principals option to be
strict about principal names
to remove a bucket policy that fails to parse with "Error reading IAM
Policy", you can follow these steps:
1. find the bucket's instance id using the 'bucket stats' command
$ radosgw-admin bucket stats --bucket {bucketname} | grep id
2. use the rados tool to remove the bucket policy attribute
(user.rgw.iam-policy) from the bucket instance metadata object
$ rados -p default.rgw.meta -N root rmxattr
.bucket.meta.{bucketname}:{bucketid} user.rgw.iam-policy
3. radosgws may be caching the existing bucket metadata and xattrs, so
you'd either need to restart them or clear their metadata caches
$ ceph daemon client.rgw.xyz cache zap
On Wed, Nov 8, 2023 at 9:06 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com> wrote:
Hello Wesley,
Thank you for the response. I tried the same but ended up with 403.
Regards,
Jayanth
On Wed, Nov 8, 2023 at 7:34 PM Wesley Dillingham <wes(a)wesdillingham.com> wrote:
>
> Jaynath:
>
> Just to be clear with the "--admin" user's key's you have attempted
to delete the bucket policy using the following method:
https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy…
>
> This is what worked for me (on a 16.2.14 cluster). I didn't attempt to interact
with the affected bucket in any way other than "aws s3api delete-bucket-policy"
>
> Respectfully,
>
> Wes Dillingham
> wes(a)wesdillingham.com
> LinkedIn
>
>
> On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
wrote:
>>
>> Hello Casey,
>>
>> We're totally stuck at this point and none of the options seem to work.
Please let us know if there is something in metadata or index to remove those applied
bucket policies. We downgraded to v17.2.6 and encountering the same.
>>
>> Regards,
>> Jayanth
>>
>> On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
wrote:
>>>
>>> Hello Casey,
>>>
>>> And on further inspection, we identified that there were bucket policies set
from the initial days; we were in v16.2.12.
>>> We upgraded the cluster to v17.2.7 two days ago and it seems obvious that the
IAM error logs are generated the next minute rgw daemon upgraded from v16.2.12 to v17.2.7.
Looks like there is some issue with parsing.
>>>
>>> I'm thinking to downgrade back to v17.2.6 and earlier, please let me know
if this is a good option for now.
>>>
>>> Thanks,
>>> Jayanth
>>> ________________________________
>>> From: Jayanth Reddy <jayanthreddy5666(a)gmail.com>
>>> Sent: Tuesday, November 7, 2023 11:59:38 PM
>>> To: Casey Bodley <cbodley(a)redhat.com>
>>> Cc: Wesley Dillingham <wes(a)wesdillingham.com>om>; ceph-users
<ceph-users(a)ceph.io>io>; Adam Emerson <aemerson(a)redhat.com>
>>> Subject: Re: [ceph-users] Re: owner locked out of bucket via bucket policy
>>>
>>> Hello Casey,
>>>
>>> Thank you for the quick response. I see
`rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please let me know.
>>>
>>> Regards
>>> Jayanth
>>>
>>> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
>>>
>>> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
>>> <jayanthreddy5666(a)gmail.com> wrote:
>>> >
>>> > Hello Wesley and Casey,
>>> >
>>> > We've ended up with the same issue and here it appears that even the
user with "--admin" isn't able to do anything. We're now unable to
figure out if it is due to bucket policies, ACLs or IAM of some sort. I'm seeing these
IAM errors in the logs
>>> >
>>> > ```
>>> >
>>> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851
0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due to Handler error.
>>> >
>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due to Handler
error.
>>>
>>> it's failing to parse the bucket policy document, but the error
>>> message doesn't say what's wrong with it
>>>
>>> disabling rgw_policy_reject_invalid_principals might help if it's
>>> failing on the Principal
>>>
>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
0.000000000s s3:list_bucket init_permissions on
:window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
>>> > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req 13293029267332025583
0.000000000s op->ERRORHANDLER: err_no=-13 new_err_no=-13
>>> >
>>> > ```
>>> >
>>> > Please help what's wrong here. We're in Ceph v17.2.7.
>>> >
>>> > Regards,
>>> > Jayanth
>>> >
>>> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham
<wes(a)wesdillingham.com> wrote:
>>> >>
>>> >> Thank you, this has worked to remove the policy.
>>> >>
>>> >> Respectfully,
>>> >>
>>> >> *Wes Dillingham*
>>> >> wes(a)wesdillingham.com
>>> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>>> >>
>>> >>
>>> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley
<cbodley(a)redhat.com> wrote:
>>> >>
>>> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham
<wes(a)wesdillingham.com>
>>> >> > wrote:
>>> >> > >
>>> >> > > Thank you, I am not sure (inherited cluster). I presume
such an admin
>>> >> > user created after-the-fact would work?
>>> >> >
>>> >> > yes
>>> >> >
>>> >> > > Is there a good way to discover an admin user other than
iterate over
>>> >> > all users and retrieve user information? (I presume
radosgw-admin user info
>>> >> > --uid=<user>" would illustrate such administrative
access?
>>> >> >
>>> >> > not sure there's an easy way to search existing users, but
you could
>>> >> > create a temporary admin user for this repair
>>> >> >
>>> >> > >
>>> >> > > Respectfully,
>>> >> > >
>>> >> > > Wes Dillingham
>>> >> > > wes(a)wesdillingham.com
>>> >> > > LinkedIn
>>> >> > >
>>> >> > >
>>> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley
<cbodley(a)redhat.com> wrote:
>>> >> > >>
>>> >> > >> if you have an administrative user (created with
--admin), you should
>>> >> > >> be able to use its credentials with awscli to delete
or overwrite this
>>> >> > >> bucket policy
>>> >> > >>
>>> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham
<
>>> >> > wes(a)wesdillingham.com> wrote:
>>> >> > >> >
>>> >> > >> > I have a bucket which got injected with bucket
policy which locks the
>>> >> > >> > bucket even to the bucket owner. The bucket now
cannot be accessed
>>> >> > (even
>>> >> > >> > get its info or delete bucket policy does not
work) I have looked in
>>> >> > the
>>> >> > >> > radosgw-admin command for a way to delete a
bucket policy but do not
>>> >> > see
>>> >> > >> > anything. I presume I will need to somehow remove
the bucket policy
>>> >> > from
>>> >> > >> > however it is stored in the bucket metadata /
omap etc. If anyone can
>>> >> > point
>>> >> > >> > me in the right direction on that I would
appreciate it. Thanks
>>> >> > >> >
>>> >> > >> > Respectfully,
>>> >> > >> >
>>> >> > >> > *Wes Dillingham*
>>> >> > >> > wes(a)wesdillingham.com
>>> >> > >> > LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
>>> >> > >> > _______________________________________________
>>> >> > >> > ceph-users mailing list -- ceph-users(a)ceph.io
>>> >> > >> > To unsubscribe send an email to
ceph-users-leave(a)ceph.io
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >> >
>>> >> _______________________________________________
>>> >> ceph-users mailing list -- ceph-users(a)ceph.io
>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
6:11 p.m.
Hello Casey,
Thank you so much for the response. I'm applying these right now and let
you know the results.
Regards,
Jayanth
On Wed, Nov 8, 2023 at 8:15 PM Casey Bodley <cbodley(a)redhat.com> wrote:
i've opened https://tracker.ceph.com/issues/63485
to allow
admin/system users to override policy parsing errors like this. i'm
not sure yet where this parsing regression was introduced. in reef,
https://github.com/ceph/ceph/pull/49395 added better error messages
here, along with a rgw_policy_reject_invalid_principals option to be
strict about principal names
to remove a bucket policy that fails to parse with "Error reading IAM
Policy", you can follow these steps:
1. find the bucket's instance id using the 'bucket stats' command
$ radosgw-admin bucket stats --bucket {bucketname} | grep id
2. use the rados tool to remove the bucket policy attribute
(user.rgw.iam-policy) from the bucket instance metadata object
$ rados -p default.rgw.meta -N root rmxattr
.bucket.meta.{bucketname}:{bucketid} user.rgw.iam-policy
3. radosgws may be caching the existing bucket metadata and xattrs, so
you'd either need to restart them or clear their metadata caches
$ ceph daemon client.rgw.xyz cache zap
On Wed, Nov 8, 2023 at 9:06 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
wrote:
Hello Wesley,
Thank you for the response. I tried the same but ended up with 403.
Regards,
Jayanth
On Wed, Nov 8, 2023 at 7:34 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
>
> Jaynath:
>
> Just to be clear with the "--admin" user's key's you have attempted
to
delete the bucket policy using the following method:
https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy…
>
> This is what worked for me (on a 16.2.14 cluster). I didn't attempt to
interact with the affected bucket in any way other than "aws s3api
delete-bucket-policy"
>
> Respectfully,
>
> Wes Dillingham
> wes(a)wesdillingham.com
> LinkedIn
>
>
> On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy <
jayanthreddy5666(a)gmail.com> wrote:
>>
>> Hello Casey,
>>
>> We're totally stuck at this point and none of the options seem to
work. Please let us know if there is something in metadata or index to
remove those applied bucket policies. We downgraded to v17.2.6 and
encountering the same.
>>
>> Regards,
>> Jayanth
>>
>> On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <
jayanthreddy5666(a)gmail.com> wrote:
>>>
>>> Hello Casey,
>>>
>>> And on further inspection, we identified that there were bucket
policies set from the initial days; we were in v16.2.12.
>>> We upgraded the cluster to v17.2.7
two days ago and it seems obvious
that the IAM error logs are generated the next
minute rgw daemon upgraded
from v16.2.12 to v17.2.7. Looks like there is some issue with parsing.
>>>
>>> I'm thinking to downgrade back to v17.2.6 and earlier, please let me
know if this is a good option for now.
>>>
>>> Thanks,
>>> Jayanth
>>> ________________________________
>>> From: Jayanth Reddy <jayanthreddy5666(a)gmail.com>
>>> Sent: Tuesday, November 7, 2023 11:59:38 PM
>>> To: Casey Bodley <cbodley(a)redhat.com>
>>> Cc: Wesley Dillingham <wes(a)wesdillingham.com>om>; ceph-users <
ceph-users(a)ceph.io>gt;; Adam Emerson <aemerson(a)redhat.com>
>>> Subject: Re: [ceph-users] Re: owner
locked out of bucket via bucket
policy
>>>
>>> Hello Casey,
>>>
>>> Thank you for the quick response. I see
`rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please
let me know.
>>>
>>> Regards
>>> Jayanth
>>>
>>> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
>>>
>>> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
>>> <jayanthreddy5666(a)gmail.com> wrote:
>>> >
>>> > Hello Wesley and Casey,
>>> >
>>> > We've ended up with the same issue and here it appears that even
the user with "--admin" isn't able to do anything. We're now
unable to
figure out if it is due to bucket policies, ACLs or IAM of some sort. I'm
seeing these IAM errors in the logs
>>> >
>>> > ```
>>> >
>>> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851
0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due to
Handler error.
>>> >
>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due
to Handler error.
>>>
>>> it's failing to parse the bucket policy document, but the error
>>> message doesn't say what's wrong with it
>>>
>>> disabling rgw_policy_reject_invalid_principals might help if it's
>>> failing on the Principal
>>>
>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
0.000000000s s3:list_bucket init_permissions on
:window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
>>> > Nov 7 22:51:40 ceph-feed-05
radosgw[4054570]: req
13293029267332025583 0.000000000s op->ERRORHANDLER:
err_no=-13
new_err_no=-13
>>> >
>>> > ```
>>> >
>>> > Please help what's wrong here. We're in Ceph v17.2.7.
>>> >
>>> > Regards,
>>> > Jayanth
>>> >
>>> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <
wes(a)wesdillingham.com> wrote:
>>> >>
>>> >> Thank you, this has worked to remove the policy.
>>> >>
>>> >> Respectfully,
>>> >>
>>> >> *Wes Dillingham*
>>> >> wes(a)wesdillingham.com
>>> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>>> >>
>>> >>
>>> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley
<cbodley(a)redhat.com>
wrote:
>>> >>
>>> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham <
wes(a)wesdillingham.com>
>>> >> > wrote:
>>> >> > >
>>> >> > > Thank you, I am not sure (inherited cluster). I presume
such
an admin
>>> >> > user created
after-the-fact would work?
>>> >> >
>>> >> > yes
>>> >> >
>>> >> > > Is there a good way to discover an admin user other than
iterate over
>>> >> > all users and retrieve
user information? (I presume
radosgw-admin user info
>>> >> >
--uid=<user>" would illustrate such administrative access?
>>> >> >
>>> >> > not sure there's an easy way to search existing users, but
you
could
>>> >> > create a temporary
admin user for this repair
>>> >> >
>>> >> > >
>>> >> > > Respectfully,
>>> >> > >
>>> >> > > Wes Dillingham
>>> >> > > wes(a)wesdillingham.com
>>> >> > > LinkedIn
>>> >> > >
>>> >> > >
>>> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <
cbodley(a)redhat.com> wrote:
>>> >> > >>
>>> >> > >> if you have an administrative user (created with
--admin),
you should
>>> >> > >> be able to use
its credentials with awscli to delete or
overwrite this
>>> >> > >> bucket policy
>>> >> > >>
>>> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham
<
>>> >> > wes(a)wesdillingham.com> wrote:
>>> >> > >> >
>>> >> > >> > I have a bucket which got injected with bucket
policy which
locks the
>>> >> > >> > bucket
even to the bucket owner. The bucket now cannot be
accessed
>>> >> > (even
>>> >> > >> > get its info or delete bucket policy does not
work) I have
looked in
>>> >> > the
>>> >> > >> > radosgw-admin command for a way to delete a
bucket policy
but do not
>>> >> > see
>>> >> > >> > anything. I presume I will need to somehow remove
the
bucket policy
>>> >> > from
>>> >> > >> > however it is stored in the bucket metadata /
omap etc. If
anyone can
>>> >> > point
>>> >> > >> > me in the right direction on that I would
appreciate it.
Thanks
>>> >> > >> >
>>> >> > >> > Respectfully,
>>> >> > >> >
>>> >> > >> > *Wes Dillingham*
>>> >> > >> > wes(a)wesdillingham.com
>>> >> > >> > LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
>>> >> > >> > _______________________________________________
>>> >> > >> > ceph-users mailing list -- ceph-users(a)ceph.io
>>> >> > >> > To unsubscribe send an email to
ceph-users-leave(a)ceph.io
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >> >
>>> >> _______________________________________________
>>> >> ceph-users mailing list -- ceph-users(a)ceph.io
>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
6:30 p.m.
Hello Casey,
Thank you so much, the steps you provided worked. I'll follow up on the
tracker to provide further information.
Regards,
Jayanth
On Wed, Nov 8, 2023 at 8:41 PM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
wrote:
Hello Casey,
Thank you so much for the response. I'm applying these right now and let
you know the results.
Regards,
Jayanth
On Wed, Nov 8, 2023 at 8:15 PM Casey Bodley <cbodley(a)redhat.com> wrote:
> i've opened https://tracker.ceph.com/issues/63485 to allow
> admin/system users to override policy parsing errors like this. i'm
> not sure yet where this parsing regression was introduced. in reef,
> https://github.com/ceph/ceph/pull/49395 added better error messages
> here, along with a rgw_policy_reject_invalid_principals option to be
> strict about principal names
>
>
> to remove a bucket policy that fails to parse with "Error reading IAM
> Policy", you can follow these steps:
>
> 1. find the bucket's instance id using the 'bucket stats' command
>
> $ radosgw-admin bucket stats --bucket {bucketname} | grep id
>
> 2. use the rados tool to remove the bucket policy attribute
> (user.rgw.iam-policy) from the bucket instance metadata object
>
> $ rados -p default.rgw.meta -N root rmxattr
> .bucket.meta.{bucketname}:{bucketid} user.rgw.iam-policy
>
> 3. radosgws may be caching the existing bucket metadata and xattrs, so
> you'd either need to restart them or clear their metadata caches
>
> $ ceph daemon client.rgw.xyz cache zap
>
> On Wed, Nov 8, 2023 at 9:06 AM Jayanth Reddy <jayanthreddy5666(a)gmail.com>
> wrote:
> >
> > Hello Wesley,
> > Thank you for the response. I tried the same but ended up with 403.
> >
> > Regards,
> > Jayanth
> >
> > On Wed, Nov 8, 2023 at 7:34 PM Wesley Dillingham <wes(a)wesdillingham.com>
> wrote:
> >>
> >> Jaynath:
> >>
> >> Just to be clear with the "--admin" user's key's you have
attempted to
> delete the bucket policy using the following method:
> https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy…
> >>
> >> This is what worked for me (on a 16.2.14 cluster). I didn't attempt to
> interact with the affected bucket in any way other than "aws s3api
> delete-bucket-policy"
> >>
> >> Respectfully,
> >>
> >> Wes Dillingham
> >> wes(a)wesdillingham.com
> >> LinkedIn
> >>
> >>
> >> On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy <
> jayanthreddy5666(a)gmail.com> wrote:
> >>>
> >>> Hello Casey,
> >>>
> >>> We're totally stuck at this point and none of the options seem to
> work. Please let us know if there is something in metadata or index to
> remove those applied bucket policies. We downgraded to v17.2.6 and
> encountering the same.
> >>>
> >>> Regards,
> >>> Jayanth
> >>>
> >>> On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <
> jayanthreddy5666(a)gmail.com> wrote:
> >>>>
> >>>> Hello Casey,
> >>>>
> >>>> And on further inspection, we identified that there were bucket
> policies set from the initial days; we were in v16.2.12.
> >>>> We upgraded the cluster to v17.2.7 two days ago and it seems
obvious
> that the IAM error logs are generated the next minute rgw daemon upgraded
> from v16.2.12 to v17.2.7. Looks like there is some issue with parsing.
> >>>>
> >>>> I'm thinking to downgrade back to v17.2.6 and earlier, please
let me
> know if this is a good option for now.
> >>>>
> >>>> Thanks,
> >>>> Jayanth
> >>>> ________________________________
> >>>> From: Jayanth Reddy <jayanthreddy5666(a)gmail.com>
> >>>> Sent: Tuesday, November 7, 2023 11:59:38 PM
> >>>> To: Casey Bodley <cbodley(a)redhat.com>
> >>>> Cc: Wesley Dillingham <wes(a)wesdillingham.com>om>; ceph-users
<
> ceph-users(a)ceph.io>gt;; Adam Emerson <aemerson(a)redhat.com>
> >>>> Subject: Re: [ceph-users] Re: owner locked out of bucket via bucket
> policy
> >>>>
> >>>> Hello Casey,
> >>>>
> >>>> Thank you for the quick response. I see
> `rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please
> let me know.
> >>>>
> >>>> Regards
> >>>> Jayanth
> >>>>
> >>>> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley
<cbodley(a)redhat.com>
> wrote:
> >>>>
> >>>> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
> >>>> <jayanthreddy5666(a)gmail.com> wrote:
> >>>> >
> >>>> > Hello Wesley and Casey,
> >>>> >
> >>>> > We've ended up with the same issue and here it appears that
even
> the user with "--admin" isn't able to do anything. We're now unable
to
> figure out if it is due to bucket policies, ACLs or IAM of some sort. I'm
> seeing these IAM errors in the logs
> >>>> >
> >>>> > ```
> >>>> >
> >>>> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req
8786689665323103851
> 0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due to
> Handler error.
> >>>> >
> >>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req
13293029267332025583
> 0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due
> to Handler error.
> >>>>
> >>>> it's failing to parse the bucket policy document, but the error
> >>>> message doesn't say what's wrong with it
> >>>>
> >>>> disabling rgw_policy_reject_invalid_principals might help if
it's
> >>>> failing on the Principal
> >>>>
> >>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req
13293029267332025583
> 0.000000000s s3:list_bucket init_permissions on
> :window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13
> >>>> > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req
> 13293029267332025583 0.000000000s op->ERRORHANDLER: err_no=-13
> new_err_no=-13
> >>>> >
> >>>> > ```
> >>>> >
> >>>> > Please help what's wrong here. We're in Ceph v17.2.7.
> >>>> >
> >>>> > Regards,
> >>>> > Jayanth
> >>>> >
> >>>> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <
> wes(a)wesdillingham.com> wrote:
> >>>> >>
> >>>> >> Thank you, this has worked to remove the policy.
> >>>> >>
> >>>> >> Respectfully,
> >>>> >>
> >>>> >> *Wes Dillingham*
> >>>> >> wes(a)wesdillingham.com
> >>>> >> LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
> >>>> >>
> >>>> >>
> >>>> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley
<cbodley(a)redhat.com>
> wrote:
> >>>> >>
> >>>> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham
<
> wes(a)wesdillingham.com>
> >>>> >> > wrote:
> >>>> >> > >
> >>>> >> > > Thank you, I am not sure (inherited cluster). I
presume such
> an admin
> >>>> >> > user created after-the-fact would work?
> >>>> >> >
> >>>> >> > yes
> >>>> >> >
> >>>> >> > > Is there a good way to discover an admin user
other than
> iterate over
> >>>> >> > all users and retrieve user information? (I presume
> radosgw-admin user info
> >>>> >> > --uid=<user>" would illustrate such
administrative access?
> >>>> >> >
> >>>> >> > not sure there's an easy way to search existing
users, but you
> could
> >>>> >> > create a temporary admin user for this repair
> >>>> >> >
> >>>> >> > >
> >>>> >> > > Respectfully,
> >>>> >> > >
> >>>> >> > > Wes Dillingham
> >>>> >> > > wes(a)wesdillingham.com
> >>>> >> > > LinkedIn
> >>>> >> > >
> >>>> >> > >
> >>>> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley
<
> cbodley(a)redhat.com> wrote:
> >>>> >> > >>
> >>>> >> > >> if you have an administrative user (created
with --admin),
> you should
> >>>> >> > >> be able to use its credentials with awscli to
delete or
> overwrite this
> >>>> >> > >> bucket policy
> >>>> >> > >>
> >>>> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley
Dillingham <
> >>>> >> > wes(a)wesdillingham.com> wrote:
> >>>> >> > >> >
> >>>> >> > >> > I have a bucket which got injected with
bucket policy
> which locks the
> >>>> >> > >> > bucket even to the bucket owner. The
bucket now cannot be
> accessed
> >>>> >> > (even
> >>>> >> > >> > get its info or delete bucket policy
does not work) I have
> looked in
> >>>> >> > the
> >>>> >> > >> > radosgw-admin command for a way to
delete a bucket policy
> but do not
> >>>> >> > see
> >>>> >> > >> > anything. I presume I will need to
somehow remove the
> bucket policy
> >>>> >> > from
> >>>> >> > >> > however it is stored in the bucket
metadata / omap etc. If
> anyone can
> >>>> >> > point
> >>>> >> > >> > me in the right direction on that I
would appreciate it.
> Thanks
> >>>> >> > >> >
> >>>> >> > >> > Respectfully,
> >>>> >> > >> >
> >>>> >> > >> > *Wes Dillingham*
> >>>> >> > >> > wes(a)wesdillingham.com
> >>>> >> > >> > LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
> >>>> >> > >> >
_______________________________________________
> >>>> >> > >> > ceph-users mailing list --
ceph-users(a)ceph.io
> >>>> >> > >> > To unsubscribe send an email to
ceph-users-leave(a)ceph.io
> >>>> >> > >> >
> >>>> >> > >>
> >>>> >> >
> >>>> >> >
> >>>> >> _______________________________________________
> >>>> >> ceph-users mailing list -- ceph-users(a)ceph.io
> >>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >>>>
>
>
194
days inactive
208
days old
15 comments
4 participants
participants (4)
-
Casey Bodley -
Jayanth Reddy -
Siddhit Renake -
Wesley Dillingham