Awesome thanks :))

I have found it

When I list bucket a bucket is exists but when I get bucket stats for that bucket it again fails with -2!

This will happens:

`couldn't put binfo cache entry, might have raced with data changes`

And rapidly gets this error for each OSD that mon trying to get info

```

osd_op(unknown.0.0:1 49.13a 49:5cdd94e7:::.dir.ac7cb16a-c4d4-4e8e-a6ad-97898dc5af11.108925803.3.0:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e214222) v8 -- 0x5622f89a6700 con 0x5622f89a3a60
conn(0x5622f89a97e0 0x5622f89abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect

```

Until I get `error getting bucket stats`

On Fri, May 8, 2020 at 6:45 PM Casey Bodley <cbodley@redhat.com> wrote:

You can add '--debug-rgw 20 --debug-ms 1' to this radosgw-admin
command. You don't have to change log levels for radosgws.

On Fri, May 8, 2020 at 10:04 AM Seena Fallah <seenafallah@gmail.com> wrote:
>
> So how can I debug that which bucket is getting error? I will still get that error without any further information!
> If you mean that I change the debug level to 20 and run this command to see what’s wrong I think it’s getting hard to debug it! I’m in a production mode and I should change config reload services and also look forward a long and heavy log file in radosgw to find out what’s wrong! This gets more cost and the error says which bucket is getting this error so I will try just on that bucket and trace that bucket with debug 20.
> If I set debug to 20 now there will be a heavy logs comes from radosgw for sharding and others that gets mess!
>
> Don’t you agree with it? It really hard to change log level in production mode :(
>
> On Fri, May 8, 2020 at 6:28 PM Casey Bodley <cbodley@redhat.com> wrote:
>>
>> On Fri, May 8, 2020 at 9:52 AM Seena Fallah <seenafallah@gmail.com> wrote:
>> >
>> > Im just calling an admin api for getting all usage for all buckets. I don’t specify any buckets there! And also sometimes getting bucket stats from radosgw-admin that if filtered by uid that is exists and against I don’t specify any bucket!
>>
>> The `radosgw-admin bucket stats` command (without additional arguments
>> like --uid or --bucket) will list all buckets the same way that the
>> admin api does.
>>
>> >
>> > On Fri, May 8, 2020 at 6:18 PM Casey Bodley <cbodley@redhat.com> wrote:
>> >>
>> >> The `radosgw-admin bucket stats` will run the same code as this admin
>> >> api, so you can debug the command itself without touching your
>> >> gateways.
>> >>
>> >> On Fri, May 8, 2020 at 9:28 AM Seena Fallah <seenafallah@gmail.com> wrote:
>> >> >
>> >> > I’m in a production mode and I can’t go on debug 20 always. This error is useless by it self, if it contains bucket name it would be so helpful than it just says error something that isn’t available (-2).
>> >> >
>> >> > On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley@redhat.com> wrote:
>> >> >>
>> >> >> On Fri, May 8, 2020 at 6:45 AM Seena Fallah <seenafallah@gmail.com> wrote:
>> >> >> >
>> >> >> > Hi all.
>> >> >> >
>> >> >> > I'm getting lots of error getting bucket stats from my rgw instances
>> >> >> > and I can't figure out what's going wrong!
>> >> >>
>> >> >> can you enable debug_rgw=20 and debug_ms=1? that output will include
>> >> >> the relevant bucket and bucket instance names. if you suspect bugs
>> >> >> here, please open a tracker issue with the logs!
>> >> >>
>> >> >> > Can we add bucket name to this error to see whats happening in the cluster?
>> >> >> > https://github.com/ceph/ceph/pull/34653
>> >> >> >
>> >> >> > Thanks.
>> >> >> > _______________________________________________
>> >> >> > Dev mailing list -- dev@ceph.io
>> >> >> > To unsubscribe send an email to dev-leave@ceph.io
>> >> >> >
>> >> >>
>> >>
>>