Awesome thanks :)) I have found it When I list bucket a bucket is exists but when I get bucket stats for that bucket it again fails with -2! This will happens: `couldn't put binfo cache entry, might have raced with data changes` And rapidly gets this error for each OSD that mon trying to get info ``` osd_op(unknown.0.0:1 49.13a 49:5cdd94e7:::.dir.ac7cb16a-c4d4-4e8e-a6ad-97898dc5af11.108925803.3.0:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e214222) v8 -- 0x5622f89a6700 con 0x5622f89a3a60 conn(0x5622f89a97e0 0x5622f89abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect ``` Until I get `error getting bucket stats` On Fri, May 8, 2020 at 6:45 PM Casey Bodley <cbodley@redhat.com> wrote:
You can add '--debug-rgw 20 --debug-ms 1' to this radosgw-admin command. You don't have to change log levels for radosgws.
On Fri, May 8, 2020 at 10:04 AM Seena Fallah <seenafallah@gmail.com> wrote:
So how can I debug that which bucket is getting error? I will still get
If you mean that I change the debug level to 20 and run this command to see what’s wrong I think it’s getting hard to debug it! I’m in a production mode and I should change config reload services and also look forward a long and heavy log file in radosgw to find out what’s wrong! This gets more cost and the error says which bucket is getting this error so I will try just on that bucket and trace that bucket with debug 20. If I set debug to 20 now there will be a heavy logs comes from radosgw for sharding and others that gets mess!
Don’t you agree with it? It really hard to change log level in
that error without any further information! production mode :(
On Fri, May 8, 2020 at 6:28 PM Casey Bodley <cbodley@redhat.com> wrote:
On Fri, May 8, 2020 at 9:52 AM Seena Fallah <seenafallah@gmail.com>
Im just calling an admin api for getting all usage for all buckets. I
don’t specify any buckets there! And also sometimes getting bucket stats from radosgw-admin that if filtered by uid that is exists and against I don’t specify any bucket!
The `radosgw-admin bucket stats` command (without additional arguments like --uid or --bucket) will list all buckets the same way that the admin api does.
On Fri, May 8, 2020 at 6:18 PM Casey Bodley <cbodley@redhat.com>
wrote:
The `radosgw-admin bucket stats` will run the same code as this admin api, so you can debug the command itself without touching your gateways.
On Fri, May 8, 2020 at 9:28 AM Seena Fallah <seenafallah@gmail.com>
wrote:
I’m in a production mode and I can’t go on debug 20 always. This
error is useless by it self, if it contains bucket name it would be so helpful than it just says error something that isn’t available (-2).
On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley@redhat.com>
wrote:
> > On Fri, May 8, 2020 at 6:45 AM Seena Fallah < seenafallah@gmail.com> wrote: > > > > Hi all. > > > > I'm getting lots of error getting bucket stats from my rgw instances > > and I can't figure out what's going wrong! > > can you enable debug_rgw=20 and debug_ms=1? that output will include > the relevant bucket and bucket instance names. if you suspect bugs > here, please open a tracker issue with the logs! > > > Can we add bucket name to this error to see whats happening in
wrote: the cluster?
> > https://github.com/ceph/ceph/pull/34653 > > > > Thanks. > > _______________________________________________ > > Dev mailing list -- dev@ceph.io > > To unsubscribe send an email to dev-leave@ceph.io > > >