8 May
2020
8 May
'20
3:45 a.m.
Hi all.
I'm getting lots of error getting bucket stats from my rgw instances
and I can't figure out what's going wrong!
Can we add bucket name to this error to see whats happening in the cluster?
https://github.com/ceph/ceph/pull/34653
Thanks.
8 May
8 May
6:25 a.m.
On Fri, May 8, 2020 at 6:45 AM Seena Fallah <seenafallah(a)gmail.com> wrote:
Hi all.
I'm getting lots of error getting bucket stats from my rgw instances
and I can't figure out what's going wrong!
can you enable debug_rgw=20 and debug_ms=1? that output will include
the relevant bucket and bucket instance names. if you suspect bugs
here, please open a tracker issue with the logs!
Can we add bucket name to this error to see whats
happening in the cluster?
https://github.com/ceph/ceph/pull/34653
Thanks.
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
6:27 a.m.
I’m in a production mode and I can’t go on debug 20 always. This error is
useless by it self, if it contains bucket name it would be so helpful than
it just says error something that isn’t available (-2).
On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Fri, May 8, 2020 at 6:45 AM Seena Fallah
<seenafallah(a)gmail.com> wrote:
Hi all.
I'm getting lots of error getting bucket stats from my rgw instances
and I can't figure out what's going wrong!
can you enable debug_rgw=20 and debug_ms=1? that output will include
the relevant bucket and bucket instance names. if you suspect bugs
here, please open a tracker issue with the logs!
Can we add bucket name to this error to see whats
happening in the
cluster?
https://github.com/ceph/ceph/pull/34653
Thanks.
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
6:48 a.m.
The `radosgw-admin bucket stats` will run the same code as this admin
api, so you can debug the command itself without touching your
gateways.
On Fri, May 8, 2020 at 9:28 AM Seena Fallah <seenafallah(a)gmail.com> wrote:
I’m in a production mode and I can’t go on debug 20 always. This error is useless by it
self, if it contains bucket name it would be so helpful than it just says error something
that isn’t available (-2).
On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> On Fri, May 8, 2020 at 6:45 AM Seena Fallah <seenafallah(a)gmail.com> wrote:
> >
> > Hi all.
> >
> > I'm getting lots of error getting bucket stats from my rgw instances
> > and I can't figure out what's going wrong!
>
> can you enable debug_rgw=20 and debug_ms=1? that output will include
> the relevant bucket and bucket instance names. if you suspect bugs
> here, please open a tracker issue with the logs!
>
> > Can we add bucket name to this error to see whats happening in the cluster?
> > https://github.com/ceph/ceph/pull/34653
> >
> > Thanks.
> > _______________________________________________
> > Dev mailing list -- dev(a)ceph.io
> > To unsubscribe send an email to dev-leave(a)ceph.io
> >
>
6:51 a.m.
Im just calling an admin api for getting all usage for all buckets. I don’t
specify any buckets there! And also sometimes getting bucket stats from
radosgw-admin that if filtered by uid that is exists and against I don’t
specify any bucket!
On Fri, May 8, 2020 at 6:18 PM Casey Bodley <cbodley(a)redhat.com> wrote:
The `radosgw-admin bucket stats` will run the same
code as this admin
api, so you can debug the command itself without touching your
gateways.
On Fri, May 8, 2020 at 9:28 AM Seena Fallah <seenafallah(a)gmail.com> wrote:
I’m in a production mode and I can’t go on debug 20 always. This error
is useless
by it self, if it contains bucket name it would be so helpful
than it just says error something that isn’t available (-2).
On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> On Fri, May 8, 2020 at 6:45 AM Seena Fallah <seenafallah(a)gmail.com>
wrote:
> >
> > Hi all.
> >
> > I'm getting lots of error getting bucket stats from my rgw instances
> > and I can't figure out what's going wrong!
>
> can you enable debug_rgw=20 and debug_ms=1? that output will include
> the relevant bucket and bucket instance names. if you suspect bugs
> here, please open a tracker issue with the logs!
>
> > Can we add bucket name to this error to see whats happening in the
cluster?
> > https://github.com/ceph/ceph/pull/34653
> >
> > Thanks.
> > _______________________________________________
> > Dev mailing list -- dev(a)ceph.io
> > To unsubscribe send an email to dev-leave(a)ceph.io
> >
>
6:58 a.m.
On Fri, May 8, 2020 at 9:52 AM Seena Fallah <seenafallah(a)gmail.com> wrote:
Im just calling an admin api for getting all usage for all buckets. I don’t specify any
buckets there! And also sometimes getting bucket stats from radosgw-admin that if filtered
by uid that is exists and against I don’t specify any bucket!
The `radosgw-admin bucket stats` command (without additional arguments
like --uid or --bucket) will list all buckets the same way that the
admin api does.
On Fri, May 8, 2020 at 6:18 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> The `radosgw-admin bucket stats` will run the same code as this admin
> api, so you can debug the command itself without touching your
> gateways.
>
> On Fri, May 8, 2020 at 9:28 AM Seena Fallah <seenafallah(a)gmail.com> wrote:
> >
> > I’m in a production mode and I can’t go on debug 20 always. This error is
useless by it self, if it contains bucket name it would be so helpful than it just says
error something that isn’t available (-2).
> >
> > On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley(a)redhat.com> wrote:
> >>
> >> On Fri, May 8, 2020 at 6:45 AM Seena Fallah <seenafallah(a)gmail.com>
wrote:
> >> >
> >> > Hi all.
> >> >
> >> > I'm getting lots of error getting bucket stats from my rgw
instances
> >> > and I can't figure out what's going wrong!
> >>
> >> can you enable debug_rgw=20 and debug_ms=1? that output will include
> >> the relevant bucket and bucket instance names. if you suspect bugs
> >> here, please open a tracker issue with the logs!
> >>
> >> > Can we add bucket name to this error to see whats happening in the
cluster?
> >> > https://github.com/ceph/ceph/pull/34653
> >> >
> >> > Thanks.
> >> > _______________________________________________
> >> > Dev mailing list -- dev(a)ceph.io
> >> > To unsubscribe send an email to dev-leave(a)ceph.io
> >> >
> >>
>
7:03 a.m.
So how can I debug that which bucket is getting error? I will still get
that error without any further information!
If you mean that I change the debug level to 20 and run this command to see
what’s wrong I think it’s getting hard to debug it! I’m in a production
mode and I should change config reload services and also look forward a
long and heavy log file in radosgw to find out what’s wrong! This gets more
cost and the error says which bucket is getting this error so I will try
just on that bucket and trace that bucket with debug 20.
If I set debug to 20 now there will be a heavy logs comes from radosgw for
sharding and others that gets mess!
Don’t you agree with it? It really hard to change log level in production
mode :(
On Fri, May 8, 2020 at 6:28 PM Casey Bodley <cbodley(a)redhat.com> wrote:
On Fri, May 8, 2020 at 9:52 AM Seena Fallah
<seenafallah(a)gmail.com> wrote:
Im just calling an admin api for getting all usage for all buckets. I
don’t
specify any buckets there! And also sometimes getting bucket stats
from radosgw-admin that if filtered by uid that is exists and against I
don’t specify any bucket!
The `radosgw-admin bucket stats` command (without additional arguments
like --uid or --bucket) will list all buckets the same way that the
admin api does.
On Fri, May 8, 2020 at 6:18 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> The `radosgw-admin bucket stats` will run the same code as this admin
> api, so you can debug the command itself without touching your
> gateways.
>
> On Fri, May 8, 2020 at 9:28 AM Seena Fallah <seenafallah(a)gmail.com>
wrote:
> >
> > I’m in a production mode and I can’t go on debug 20 always. This
error is
useless by it self, if it contains bucket name it would be so
helpful than it just says error something that isn’t available (-2).
> >
> > On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
> >>
> >> On Fri, May 8, 2020 at 6:45 AM Seena Fallah <seenafallah(a)gmail.com>
wrote:
> >> >
> >> > Hi all.
> >> >
> >> > I'm getting lots of error getting bucket stats from my rgw
instances
> >> > and I can't figure out
what's going wrong!
> >>
> >> can you enable debug_rgw=20 and debug_ms=1? that output will include
> >> the relevant bucket and bucket instance names. if you suspect bugs
> >> here, please open a tracker issue with the logs!
> >>
> >> > Can we add bucket name to this error to see whats happening in the
cluster?
> >> >
https://github.com/ceph/ceph/pull/34653
> >> >
> >> > Thanks.
> >> > _______________________________________________
> >> > Dev mailing list -- dev(a)ceph.io
> >> > To unsubscribe send an email to dev-leave(a)ceph.io
> >> >
> >>
>
7:15 a.m.
You can add '--debug-rgw 20 --debug-ms 1' to this radosgw-admin
command. You don't have to change log levels for radosgws.
On Fri, May 8, 2020 at 10:04 AM Seena Fallah <seenafallah(a)gmail.com> wrote:
So how can I debug that which bucket is getting error? I will still get that error
without any further information!
If you mean that I change the debug level to 20 and run this command to see what’s wrong
I think it’s getting hard to debug it! I’m in a production mode and I should change config
reload services and also look forward a long and heavy log file in radosgw to find out
what’s wrong! This gets more cost and the error says which bucket is getting this error so
I will try just on that bucket and trace that bucket with debug 20.
If I set debug to 20 now there will be a heavy logs comes from radosgw for sharding and
others that gets mess!
Don’t you agree with it? It really hard to change log level in production mode :(
On Fri, May 8, 2020 at 6:28 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> On Fri, May 8, 2020 at 9:52 AM Seena Fallah <seenafallah(a)gmail.com> wrote:
> >
> > Im just calling an admin api for getting all usage for all buckets. I don’t
specify any buckets there! And also sometimes getting bucket stats from radosgw-admin that
if filtered by uid that is exists and against I don’t specify any bucket!
>
> The `radosgw-admin bucket stats` command (without additional arguments
> like --uid or --bucket) will list all buckets the same way that the
> admin api does.
>
> >
> > On Fri, May 8, 2020 at 6:18 PM Casey Bodley <cbodley(a)redhat.com> wrote:
> >>
> >> The `radosgw-admin bucket stats` will run the same code as this admin
> >> api, so you can debug the command itself without touching your
> >> gateways.
> >>
> >> On Fri, May 8, 2020 at 9:28 AM Seena Fallah <seenafallah(a)gmail.com>
wrote:
> >> >
> >> > I’m in a production mode and I can’t go on debug 20 always. This error
is useless by it self, if it contains bucket name it would be so helpful than it just says
error something that isn’t available (-2).
> >> >
> >> > On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
> >> >>
> >> >> On Fri, May 8, 2020 at 6:45 AM Seena Fallah
<seenafallah(a)gmail.com> wrote:
> >> >> >
> >> >> > Hi all.
> >> >> >
> >> >> > I'm getting lots of error getting bucket stats from my rgw
instances
> >> >> > and I can't figure out what's going wrong!
> >> >>
> >> >> can you enable debug_rgw=20 and debug_ms=1? that output will
include
> >> >> the relevant bucket and bucket instance names. if you suspect bugs
> >> >> here, please open a tracker issue with the logs!
> >> >>
> >> >> > Can we add bucket name to this error to see whats happening in
the cluster?
> >> >> > https://github.com/ceph/ceph/pull/34653
> >> >> >
> >> >> > Thanks.
> >> >> > _______________________________________________
> >> >> > Dev mailing list -- dev(a)ceph.io
> >> >> > To unsubscribe send an email to dev-leave(a)ceph.io
> >> >> >
> >> >>
> >>
>
8:17 a.m.
Awesome thanks :))
I have found it
When I list bucket a bucket is exists but when I get bucket stats for that
bucket it again fails with -2!
This will happens:
`couldn't put binfo cache entry, might have raced with data changes`
And rapidly gets this error for each OSD that mon trying to get info
```
osd_op(unknown.0.0:1 49.13a
49:5cdd94e7:::.dir.ac7cb16a-c4d4-4e8e-a6ad-97898dc5af11.108925803.3.0:head
[call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e214222)
v8 -- 0x5622f89a6700 con 0x5622f89a3a60
conn(0x5622f89a97e0 0x5622f89abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0
tx=0).connect
```
Until I get `error getting bucket stats`
On Fri, May 8, 2020 at 6:45 PM Casey Bodley <cbodley(a)redhat.com> wrote:
You can add '--debug-rgw 20 --debug-ms 1' to
this radosgw-admin
command. You don't have to change log levels for radosgws.
On Fri, May 8, 2020 at 10:04 AM Seena Fallah <seenafallah(a)gmail.com>
wrote:
So how can I debug that which bucket is getting error? I will still get
that error
without any further information!
If you mean that I change the debug level to 20
and run this command to
see what’s wrong I think it’s getting hard to debug it! I’m
in a production
mode and I should change config reload services and also look forward a
long and heavy log file in radosgw to find out what’s wrong! This gets more
cost and the error says which bucket is getting this error so I will try
just on that bucket and trace that bucket with debug 20.
If I set debug to 20 now there will be a heavy
logs comes from radosgw
for sharding and others that gets mess!
Don’t you agree with it? It really hard to change log level in
production mode :(
On Fri, May 8, 2020 at 6:28 PM Casey Bodley <cbodley(a)redhat.com> wrote:
>
> On Fri, May 8, 2020 at 9:52 AM Seena Fallah <seenafallah(a)gmail.com>
wrote:
> >
> > Im just calling an admin api for getting all usage for all buckets. I
don’t specify any buckets there! And also sometimes getting bucket stats
from radosgw-admin that if filtered by uid that is exists and against I
don’t specify any bucket!
>
> The `radosgw-admin bucket stats` command (without additional arguments
> like --uid or --bucket) will list all buckets the same way that the
> admin api does.
>
> >
> > On Fri, May 8, 2020 at 6:18 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
> >>
> >> The `radosgw-admin bucket stats` will run the same code as this admin
> >> api, so you can debug the command itself without touching your
> >> gateways.
> >>
> >> On Fri, May 8, 2020 at 9:28 AM Seena Fallah <seenafallah(a)gmail.com>
wrote:
> >> >
> >> > I’m in a production mode and I can’t go on debug 20 always. This
error is useless by it self, if it contains bucket name it would be so
helpful than it just says error something that isn’t available (-2).
> >> >
> >> > On Fri, May 8, 2020 at 5:55 PM Casey Bodley <cbodley(a)redhat.com>
wrote:
> >> >>
> >> >> On Fri, May 8, 2020 at 6:45 AM Seena Fallah <
seenafallah(a)gmail.com> wrote:
> >> >> >
> >> >> > Hi all.
> >> >> >
> >> >> > I'm getting lots of error getting bucket stats from my rgw
instances
> >> >> > and I can't figure
out what's going wrong!
> >> >>
> >> >> can you enable debug_rgw=20 and debug_ms=1? that output will
include
> >> >> the relevant bucket and
bucket instance names. if you suspect bugs
> >> >> here, please open a tracker issue with the logs!
> >> >>
> >> >> > Can we add bucket name to this error to see whats happening in
the cluster?
> >> >> >
https://github.com/ceph/ceph/pull/34653
> >> >> >
> >> >> > Thanks.
> >> >> > _______________________________________________
> >> >> > Dev mailing list -- dev(a)ceph.io
> >> >> > To unsubscribe send an email to dev-leave(a)ceph.io
> >> >> >
> >> >>
> >>
>
1449
days inactive
1449
days old
8 comments
2 participants
participants (2)
-
Casey Bodley -
Seena Fallah